Intelligent lock face privacy processing method and system

Abstract

The invention discloses an intelligent lock face privacy processing method and system, and the method comprises the steps: an intelligent lock A detects a plurality of faces, and carries out the face segmentation and recognition comparison; if the comparison is successful, performing face privacy processing; if the comparison fails, collaborative privacy processing is carried out; after the other intelligent locks receive the multicast message, the unmatched face images are compared, and if comparison succeeds, face privacy processing is carried out; the intelligent lock A counts a multicast reply message, and if an unreplied intelligent lock exists, a query request is sent to the independent transfer gateway; the independent transfer gateway performs health check, and sends a prompt reply message to the intelligent lock which succeeds in health check; and the intelligent lock A integrates the reply message and the privacy processing result, encodes and generates a video image data stream, and sends the video image data stream to the platform for decoding processing. By using the embodiment of the invention, the protection of the face privacy of the owner can be ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent locks, and particularly relates to a method and system for face privacy processing of intelligent locks. Background Art

[0002] With the needs of community cluster management and social governance, it is becoming increasingly common for householders to install intelligent door locks that can be remotely networked. The intelligent door locks are integrated into the remote management platform through the Internet. The householder can unlock and open the door through the face recognition technology of the intelligent door lock, which greatly facilitates the householder. The management agency can remotely monitor the video images of the people at the door through the remote management platform.

[0003] However, there is a risk of leakage of the face privacy of the householder in the face images transmitted to the remote management platform through the Internet. In order to protect the personal privacy in the face images, active protection measures should be taken for the privacy of the face video images. In actual management work, the management agency does not pay attention to the face images of the house owners that can be normally recognized and compared in the intelligent lock (only needs to know who opens the door for management needs), and what it really cares about is the access situation of non-household members (such as strangers, visitors, etc., who may bring security risks). Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for face privacy processing of intelligent locks to solve the deficiencies in the prior art and ensure the protection of the face privacy of the house owners.

[0005] An embodiment of the present application provides a method for face privacy processing of intelligent locks, and the method includes: Face detection and privacy processing: Intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the uncompared face images are sent to other intelligent locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other intelligent locks compare the uncompared face images with the family member list of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in the form of unicast; Message check and reminder: Intelligent lock A counts the multicast reply messages. If there are intelligent locks that have not replied, a query request is sent to the independent transfer gateway; The independent transfer gateway performs a health check and sends a reminder reply message to the intelligent locks that pass the health check; Image integration and encoding: Intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; Intelligent lock A sends the integrated video image data stream to the platform for decoding processing.

[0006] Optionally, the face detection and privacy processing includes: The intelligent lock A detects multiple faces through video images, performs face segmentation, and labels the face image ID and position coordinates. Calculate the similarity between the segmented face images and the list of family members of this lock. If the similarity exceeds the threshold, replace the face image with the person ID. If the comparison fails, send the unmatched face image and face image ID to other intelligent locks in the community in the form of multicast.

[0007] Optionally, the collaborative face privacy processing includes: The intelligent lock B receives the multicast message, extracts the unmatched face image, and compares it with the list of family members of this lock. If the comparison is successful, replace the face image with the person ID and reply with the processing result in the form of unicast. If the comparison fails, reply with the unmatched information in the form of unicast.

[0008] Optionally, the message check and reminder include: The intelligent lock A counts the multicast reply messages and checks whether there are intelligent locks that have not replied. If there are intelligent locks that have not replied, send a query request to the independent relay gateway. The independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check.

[0009] Optionally, the image integration and encoding include: The intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock. Replace the face image with the person ID and blank out the panoramic image. Generate a video image data stream containing descriptive metadata and embed the metadata into the encoded data.

[0010] Another embodiment of the present application provides a system for intelligent lock face privacy processing, and the system includes: A detection module for face detection and privacy processing: The intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, performs face privacy processing; if the comparison fails, sends the unmatched face image to other intelligent locks in the form of a multicast message for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other intelligent locks compare the unmatched face image with the list of family members of this lock. If the comparison is successful, perform face privacy processing and reply with the processing result in the form of unicast. Processing module for message checking and urging: Smart Lock A counts multicast reply messages. If there are smart locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Image integration and encoding: Smart Lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; Smart Lock A sends the integrated video image data stream to the platform for decoding processing.

[0011] Another embodiment of the present application provides a storage medium in which a computer program is stored. Wherein, the computer program is configured to execute the method described in any one of the above when running.

[0012] Another embodiment of the present application provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the method described in any one of the above.

[0013] Compared with the prior art, a method for face privacy processing of a smart lock provided by the present invention is as follows: Smart Lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other smart locks receive the multicast message, they compare the unrecognized face images. If the comparison is successful, face privacy processing is performed; Smart Lock A counts the multicast reply messages. If there are smart locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Smart Lock A integrates the reply messages and privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby being able to ensure the protection of the owner's face privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a hardware structure block diagram of a computer terminal for a method of face privacy processing of a smart lock provided by an embodiment of the present invention; Figure 2 It is a flow schematic diagram of a method of face privacy processing of a smart lock provided by an embodiment of the present invention; Figure 3 It is a structural schematic diagram of a system for face privacy processing of a smart lock provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] The embodiments described below by referring to the drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] An embodiment of the present invention first provides a method for processing face privacy of an intelligent lock, which can be applied to an electronic device, such as a computer terminal, specifically, a general computer, etc.

[0017] The following takes running on a computer terminal as an example to illustrate it in detail. Figure 1 It is a hardware structure block diagram of a computer terminal for a method of processing face privacy of an intelligent lock provided by an embodiment of the present invention. As Figure 1 shown, the computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the memory may include a non-volatile storage medium and an internal memory.

[0018] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any method for processing face privacy of an intelligent lock.

[0019] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0020] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any method for processing face privacy of an intelligent lock.

[0021] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 1 the structure shown in is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0022] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0023] See Figure 2, embodiments of the present invention provide a method for intelligent lock face privacy processing, which may include the following steps: S201, Face detection and privacy processing: Intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the uncompared face images are sent to other intelligent locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other intelligent locks compare the uncompared face images with the family member list of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in the form of unicast; Specifically, the face detection and privacy processing includes: Intelligent lock A detects multiple faces through video images, performs face segmentation and labels the face image ID and position coordinates; calculates the similarity between the segmented face images and the family member list of this lock. If the similarity exceeds the threshold, the face image is replaced with the person ID; if the comparison fails, the uncompared face images and face image IDs are sent to other intelligent locks in the community in the form of multicast.

[0024] Specifically, the collaborative face privacy processing includes: Intelligent lock B receives the multicast message, extracts the uncompared face images and compares them with the family member list of this lock; if the comparison is successful, the face image is replaced with the person ID, and the processing result is replied in the form of unicast; if the comparison fails, the uncompared information is replied in the form of unicast.

[0025] S202, Message check and reminder: Intelligent lock A counts the multicast reply messages. If there are intelligent locks that have not replied, a query request is sent to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that pass the health check; Image integration and encoding: Intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock, encodes and generates a video image data stream containing descriptive metadata; Intelligent lock A sends the integrated video image data stream to the platform for decoding processing. Specifically, the message check and reminder includes: Intelligent lock A counts the multicast reply messages and checks if there are intelligent locks that have not replied; if there are intelligent locks that have not replied, a query request is sent to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that pass the health check.

[0026] Specifically, the image integration and encoding includes: Intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock; replaces the face image with the person ID and performs blanking processing on the panoramic image; generates a video image data stream containing descriptive metadata and embeds the metadata into the encoded data.

[0027] Intelligent lock: It refers to a lock improved on the basis of traditional mechanical locks, which is more intelligent and simpler in terms of user security, identification, and management. The intelligent door lock mentioned in this article has the function of opening the door by face recognition.

[0028] Privacy computing: It is a technology that realizes data analysis and calculation on the premise of ensuring that data is not leaked externally. The privacy computing mentioned in this article is used for privacy protection in the transmission of face images collected by intelligent door locks through the Internet and remote management.

[0029] Lock event: It refers to an event detected and recognized by an intelligent lock. The intelligent lock will extract the information related to this event as lock event information, which includes lock event serial number, door lock ID, time, personnel identity information, face pictures, video clips, etc.

[0030] The purpose of the present invention is to propose a method for protecting the face privacy of community owners. It is sent to other intelligent locks in the community through multicast for each to perform face privacy processing for their own household members. At the same time, multicast reply messages and the processing of urging replies by an independent relay gateway are adopted to ensure that all intelligent locks complete the face privacy processing for their own household members, thereby realizing the face privacy protection of all community owners.

[0031] The present invention proposes a method. After the face image in the intelligent lock is replaced by face cropping in this lock, it is then sent to other intelligent locks for face cropping replacement in the form of multicast. When not all multicast messages are replied, a message urging reply is sent after querying using the IP address of the independent relay gateway, and the health of the intelligent lock is checked and the results are updated in real time to all intelligent locks. The complete process of a technology is as follows: (1) Prerequisites: Intelligent lock: An intelligent lock is installed on the household entrance door of each household (each lock has an independent intelligent lock ID, intelligent lock IP address, family member list, and community door lock list), and is connected to a remote management platform (for convenience of description, hereinafter referred to as the platform), registers with the platform and updates periodically.

[0032] Explanation: The family member list is the list of household members stored in the intelligent lock. The scope of this list only includes the registered residents of this door lock, including unique personnel ID, name, face image collected at the time of registration, etc. Among them, the personnel ID is a unique number generated by the platform for each person in the community. The personnel ID must be resolved in the platform to query the real identity information. The family member list is updated and distributed by the platform regularly; the community door lock list is a set of intelligent door locks constructed and updated by the independent relay gateway regularly, including multicast IP address, intelligent lock ID number, etc.

[0033] Platform: Maintains a list of community members (data items include door lock ID, door lock geographical address, member ID of family members, name, face images collected during registration, etc.), and preserves the correspondence between the face images and member IDs of each formal owner in the community.

[0034] Description: To better manage the floating tenants in the community, the door lock needs to send the picture to the platform every time face recognition is used to open the door, and the platform records the access situation of each tenant. However, to prevent ordinary management personnel from illegally learning the access privacy of formal owners, the pictures reported by the door lock to the platform must be privacy-processed, and the faces of formal owners must be hidden.

[0035] Independent relay gateway: Maintains a list of community door locks (data items include door lock ID, door lock IP address, etc.), preserves the correspondence between the door lock ID and the door lock IP, and the independent relay gateway can also receive multicast messages.

[0036] (2) Face detection and face privacy processing of Smart Lock A: After detecting multiple faces in the video image, Smart Lock A performs face segmentation, recognition and comparison. If the comparison and recognition result is a family member, it unlocks and performs face privacy processing for this family member. If a non-family member is recognized, it is sent to other smart locks in the form of multicast for collaborative face privacy processing.

[0037] Description: For the sake of convenience of description, the smart lock that initiates multicast will be abbreviated as Smart Lock A later, the smart locks of the community door locks will be abbreviated as Smart Lock B, the smart locks that do not reply to the multicast message in time will be abbreviated as Smart Locks E, F, G, and the smart locks with failed health detection will be abbreviated as Smart Locks E, F.

[0038] Face detection and segmentation: When multiple people appear in front of the smart lock, after the smart lock detects the face through the intelligent detection and recognition algorithm, it performs face segmentation to form multiple face images and performs annotation. The labels include lock event ID (which refers to the event of collecting a face, and this serial number can adopt the door lock ID + time + serial number), door lock ID, time, panoramic image ID (the serial number extracted from the video frame, and the number can adopt the lock event serial number + serial number), panoramic image, face image ID (the ID number is unique and is the serial number assigned when segmenting the face area image from the panoramic image, and the number can adopt the lock event serial number + panoramic image ID + serial number), face image position coordinates (upper left corner X, Y coordinates, lower right corner X, Y coordinates), face image); Description: To distinguish the segmented multiple face images and perform image re-encoding, it is necessary to locate and annotate them; if only 1 person is detected and recognized in the video image, it is processed according to the special case of multiple people; assume that the people detected in the current image are four people a, b, c, and d.

[0039] Facial privacy processing: Calculate the similarity between the segmented facial image and the facial images in the household member list of the door lock. When the similarity meets the threshold (usually set to 80%), it indicates a successful comparison. Extract the person ID number of the person in the compared facial image from the household member list to form comparison information including: lock event ID, panoramic image ID, whether it is compared successfully, compared facial image ID, coordinates of the compared facial image, compared facial image, compared person ID.

[0040] Explanation: When this lock conducts a successful facial comparison, it indicates that the person is a household member of this lock. For the purpose of protecting the facial privacy of the formal property owners (household members), the person ID number is used as a label instead. Other door locks cannot resolve the true identity of the person by obtaining this person ID. Only the platform can obtain the true identity information of the person corresponding to this person ID based on the corresponding relationship in the community personnel list under authorized circumstances; The uncompared facial image is not a household member of this lock and may be a household member of other door locks. It is necessary to conduct facial privacy protection processing for the formal property owners (household members) of other door locks; Assume that the person identified by smart lock A is a.

[0041] Multicast to send a collaborative facial privacy processing request message: After smart lock A conducts facial privacy processing, if there is an uncompared facial image, it will multicast the uncompared facial image and the facial image ID to other smart locks in the community for collaborative facial privacy processing (for the facial privacy processing of the household members of their door locks); The message content of the multicast message includes the smart lock ID, lock event ID, facial image ID, and facial image.

[0042] Explanation: Multicast is a network communication method that allows one or more senders / community door locks (smart door locks) to send the same data message to multiple receivers (smart door locks), that is, the sender only needs to send it once and other members can receive it. Multicast uses a specific multicast address to identify a group of receivers; This invention does not care about the multicast technology, but only uses the multicast technology and applies multicast using the API of the network management system (NMS) or specialized multicast management software.

[0043] (3) Smart lock B conducts collaborative facial privacy processing: Smart lock B receives the multicast message of the collaborative facial privacy processing request sent by A, extracts the facial image in the multicast message and compares it with the facial images in the household member list of B. If it is compared successfully, replace the facial image with the person ID number and reply to the processing result of the collaborative facial privacy processing request in a unicast form.

[0044] Receive the multicast message: Smart lock B receives the multicast message sent by smart lock A, extracts the lock event ID, facial image ID, and facial image from the multicast message, and records the door lock IP address of smart lock A.

[0045] Note: There may be multiple face image IDs and their corresponding face images received by the smart lock B from the multicast message. They need to be extracted according to the corresponding relationship. Otherwise, it will cause the wrong splicing position when the smart lock A splices the images.

[0046] Face privacy processing: The smart lock B calculates the similarity between the received face image and the face images in the family member list of the door lock B. When the similarity meets the threshold (usually set to 80%), it indicates a successful comparison. The person ID number of the person corresponding to the face image is extracted from the family member list to form the comparison information, including: lock event ID, whether it is a match, the face image ID of the match, and the person ID of the match.

[0047] Note: If the comparison is successful, it means that the face image is a family member of the smart lock B, and face privacy processing should be performed (replaced with the person ID). To prevent the smart lock A from knowing the person's identity information, after replacing it with the person ID number, A still cannot resolve the person's identity information because there is no person ID and person identity comparison table; assume that the person recognized by the smart lock B is b.

[0048] Unicast reply of face privacy processing result: There are two cases for the comparison result of the smart lock B, which need to be processed separately. The first is that there is no match. The comparison information is replied to the multicast message in unicast form (both the source IP address and the source MAC address are randomly filled), including the ID of the smart lock B, the lock event ID of the received multicast message, and whether it is a match / no. The second case is a match. The comparison information is replied to the multicast message in unicast form (both the source IP address and the source MAC address are randomly filled), including the ID of the smart lock B, the lock event ID in the received multicast message, whether it is a match / yes, the face image ID of the match, and the person ID of the match (if there are multiple people, multiple pairs of face image IDs of the match and person IDs of the match must be replied).

[0049] Note: Replying to the collaborative face privacy processing request in unicast form is to reduce the flooding of multicast messages in the network; randomly filling both the source IP address and the source MAC address is to prevent the smart lock A from guessing the door lock information of B through the source IP address and the source MAC address; if multiple face comparisons are successful, the face image ID of the match and the person ID of the match must be in pairs. For example, face image ID (1) corresponds to person ID (a), and face image ID (2) corresponds to person ID (b). Otherwise, the smart lock A cannot accurately pair after receiving the reply message.

[0050] (4) Processing of the reply to the collaborative face privacy processing request by the smart lock A: The smart lock A receives the unicast reply message from the smart lock B and counts whether there is any missing multicast reply message. If there is a missing message, a query request is sent to the independent relay gateway.

[0051] Receiving the multicast reply message for the collaborative face privacy processing request: After Smart Lock A receives the reply message from B for the collaborative face privacy processing request sent by A, it extracts the comparison information in the message packet, including the ID of Smart Lock B, the lock event ID, whether there is a match, the ID of the face image that matches, and the ID of the person who matches, and then proceeds with the processing. First, it performs reply message statistics, recording the lock event ID / door lock ID / reply message sequence number (numbered in the order of receipt); then it processes the comparison information. If "no" is indicated for whether there is a match, no processing is done. If "yes" is indicated for whether there is a match, it extracts the comparison message of Smart Lock B (the ID of Smart Lock B, the lock event ID, whether there is a match, the ID of the face image that matches, and the ID of the person who matches).

[0052] Note: If the reply message from B is received, reply message statistics must be performed based on the lock event ID because different multicast messages sent with different lock event IDs may have different reply situations; reply message statistics are for checking whether all door locks have replied; the method of reply message statistics is to sequentially number them in the order of receipt of the reply message, that is, the reply message sequence number is incremented by 1 each time a reply message is received; if "yes" is indicated for whether there is a match, the ID of the face image and the ID of the person need to be extracted according to the corresponding relationship.

[0053] Checking for missing multicast reply messages: Smart Lock A starts timing after sending the multicast message for the collaborative face privacy processing request. After a delay of 5 seconds, it checks the reply message statistics situation, compares the recorded lock event ID / door lock ID / reply message sequence number with the door lock IDs in the community door lock list one by one to check whether all have replied. If all have replied, it proceeds to step (7). If there are unreplied messages, first check if there are invalid smart lock IDs (sent by the independent relay gateway) and exclude them, then extract the unreplied door lock IDs from the community door lock list as the set of unreplied door lock IDs (assuming the unreplied door locks are E, F, G), and the data items include the lock event ID and the unreplied door lock ID.

[0054] Note: After the multicast message is sent, due to network connection failures, device offline, etc. of the smart door locks, Smart Lock A may not receive all the expected replies (all the smart locks in the community door lock list).

[0055] Unicast sending a query request for the smart locks with unreplied messages to the independent relay gateway: Smart Lock A unicasts a query request for the smart locks with unreplied messages to the independent relay gateway, including the ID of Smart Lock A, the lock event ID, the unreplied door lock IDs (E, F, G), etc.

[0056] Description: Sending a query request message for smart locks that have not replied to unicast to the independent relay gateway is to send a further query to smart locks that have not replied, and the independent relay gateway will individually notify smart locks that have not replied to multicast messages to reply again.

[0057] (5)Processing of the query request message for smart locks that have not replied sent by the independent relay gateway according to A: The independent relay gateway receives the query request message for smart locks that have not replied sent by smart lock A, queries the corresponding door lock IP address in its community member list (data items include door lock ID, door lock IP address, etc.) according to the door lock ID of the un-answered message, and performs a health check on the smart lock (such as in the form of a ping). If the health check is successful (such as a successful ping), then unicast reminder reply messages are sent one by one to notify the un-answered smart locks to reply. If the health check fails (such as a failed ping), the smart lock ID with a failed health check is synchronized to all smart locks including the community door lock list or replied to smart lock A.

[0058] Receiving and checking smart locks that have not replied: The independent relay gateway receives the query request message for smart locks that have not replied from smart lock A and extracts relevant information, including the ID of smart lock A, the lock event ID, the IDs of un-answered smart locks (E, F, G), etc.

[0059] Description: When the independent relay gateway extracts relevant information from the query request message for smart locks that have not replied from smart lock A, it needs to extract the ID of smart lock A and the lock event ID at the same time because the independent relay gateway may receive query requests from many other smart locks.

[0060] Performing a health check on the smart lock: The independent relay gateway performs a health check (generally in the form of a ping) on smart locks (E, F, G). If the health checks of smart locks E and F fail (such as failed pings for smart locks E and F, and door locks with failed pings are simply called failed door locks), it notifies smart lock A and adopts two information sharing schemes. If the health check of smart lock G is successful (such as a successful ping), a reminder reply message is sent to G.

[0061] Description: The health check generally adopts a method of executing upon request, rather than a periodic method, because periodic checks will periodically wake up all smart locks, causing additional power consumption of the door locks.

[0062] Smart Lock E and F Failure Information Sharing Solution 1: The independent relay gateway caches the failure information of Smart Locks E and F for 5 minutes and deletes it after the expiration; if at this time Door Lock Z detects the presence of multiple people and multicasts a collaborative face privacy processing request, and then finds that Smart Locks E and F do not give feedback, it sends the request to the relay gateway. The relay gateway judges the failure door lock information cached locally. If it finds that there are failed Smart Locks E and F, it directly replies to Door Lock Z with "E and F failed". If it finds that there is no longer any failed door lock information (it may have been deleted after the expiration), it performs a health check on E and F. If it still fails the check, it feeds back to Door Lock Z "E and F failed".

[0063] Smart Lock E and F Failure Information Sharing Solution 2: The relay gateway broadcasts the failure information of Smart Locks E and F to all door locks, specifying a validity period of 5 minutes. All smart locks that receive this failure door lock information only cache it for 5 minutes and delete it after the expiration; if at this time Door Lock Z detects the presence of multiple people and multicasts a collaborative face privacy processing request, and then finds that Smart Locks E and F do not give feedback, it checks the failure door lock information cached locally in Z. If there is still the failure information of Smart Locks E and F, it does not need to send it to the relay gateway again; if there is no longer the failure information of Smart Locks E and F, it sends it to the relay gateway. The relay gateway performs a health check on E and F. If it still fails, it feeds back to Door Lock Z "E and F failed".

[0064] (6) Smart Lock G Receives and Processes the Urging Reply Message from the Independent Relay Gateway: Smart Lock G receives the urging reply message unicast by the independent relay gateway, extracts the face image in the urging reply message and compares it with the face images in the family member list of this lock. If there is a match, it replaces the face image with the person ID number and unicasts the processing result of the collaborative face privacy processing request to A.

[0065] Receiving Unicast Message: Smart Lock G receives the urging reply message from the independent relay gateway, extracts the lock event ID, face image ID, face image, and the IP address of Smart Lock A.

[0066] Explanation: Smart Lock G may have already processed the collaborative face privacy processing request of Smart Lock A, but the message may be lost during the process of sending the reply message. When it receives the urging reply message sent by the independent relay gateway, it processes it again as a new message; the door lock IP address of Smart Lock A needs to be extracted from the message, rather than from the source IP address of the message sent by the independent relay gateway (because the source IP address extracted here is the IP address of the independent relay gateway).

[0067] Face Privacy Processing: Process according to the method in step (3). Assume that the person identified by Smart Lock G is c.

[0068] Unicast reply for face privacy processing result: Process according to the method in step (3).

[0069] Note: At this time, the reply message should be sent according to the IP address of Smart Lock A extracted from the reply message urged by the independent relay gateway.

[0070] (7) Smart Lock A integrates the reply information of face privacy processing for each smart lock: Smart Lock A receives the messages unicast replied by Smart Lock B, the reply messages of Smart Lock G according to the urging requests of the independent relay gateway, the set of smart locks with failed health check by the independent relay gateway, and the face privacy processing results completed by Smart Lock A, and integrates them. Respectively, replace the face image corresponding to the face image ID with the processed person ID, and then re-encode and compress it into a video image data stream.

[0071] Reply message of Smart Lock B: The reply message of Smart Lock B is the reply to the multicast sent collaborative face privacy processing request by Smart Lock A, and extracts the information that has been processed for face privacy, including the ID of Smart Lock B, the lock event ID in the received multicast message, whether it is matched / yes, the face image ID, and the person ID.

[0072] Note: The situation where there is no match during the face privacy processing of Smart Lock B has been processed in step (4). Here, receive the information that has been matched and the person ID is given; assume that the person matched by Smart Lock B is b, and extract the lock event ID, face image ID (b), and person ID (b).

[0073] Reply message of Smart Lock G: The reply message of Smart Lock G is the reply in the form of unicast to Smart Lock A by the independent relay gateway according to the collaborative face privacy processing request of Smart Lock A with an urging request, and extracts the information that has been processed for face privacy, including the ID of Smart Lock G, the lock event ID in the received multicast message, whether it is matched / yes, the face image ID, and the person ID.

[0074] Note: The situation where there is no match during the face privacy processing of Smart Lock G has been processed in step (6). Here, receive the information that has been matched and the person ID is given; assume that the person matched by Smart Lock G is c, and extract the lock event ID, face image ID (c), and person ID (c).

[0075] Set of time-limited door locks sent by the independent relay gateway: Smart Lock A receives the information of the invalid smart locks with failed health check by the independent relay gateway (including the door lock ID numbers with failed health check, assume that the door locks are E and F at this time).

[0076] Comparison information for face privacy processing of intelligent lock A: According to step (2), the person identified by intelligent lock A after face privacy processing is person a. Extract the lock event ID, face image ID (a), person ID (a), face image coordinates (a), and panoramic image of this person. (The comparison information of intelligent lock A includes: lock event ID, panoramic image ID, whether it is matched, matched face image ID, matched face image coordinates, matched face image, and matched person ID).

[0077] Check the completeness of the reply message: Intelligent lock A checks one by one with the community door lock list whether it is complete according to the direct reply to the multicast message received, the reply after urging through the independent relay gateway, and the intelligent lock notified by the independent relay gateway that the health check fails. If it is not complete (if intelligent lock D is missing) and still no message from intelligent lock D is received after waiting for 2 seconds (it may be that the independent relay gateway has sent a reminder reply through the health check, but there may still be a situation where the reply message is lost), then ignore intelligent lock D and enter the image integration processing flow.

[0078] Image integration processing method: Segment and leave blank the panoramic image of the homeowner's face area, perform encoding and compression, and generate descriptive metadata from the face image segmentation area position coordinates and person ID and embed it into the encoded data to form the final video image data stream.

[0079] 1) Intelligent lock A performs a blanking process on the corresponding area in the panoramic image according to the face image ID and the corresponding face image position coordinates (upper left corner X, Y coordinates, lower right corner X, Y coordinates) segmented from the panoramic image collected in step (2) (usually filled with a specific color or transparent color, or set to invalid pixel values); 2) Generate descriptive metadata from the face image ID, the corresponding face image position coordinates (upper left corner X, Y coordinates, lower right corner X, Y coordinates), and the person ID given after face privacy processing; 3) Re-encode the blanked image (usually using a video image encoder that supports metadata, such as FFmpeg, etc.), and embed the descriptive metadata into the video image data stream (usually the API of the FFmpeg encoder can be used).

[0080] Example: Suppose there are 4 faces in the image X of the lock event ID collected by the intelligent lock A according to step (2). The ID of the segmented face images are face (1), face (2), face (3), and face (4) respectively. After being recognized by the intelligent lock A, face (1) belongs to person (a), face (2) belongs to person (b) by the intelligent lock B, and face (3) belongs to person (c) by the intelligent lock G. When integrating, person (a), person (b), person (c) and their corresponding face image position coordinates are used as descriptive metadata (while face (4) is not recognized as the owner by any intelligent lock, so this face is retained) and embedded into the encoded data.

[0081] (8) The intelligent lock A sends the picture after face privacy processing to the platform: The image of the lock event ID integrated by the intelligent lock A is sent to the platform. After decoding by the platform, the corresponding person ID is displayed in each face segmentation area. The authorized person can query the community personnel list (data items include door lock ID, door lock geographical address, person ID of family members, name, face image collected during registration, etc.) according to the person ID to obtain their identity information.

[0082] The intelligent lock A sends the image of the lock event ID: The intelligent lock sends the image of the lock event ID (already encoded into a video image data stream) to the platform in a unicast form.

[0083] The platform receives and decodes: After receiving the image data stream of the intelligent lock A, the platform performs video image decoding (using a video image decoder that supports metadata), extracts the descriptive metadata, identifies the blank area according to the face image position coordinates in the metadata, and uses image synthesis technology (using the FFmpeg video encoder that supports text overlay via API) to add the corresponding person ID to this blank area. For the faces of non-owner personnel in the video image, they are not segmented and left blank, and the faces can be normally displayed after video image decoding.

[0084] Note: The descriptive metadata embedded during encoding includes the position coordinates of the segmented face images and the person ID number after face privacy processing. After video image decoding, the corresponding area can be located and the person ID number can be overlaid based on this.

[0085] The platform authorizes user to query personnel identity: According to the person ID displayed in the video image, querying the community personnel list (data items include door lock ID, door lock geographical address, person ID of family members, name, face image collected during registration, etc.) can obtain their identity information, such as the name, door lock geographical address, face image collected during registration, etc. corresponding to this person ID.

[0086] Note: At all stages of the entire process of receiving from the smart lock, independent relay gateway, and platform, the identity information of the personnel cannot be known. Only the platform stores the correspondence between the personnel ID and the identity information (the list of community residents). To protect the privacy of the property owners' faces, unauthorized platform users should not query the identity information of the personnel.

[0087] The smart lock collaborates with neighboring smart locks in a multicast manner, removing the face images of the householders and their neighbors from the video images captured and sent to the platform, thus realizing the protection of the personal privacy of the householders and their neighbors. Since the independent relay gateway is responsible for processing, ordinary staff members, the independent relay gateway, and door lock A on the platform without sufficient authorization cannot see which face belongs to which household, ensuring the protection of the face privacy of the property owners.

[0088] It can be seen that smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other smart locks receive the multicast message, they compare the uncompared face images. If the comparison is successful, face privacy processing is performed; smart lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; smart lock A integrates the reply messages and the privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby being able to ensure the protection of the face privacy of the property owners.

[0089] Another embodiment of the present invention provides a system for processing the face privacy of a smart lock. Refer to Figure 3 , the system may include: A detection module 301 for face detection and privacy processing: Smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the uncompared face images are sent to other smart locks in the form of a multicast message for collaborative privacy processing; Collaborative face privacy processing: After other smart locks receive the multicast message, they compare the uncompared face images with the list of family members of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in a unicast form; A processing module 302 for message checking and reminder: Smart lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Image integration and encoding: Smart lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, encodes them to generate a video image data stream containing descriptive metadata; Smart lock A sends the integrated video image data stream to the platform for decoding processing.

[0090] It can be seen that the intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other intelligent locks receive the multicast message, they compare the unmatched face images, and if the comparison is successful, face privacy processing is performed; the intelligent lock A counts the multicast reply messages, and if there are intelligent locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that pass the health check; the intelligent lock A integrates the reply messages and privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby ensuring the protection of the face privacy of the property owners.

[0091] An embodiment of the present invention further provides a storage medium, in which a computer program is stored, and the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0092] Specifically, in this embodiment, the above storage medium can be configured to store a computer program for executing the following steps: S201, Face detection and privacy processing: The intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the unmatched face images are sent to other intelligent locks in the form of a multicast message for collaborative privacy processing; Collaborative face privacy processing: After other intelligent locks receive the multicast message, they compare the unmatched face images with the family member list of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in a unicast form. S202, Message check and reminder: The intelligent lock A counts the multicast reply messages. If there are intelligent locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that pass the health check; Image integration and encoding: The intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; The intelligent lock A sends the integrated video image data stream to the platform for decoding processing.

[0093] It can be seen that the intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other intelligent locks receive the multicast message, they compare the unmatched face images, and if the comparison is successful, face privacy processing is performed; the intelligent lock A counts the multicast reply messages, and if there are intelligent locks that have not replied, a query request is sent to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; the intelligent lock A integrates the reply messages and the privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby ensuring the protection of the face privacy of the property owner.

[0094] An embodiment of the present invention further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0095] Specifically, the above electronic device may further include a transmission device and an input / output device. Among them, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0096] Specifically, in this embodiment, the above processor may be configured to execute the following steps through a computer program: S201, Face detection and privacy processing: The intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the unmatched face images are sent to other intelligent locks in the form of a multicast message for collaborative privacy processing; Collaborative face privacy processing: After other intelligent locks receive the multicast message, they compare the unmatched face images with the family member list of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in a unicast form. S202, Message check and reminder: The intelligent lock A counts the multicast reply messages. If there are intelligent locks that have not replied, a query request is sent to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; Image integration and encoding: The intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; The intelligent lock A sends the integrated video image data stream to the platform for decoding processing.

[0097] It can be seen that the intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other intelligent locks receive the multicast message, they compare the uncompared face images, and if the comparison is successful, face privacy processing is performed; the intelligent lock A counts the multicast reply messages, and if there are intelligent locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; the intelligent lock A integrates the reply messages and the privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, so as to ensure the protection of the face privacy of the property owner.

[0098] The above has described in detail the structure, features and effects of the present invention according to the illustrated embodiments. The above is only the preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the drawings. Any changes made according to the concept of the present invention, or equivalent embodiments modified into equivalent changes, still within the spirit covered by the description and the drawings, shall be within the protection scope of the present invention.

Claims

1. A method for processing face privacy in a smart lock, characterized in that: The method comprises: Face detection and privacy processing: Smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the face images that were not matched are sent to other smart locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other smart locks compare the face images that were not matched with the family member list of this lock. If the comparison is successful, face privacy processing is performed and the processing results are replied in unicast form; Message check and urging: Smart lock A counts multicast reply messages. If there are smart locks that have not responded, it sends a query request to the independent transit gateway; the independent transit gateway performs a health check and sends a urging reply message to the smart locks that have passed the health check; image integration and encoding: Smart lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes and generates a video image data stream containing descriptive metadata; smart lock A sends the integrated video image data stream to the platform for decoding processing.

2. The method according to claim 1, characterized in that The face detection and privacy processing include: Smart lock A detects multiple faces through video images, performs face segmentation and marks the face image ID and location coordinates; Calculate the similarity between the segmented face image and the family member list of this lock. If the similarity exceeds the threshold, replace the face image with the person ID. If the matching fails, the unmatched facial image and facial image ID will be sent to other smart locks in the community in multicast form.

3. The method according to claim 2, characterized in that The collaborative face privacy processing includes: Smart lock B receives the multicast message, extracts the unmatched face image and compares it with the family member list of this lock; If the match is successful, the facial image is replaced by the person ID, and the processing result is replied in unicast form; If the comparison fails, a unicast message will be sent to indicate that the match did not succeed.

4. The method according to claim 3, characterized in that: The message checking and urging include: Smart lock A counts the multicast reply messages and checks whether there are any smart locks that have not responded. If there is a smart lock that has not responded, a query request is sent to the independent transit gateway; The independent transit gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check.

5. The method according to claim 4, characterized in that The image integration and encoding include: Smart lock A integrates the reply messages from other smart locks and the privacy processing results of this lock; Replace the face image with the person ID and leave the panoramic image blank; Generates a video image data stream containing descriptive metadata and embeds the metadata into the encoded data.

6. A system for smart lock face privacy processing, characterized in that: The system comprises: Detection module, used for face detection and privacy processing: Smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, performs face privacy processing; if the comparison fails, sends the unmatched face image to other smart locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other smart locks compare the unmatched face image with the family member list of this lock. If the comparison is successful, perform face privacy processing and reply the processing result in unicast form; Processing module, used for message checking and urging: Smart lock A counts multicast reply messages, and if there are smart locks that have not responded, it sends a query request to the independent transit gateway; the independent transit gateway performs a health check and sends a prompt reply message to the smart locks that have passed the health check; image integration and encoding: Smart lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes and generates a video image data stream containing descriptive metadata; Smart lock A sends the integrated video image data stream to the platform for decoding processing.

7. The system according to claim 6, characterized in that The detection module is specifically used for: Smart lock A detects multiple faces through video images, performs face segmentation and marks the face image ID and location coordinates; Calculate the similarity between the segmented face image and the family member list of this lock. If the similarity exceeds the threshold, replace the face image with the person ID. If the matching fails, the unmatched facial image and facial image ID will be sent to other smart locks in the community in multicast form.

8. The system according to claim 7, characterized in that The detection module is specifically used for: Smart lock B receives the multicast message, extracts the unmatched face image and compares it with the family member list of this lock; If the match is successful, the facial image is replaced by the person ID, and the processing result is replied in unicast form; If the comparison fails, a unicast message will be sent to indicate that the match did not succeed.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 5 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.

Images