Control method, device and equipment of intelligent door lock, storage medium and intelligent door lock

Abstract

The invention provides a control method, device and equipment of an intelligent door lock, a storage medium and the intelligent door lock, the method is applied to the technical field of intelligent home furnishing.The method comprises the steps that a pre-trained deep learning model stored locally is obtained, and the deep learning model is issued after being trained by a server; under the condition that the human body information is detected, obtaining a first video stream; inputting the first video stream into a deep learning model to obtain output biological recognition information and unlocking behavior data; verifying the biological recognition information and the unlocking behavior data according to pre-stored user physiological information and user behavior data; and under the condition that verification is passed, an unlocking signal is generated, and unlocking is controlled according to the unlocking signal. According to the method, the locally stored pre-trained deep learning model is used for verification, the data does not need to be uploaded to the cloud, the problem that the intelligent door lock depends on an external server is solved, and privacy data can be protected.

Description

Technical Field

[0001] The present invention belongs to the field of smart home technology, and in particular relates to a control method, device, equipment, storage medium and smart door lock of a smart door lock. Background Art

[0002] The technologies currently used in smart door lock systems are mainly focused on biometric technologies such as fingerprint recognition, face recognition, password input, and cloud management technology. With the development of edge computing technology, computing power is distributed to terminals, reducing dependence on central servers, processing data faster, and achieving lower latency, while improving security. Deep learning is developing rapidly in the fields of image processing and pattern recognition, and the application of this technology has greatly improved the level of door lock intelligence.

[0003] However, this approach still relies on external servers and pays less attention to the protection of user privacy data. Summary of the invention

[0004] The embodiments of the present invention provide a control method, device, equipment, storage medium and smart door lock for a smart door lock, which can solve the problem that the smart door lock relies on an external server and cannot protect privacy data.

[0005] In a first aspect, the present application provides a control method for a smart door lock, the method comprising:

[0006] Obtain a locally stored pre-trained deep learning model, where the deep learning model is sent by the server after training is completed;

[0007] When human body information is detected, obtaining a first video stream;

[0008] Inputting the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data;

[0009] Verifying the biometric information and the unlocking behavior data according to pre-stored user physiological information and user behavior data;

[0010] When the verification is successful, an unlocking signal is generated and the unlocking is controlled according to the unlocking signal.

[0011] Optionally, after generating an unlocking signal and controlling unlocking according to the unlocking signal when the verification is passed, the method further includes:

[0012] Encrypting the biometric information and the unlocking behavior data and storing them in a local database;

[0013] The pruned deep learning model is updated according to the local database.

[0014] Optionally, the method further comprises:

[0015] Determining the contribution of multiple parameters in the deep learning model by minimum absolute error;

[0016] Parameters whose contribution is less than a preset contribution threshold are regarded as parameters that should be clipped;

[0017] Setting the parameters to be pruned to zero to obtain the pruned deep learning model;

[0018] The underlying calculation function of the pruned deep learning model is modified so that the pruned deep learning model skips calculating the zeroed parameters during operation.

[0019] Optionally, the biometric information includes face information, the unlocking behavior data includes unlocking time and unlocking action data, and the verifying of the biometric information and the unlocking behavior data according to the pre-stored user physiological information and user behavior data includes:

[0020] Matching the facial information with the pre-stored user face to obtain a first matching result;

[0021] Matching the unlocking time, the unlocking action data and the pre-stored behavior data to obtain a second matching result;

[0022] A verification result is determined according to the first matching result and the second matching result.

[0023] Optionally, determining a verification result according to the first matching result and the second matching result includes:

[0024] Determining a weight ratio between the first matching result and the second matching result according to the current ambient light;

[0025] According to the weight ratio, weighted sum of the first matching result and the second matching result is obtained to obtain a target matching degree;

[0026] When the target matching degree is greater than a preset matching degree threshold, determining the verification result as verification passed;

[0027] When the target matching degree is less than or equal to the matching degree threshold, the verification result is determined to be a verification failure.

[0028] Optionally, after verifying the biometric information and the unlocking behavior data according to pre-stored user physiological information and user behavior data, the method further includes:

[0029] Performing anomaly detection on the first video stream using a deep learning model to obtain an output detection result;

[0030] If the detection result is abnormal, generate security data and log information according to the first video stream;

[0031] The encrypted security data and log information are sent to a server, so that the server generates historical records and monitoring data according to the security data and log information.

[0032] Optionally, the performing abnormality detection on the first video stream includes:

[0033] Inputting the first video stream into the deep learning model to obtain output unlocking sound data;

[0034] Get the current ambient temperature;

[0035] When the unlocking behavior data, the unlocking sound data and the ambient temperature meet a preset abnormal condition, the detection result is determined to be abnormal.

[0036] Optionally, the method further comprises:

[0037] Obtaining the unlocking password input by the target object in the password input area of ​​the front panel;

[0038] Determining the number of times the target object inputs the unlocking password;

[0039] When the number of inputs is less than the preset number of times and the unlocking password is consistent with the default password, an unlocking signal is generated according to the received human physiological information of the target object and the coded data.

[0040] In a second aspect, the present application provides a control device for a smart door lock, the device comprising:

[0041] A first acquisition module is used to acquire a pre-trained deep learning model stored locally, where the deep learning model is issued by the server after training is completed;

[0042] A second acquisition module is used to acquire a first video stream when human body information is detected;

[0043] A model application module, used for inputting the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data;

[0044] A verification module, used to verify the biometric information and the unlocking behavior data according to pre-stored user physiological information and user behavior data;

[0045] The unlocking module is used to generate an unlocking signal and control the unlocking according to the unlocking signal when the verification is passed.

[0046] In a third aspect, the present application provides an electronic device comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the control method of the above-mentioned smart door lock when executing the program.

[0047] In a fourth aspect, the present application provides a readable storage medium, which, when the instructions in the readable storage medium are executed by a processor of an electronic device, enables the electronic device to execute the above-mentioned smart door lock control method.

[0048] In a fifth aspect, the present application provides a smart door lock, which includes the control device of the smart door lock as described in the second aspect, or the electronic device as described in the third aspect.

[0049] In the embodiment of the present application, first, obtain the locally stored pre-trained deep learning model that is sent down after training by the server; second, obtain the first video stream and perform human body detection on the first video stream; third, when human body information is detected, input the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data; then, verify the biometric information and unlocking behavior data based on the pre-stored user physiological information and user behavior data; finally, if the verification is passed, generate an unlocking signal and control the unlocking according to the unlocking signal. Through the above technical solution, the locally stored pre-trained deep learning model is used for verification, and there is no need to upload the data to the cloud, which solves the problem of smart door locks relying on external servers and is conducive to protecting privacy data. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 It is a schematic diagram of the architecture of a smart door lock provided in an embodiment of the present application;

[0052] Figure 2 This is a flowchart of a method for controlling a smart door lock provided by an embodiment of the present application;

[0053] Figure 3 It is a flowchart of the specific steps of a control method of a smart door lock provided in an embodiment of the present application;

[0054] Figure 4 is a flowchart of another method for controlling a smart door lock provided by an embodiment of the present application;

[0055] Figure 5 is a structural diagram of a control device for a smart door lock provided in an embodiment of the present application;

[0056] Figure 6 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0057] Figure 7 It is a structural diagram of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the target object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0060] In the description of the present disclosure, unless otherwise specified, "multiple" means two or more than two, and other quantifiers are similar thereto; "at least one item", "one item or multiple items" or similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one item a can represent any number of a; for another example, one item or multiple items among a, b and c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple; "and / or" is a kind of description of the association relationship of associated objects, indicating that there can be three kinds of relationships, for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " indicates that the associated objects before and after are in an "or" relationship.

[0061] Although operations or steps are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood that it is required to perform these operations or steps in the specific order shown or in a serial order, or to perform all the operations or steps shown to obtain the desired results. In the embodiments of the present disclosure, these operations or steps can be performed in series; these operations or steps can also be performed in parallel; or some of these operations or steps can be performed.

[0062] It should be noted that the execution information of memory access instructions, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0063] First, the architecture of a smart door lock involved in an embodiment of the present application is described.

[0064] See also Figure 1 As shown, the smart door lock includes a door lock body 10, an embedded edge computing unit 20, a multimodal sensor 30, a communication module 40 and an alarm 50.

[0065] In an embodiment of the present application, the embedded edge computing unit 20 integrates an Advanced RISC Machines (ARM) processor or a neural network processing unit (NPU) chip for local processing and real-time reasoning.

[0066] In some embodiments, the embedded edge computing unit 20 is used to process local sensor data, video streams and other information, and perform computational analysis in real time without relying on cloud servers.

[0067] In the embodiment of the present application, the multimodal sensor 30 includes a camera, an infrared sensor, an acceleration sensor, a fingerprint recognition sensor, a microphone, etc., which are used for data collection and anomaly detection.

[0068] In the embodiment of the present application, the communication module 40 supports wireless network communication technology (Wireless Fidelity, Wi-Fi), Bluetooth, and cellular network for local and remote communication.

[0069] In the embodiment of the present application, the alarm 50 is used to send an alarm to the user when an abnormality is detected.

[0070] The control method of the smart door lock provided in the embodiment of the present application is described in detail below.

[0071] Figure 2 is a flowchart of a method for controlling a smart door lock provided by an embodiment of the present application. The method for controlling a smart door lock is applied to Figure 1 Smart door locks such as Figure 2 As shown, the method may include:

[0072] Step 101: Obtain a pre-trained deep learning model stored locally.

[0073] In an embodiment of the present application, the deep learning model is issued by the server after training is completed.

[0074] It should be noted that in the embodiment of the present application, the server is used to train the deep learning model and send the updated model to the edge device such as the smart door lock. Optionally, the server is also responsible for the remote management of the system, model update, and log upload and analysis.

[0075] For example, the framework TensorFlow and pre-trained deep learning models are used to achieve efficient face recognition through deep neural network models (Convolutional Neural Networks, CNN). Among them, TensorFlow can train and run deep neural networks, which can be applied in many scenarios, such as image recognition, handwritten digit classification, recursive neural networks, word embedding, natural language processing, video detection, etc. TensorFlow can run on multiple central processing units (CPUs) or graphics processing units (GPUs), and it can also run on mobile operating systems. Its architecture is flexible, has good scalability, and can support various network models.

[0076] Understandably, real-time performance is very important in security systems. With edge computing, data can be processed and responded to directly in local devices such as smart door locks, without uploading data to the cloud and waiting for feedback, which greatly reduces latency. In addition, since edge computing devices can process large amounts of data locally, only critical data or abnormal situations are transmitted to the cloud, which reduces the demand for network bandwidth. Improve offline working capabilities and robustness. There is no need to upload sensitive data to the cloud, which improves the level of user privacy protection.

[0077] In some embodiments, before step 101, the control method of the above-mentioned smart door lock also includes: regularly receiving a deep learning model sent by a server, and the deep learning model is obtained by periodic updates of the server.

[0078] Step 102: When human body information is detected, obtain a first video stream.

[0079] In some embodiments, before step 102, the control method of the above-mentioned smart door lock may include: detecting in real time whether there is a person in front of the smart door lock through a passive infrared (PIR) detector; correspondingly, step 102 includes: when it is determined that there is a person in front of the smart door lock, starting a time of flight (TOF) camera to obtain a first video stream.

[0080] Optionally, the smart door lock may include at least two cameras, namely a direct time of flight (DTOF) camera and an indirect time of flight (ITOF) camera.

[0081] Optionally, the smart door lock may include a microcontroller unit (MCU) chip and an Internet service provider (ISP) chip. The DTOF camera can be deployed on the MCU, and the ITOF camera can be deployed on the ISP chip. The MCU is a low-power device and can be used as the control center of the entire smart door lock. The ISP chip is a high-power device, and the face recognition algorithm configured in the smart door lock generally runs on the ISP chip.

[0082] In some embodiments, after step 102, the control method of the above-mentioned smart door lock may further include: preprocessing the first video stream, wherein the preprocessing includes noise reduction, resolution adjustment, noise filtering, etc.

[0083] Step 103: Input the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data.

[0084] In an embodiment of the present application, biometric information may include: face information, voice information, facial information, fingerprint information, palm print information, palm vein information, retinal information, etc.

[0085] In the embodiment of the present application, the unlocking behavior data may include: unlocking time, unlocking action data, wherein the unlocking action data may include posture when opening the door, habitual actions, etc.

[0086] In some embodiments, before step 103, the control method of the smart door lock may include: extracting an infrared (IR) image from the first video stream. Correspondingly, step 103 may include: using a trained deep learning model to detect facial key points on the IR image, where the facial key points may include positions such as eyes, nose, and mouth corners; using the trained deep learning model to perform behavioral analysis on the IR image to obtain unlocking time, unlocking posture, unlocking action, etc.

[0087] In some embodiments, the control method of the smart door lock may further include: using quantization technology to convert floating point operations into low-precision integer operations. The above technical means can further reduce the computational burden.

[0088] Exemplarily, the deep learning model is a lightweight neural network model, such as MobileNet, TinyYOLO and other lightweight models. Among them, the core principle of MobileNet is to use Depthwise Separable Convolution to build the network. This design significantly reduces the number of parameters and the amount of calculation of the model while maintaining a high accuracy. Tiny YOLO is a lightweight target detection algorithm designed to achieve fast real-time target detection. It is a simplified version of the YOLO algorithm with high detection speed and low computing resource requirements.

[0089] Step 104: Verify the biometric information and unlocking behavior data based on the pre-stored user physiological information and user behavior data.

[0090] In an embodiment of the present application, the user's physiological information may be preset by the user, and the user's behavior data may be obtained by the smart door lock based on historical unlocking data.

[0091] In some embodiments, step 104 may include: matching facial information with pre-stored user faces to obtain a first matching result; matching unlocking time and unlocking action data with pre-stored behavior data to obtain a second matching result; when the first matching result is greater than a first matching degree threshold or the second matching result is greater than a second matching degree threshold, determining that the verification result is verification passed; otherwise, determining that the verification result is verification failed.

[0092] It is understandable that the purpose of verification in step 104 is to determine whether the object in the video stream has the unlocking authority. For example, if the identity information of the object in the video stream belongs to the preset identity information, it is determined that the object in the video stream has the unlocking authority.

[0093] Step 105: If the verification is successful, generate an unlocking signal and control the unlocking according to the unlocking signal.

[0094] In the embodiment of the present application, the unlocking information is used to control the door lock assembly in the door lock body to be in an unlocked state.

[0095] Exemplarily, the door lock assembly includes a mechanical lock core and an electronic lock core. The electronic lock core and the mechanical lock core are arranged on the door body in parallel. The electronic lock core is driven by a lock core drive motor. After the electronic lock core receives an unlocking signal, the lock core drive motor drives the electronic lock core to retract into the door body and fix it.

[0096] It can be understood that unlocking according to the unlocking signal in step 105 means controlling the lock body of the smart door lock to unlock based on the unlocking signal. Among them, the unlocking signal, unlocking instruction, etc. mentioned in this application are all generated based on the premise that the door lock state of the smart door lock is closed, and the embodiment of this application does not limit the specific form of the unlocking signal.

[0097] In summary, in the embodiment of the present application, first, a locally stored pre-trained deep learning model that is sent down after training by a server is obtained; second, a first video stream is obtained and human body detection is performed on the first video stream; third, when human body information is detected, the first video stream is input into the deep learning model to obtain output biometric information and unlocking behavior data; then, the biometric information and unlocking behavior data are verified based on the pre-stored user physiological information and user behavior data; finally, when the verification is passed, an unlocking signal is generated and unlocking is controlled based on the unlocking signal. Through the above technical solution, a locally stored pre-trained deep learning model is used for verification, and there is no need to upload data to the cloud, which solves the problem of smart door locks relying on external servers and is conducive to protecting privacy data.

[0098] Figure 3 This is a flowchart of the specific steps of a control method of a smart door lock provided by an embodiment of the present application, which is applied to Figure 1 Smart door locks such as Figure 3 As shown, the method may include:

[0099] Step 201: Obtain a pre-trained deep learning model stored locally.

[0100] In an embodiment of the present application, the deep learning model is issued by the server after training is completed.

[0101] The method of this step has been described in the aforementioned step 101 and will not be repeated here.

[0102] Step 202: When human body information is detected, obtain a first video stream.

[0103] The method of this step has been described in the aforementioned step 102 and will not be repeated here.

[0104] Step 203: Input the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data.

[0105] In the embodiment of the present application, the biometric information includes facial information, and the unlocking behavior data includes unlocking time and unlocking action data.

[0106] The method of this step has been described in the aforementioned step 103 and will not be repeated here.

[0107] It should be noted that other biometric information such as voice information, facial information, fingerprint information, palm print information, palm vein information, retinal information, etc. can be verified in the following ways, which will not be repeated in the embodiments of this application.

[0108] Step 204: Match the facial information with the pre-stored user face to obtain a first matching result.

[0109] In the embodiment of the present application, the pre-stored user face is obtained when the user registers the face.

[0110] In some embodiments, after a user registers his / her face, the user's face is stored in a local database.

[0111] Optionally, the user's face is encrypted by the Advanced Encryption Standard (AES) and stored in a local non-volatile memory. When it is needed to match the face information for unlocking, the biometric information is decrypted by the pre-stored AES key and matched with the current user information.

[0112] Step 205: Match the unlocking time and unlocking action data with the pre-stored behavior data to obtain a second matching result.

[0113] Optionally, the pre-stored behavior data can be obtained by analyzing historical unlocking data using a deep learning model.

[0114] In some embodiments, after the deep learning model analyzes the historical unlocking data to obtain the behavior data, the behavior data is stored in a local database.

[0115] For example, based on the Long Short Term Memory (LSTM) or Recurrent Neural Network (RNN) model, the user's unlocking habits and behavior patterns are analyzed to obtain behavior data and store it in a local database.

[0116] For example, ensemble learning or other algorithms based on sensor data fusion are used to obtain behavioral data and store them in a local database.

[0117] Step 206: Determine a verification result according to the first matching result and the second matching result.

[0118] In some embodiments, step 206 includes:

[0119] Sub-step 2061: determining a weight ratio between the first matching result and the second matching result according to the current ambient light;

[0120] Sub-step 2062: according to the weight ratio, weighted summing the first matching result and the second matching result to obtain a target matching degree;

[0121] Sub-step 2063: when the target matching degree is greater than a preset matching degree threshold, determining the verification result as verification passed;

[0122] Sub-step 2064: When the target matching degree is less than or equal to the matching degree threshold, determine that the verification result is a verification failure.

[0123] In the embodiment of the present application, different ambient lights correspond to different weight ratios, and the specific weight ratio can be set according to actual needs.

[0124] Optionally, when the current environmental relation is lower than a preset ambient light threshold, the weight of the first matching result is less than the weight of the second matching result.

[0125] In some embodiments, step 206 may also include: determining the verification result based on the first matching result, the second matching result, and the data collected by other sensors. The first matching result, the second matching result, and the data collected by other sensors have different weight ratios. Optionally, the weight corresponding to the data collected by other sensors can be adjusted according to the current ambient light. For example, the other sensors include infrared sensors. In a night environment, the data weight of the infrared sensor increases, and the first matching result and the second matching result decrease. In a high-light environment, the weights of the first matching result and the second matching result increase, and the weight of the infrared sensor decreases.

[0126] In some embodiments, after sub-step 2063, the smart door lock method further includes:

[0127] Sub-step A1: using a deep learning model to perform anomaly detection on the first video stream to obtain an output detection result;

[0128] Sub-step A2: when the detection result is abnormal, generating security data and log information according to the first video stream;

[0129] Sub-step A3: sending the encrypted security data and log information to the server, so that the server generates historical records and monitoring data according to the security data and log information.

[0130] For example, sub-step A1 may include: analyzing the unlocking habits and behavior patterns of the user based on a long short-term memory network (LSTM) or RNN model to determine whether there is any abnormality.

[0131] For another example, sub-step A1 may also include: using ensemble learning or other algorithms based on sensor data fusion to perform anomaly detection.

[0132] In the embodiment of the present application, all transmitted data in sub-step A3 is encrypted via TLS (Transport Layer Security) to ensure communication security.

[0133] Optionally, sub-step A1 includes:

[0134] Sub-step B1: inputting the first video stream into the deep learning model to obtain output unlocking sound data;

[0135] Sub-step B2: obtaining the current ambient temperature;

[0136] Sub-step B3: when the unlocking behavior data, the unlocking sound data and the ambient temperature meet the preset abnormal conditions, determining that the detection result is abnormal.

[0137] In one possible implementation, the abnormal conditions may include: unlocking behavior data indicating prying, knocking actions, and long-term stay of unauthorized personnel; unlocking sound data indicating abnormal knocking sounds and prying sounds; and the ambient temperature being greater than a preset temperature threshold.

[0138] In some embodiments, for remote control scenarios, the door lock data in the local database may need to be synchronized with the cloud server, and the door lock data is encrypted by AES and transmitted to the cloud through an encrypted channel in a specific time period for data backup and behavior analysis. The door lock data may include video surveillance, sensor logs, etc.

[0139] For example, when the preset abnormal conditions are met, the smart door lock triggers an alarm and sends a notification to the user's mobile phone, and the user can remotely view the real-time video and control the door lock. After receiving the abnormal notification through the mobile application, the user can choose to remotely view the real-time video monitoring and view the detailed data of the sensor. The smart door lock responds to the control instructions issued by the server and can realize remote emergency operation functions, such as locking the door lock and triggering the alarm.

[0140] Through the above technical solution, it is possible to conduct in-depth learning and analysis on the collected multimodal data, and realize dynamic judgment of multiple data analyses.

[0141] Optionally, after sub-step 2064, the control method of the above-mentioned smart door lock may further include:

[0142] Sub-step S1, obtaining the unlocking password input by the target object in the password input area of ​​the front panel;

[0143] Sub-step S2, determining the number of times the target object inputs the unlocking password;

[0144] Sub-step S3: when the number of input times is less than the preset number of times and the unlocking password is consistent with the default password, generating an unlocking signal.

[0145] It is understandable that when the biometric information cannot unlock the door, the user can also unlock the door using the password.

[0146] Step 207: If the verification is successful, generate an unlocking signal and control the unlocking according to the unlocking signal.

[0147] The method of this step has been described in the aforementioned step 105 and will not be repeated here.

[0148] Step 208: Encrypt the biometric information and unlocking behavior data and store them in a local database.

[0149] In the embodiment of the present application, all biometric information and sensor data are encrypted and stored locally, and advanced encryption algorithms such as AES-256 can be used. When a user registers biometric information such as fingerprint or face, the data is first encrypted by AES and stored in a local non-volatile memory. When the user unlocks the door, the biometric information is decrypted by the AES key and matched with the current user information. All door lock operation logs are encrypted using the same AES key when stored.

[0150] Step 209: Update the pruned deep learning model according to the local database.

[0151] Among them, model pruning (Pruning) is also called model sparsification. Different from model quantization, which compresses each weight parameter, the sparsification method tries to directly "delete" some weight parameters. The principle of model pruning is to reduce the number of parameters and calculations in the model by removing "unimportant" weights in the model, while trying to ensure that the accuracy of the model is not affected.

[0152] Optionally, step 209 includes:

[0153] Sub-step 2091, determining the contribution of multiple parameters in the deep learning model through the minimum absolute error;

[0154] Sub-step 2092: taking the parameters whose contribution is less than a preset contribution threshold as the parameters to be clipped;

[0155] Sub-step 2093: setting the parameters to be pruned to zero to obtain a pruned deep learning model;

[0156] Sub-step 2094: modify the underlying calculation function of the pruned deep learning model so that the pruned deep learning model skips calculating the zeroed parameters during operation.

[0157] It should be noted that model pruning is an important technology for deep learning model optimization. It reduces redundant computing nodes (such as unimportant weights and neurons), reduces model complexity and inference overhead, and optimizes the performance of deep learning models in door lock systems, especially for models deployed on edge computing units.

[0158] For example, the importance of weights and convolution kernels is evaluated using methods such as the L1 norm, and the contribution threshold is set to obtain parameters with lower contribution to the door lock function. Then, redundant convolution kernels or channels are removed through structured pruning to optimize the model structure to adapt to the computing power limitations of the embedded edge computing unit. After pruning, the system restores the model performance through fine-tuning training, and further combines quantization technology (such as converting floating-point operations to INT8 format) to reduce the computing burden and storage requirements.

[0159] In summary, the control method of the smart door lock in the embodiment of the present application, first of all, can perform deep learning and analysis on the collected multi-modal data, and realize dynamic judgment of multiple data analysis by combining the adaptive threshold adjustment mechanism of deep learning. Secondly, all facial data and behavioral pattern features are stored locally and encrypted by the Advanced Encryption Standard to prevent unauthorized access. Finally, the model pruning technology is used to lightweight the deep learning model, which reduces the computing power performance requirements of the smart door lock and reduces the computing burden and storage requirements of the smart door lock.

[0160] Figure 4 This is a flowchart of another method for controlling a smart door lock provided by an embodiment of the present application, which is applied to Figure 1 Smart door locks such as Figure 4 As shown, the method may include:

[0161] Step 301: Multimodal data collection.

[0162] Step 302: Data preprocessing.

[0163] Step 303: The edge computing module unit processes local data and performs AES algorithm encryption.

[0164] Step 304: Send the encrypted data to the deep learning module.

[0165] Step 305: The deep learning model performs anomaly detection according to the anomaly detection algorithm.

[0166] In summary, the control method of the smart door lock in the embodiment of the present application, multimodal data fusion anomaly detection, combines multimodal sensor data (such as video, sound, vibration, etc.) with deep learning technology, and performs real-time processing and analysis through edge computing to improve the door lock's perception and response capabilities to abnormal situations in complex environments. Offline security functions and multiple data encryption protection mechanisms. The real-time reasoning system of deep learning can continuously learn the user's daily behavior habits and train to improve the accuracy of the model. Adaptive adjustment based on user behavior optimizes the security level of the system.

[0167] Figure 5 is a structural diagram of a control device for a smart door lock provided in an embodiment of the present application. The control device 500 for the smart door lock may include:

[0168] The first acquisition module 501 is used to acquire a pre-trained deep learning model stored locally, where the deep learning model is sent by the server after training is completed;

[0169] The second acquisition module 502 is used to acquire the first video stream when human body information is detected;

[0170] A model application module 503, used to input the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data;

[0171] Verification module 504, used to verify the biometric information and unlocking behavior data based on pre-stored user physiological information and user behavior data;

[0172] The unlocking module 505 is used to generate an unlocking signal and control unlocking according to the unlocking signal when the verification is passed.

[0173] Optionally, the control device 500 of the smart door lock further includes:

[0174] An encryption module, used to encrypt biometric information and unlocking behavior data and store them in a local database;

[0175] The update module is used to update the pruned deep learning model according to the local database.

[0176] Optionally, the control device 500 of the smart door lock further includes:

[0177] A first determination module, used to determine the contribution of multiple parameters in the deep learning model through the minimum absolute error;

[0178] A second determination module is used to take parameters whose contribution is less than a preset contribution threshold as parameters that should be clipped;

[0179] The first processing module is used to set the parameters to be pruned to zero to obtain a pruned deep learning model;

[0180] The underlying calculation function of the pruned deep learning model is modified so that the pruned deep learning model skips calculating the zeroed parameters during operation.

[0181] Optionally, the biometric information includes face information, the unlocking behavior data includes unlocking time and unlocking action data, and the verification module 504 includes:

[0182] A first matching submodule, used to match the face information with a pre-stored user face to obtain a first matching result;

[0183] A second matching submodule, used for matching the unlocking time and unlocking action data with the pre-stored behavior data to obtain a second matching result;

[0184] The determination submodule is used to determine the verification result according to the first matching result and the second matching result.

[0185] Optionally, submodules are determined, including:

[0186] A first determining unit, used to determine a weight ratio between the first matching result and the second matching result according to the current ambient light;

[0187] A calculation unit, configured to perform a weighted summation of the first matching result and the second matching result according to a weight ratio to obtain a target matching degree;

[0188] A second determination unit, configured to determine that the verification result is a verification pass when the target matching degree is greater than a preset matching degree threshold;

[0189] The third determining unit is configured to determine that the verification result is a verification failure when the target matching degree is less than or equal to the matching degree threshold.

[0190] Optionally, the control device 500 of the smart door lock further includes:

[0191] An anomaly detection module, used to perform anomaly detection on the first video stream using a deep learning model to obtain an output detection result;

[0192] A data processing module, used for generating security data and log information according to the first video stream when the detection result is abnormal;

[0193] The sending module is used to send the encrypted security data and log information to the server, so that the server generates historical records and monitoring data based on the security data and log information.

[0194] Optionally, the anomaly detection module includes:

[0195] A model application submodule, used for inputting the first video stream into the deep learning model to obtain output unlocking sound data;

[0196] The acquisition submodule is used to obtain the current ambient temperature;

[0197] The detection submodule is used to determine that the detection result is abnormal when the unlocking behavior data, the unlocking sound data and the ambient temperature meet the preset abnormal conditions.

[0198] Optionally, the control device 500 of the smart door lock further includes:

[0199] The acquisition module is used to acquire the unlocking password input by the target object in the password input area of ​​the front panel;

[0200] The third determination module is used to determine the number of times the target object inputs the unlocking password;

[0201] The second processing module is used to generate an unlocking signal when the number of input times is less than a preset number of times and the unlocking password is consistent with the default password.

[0202] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0203] In summary, the control device of the smart door lock provided in the embodiment of the present application first obtains the locally stored pre-trained deep learning model sent down after training by the server; secondly, obtains the first video stream and performs human body detection on the first video stream; thirdly, when human body information is detected, the first video stream is input into the deep learning model to obtain output biometric information and unlocking behavior data; then, the biometric information and unlocking behavior data are verified according to the pre-stored user physiological information and user behavior data; finally, when the verification is passed, an unlocking signal is generated and the unlocking is controlled according to the unlocking signal. Through the above technical solution, the locally stored pre-trained deep learning model is used for verification, and there is no need to upload the data to the cloud, which solves the problem of smart door locks relying on external servers and is conducive to protecting privacy data.

[0204] The present application also provides an electronic device, see Figure 6, the electronic device 600 may include one or more of the following components: a processing component 602 , a memory 604 , a power component 606 , a multimedia component 608 , an audio component 610 , an input / output (I / O) interface 612 , a sensor component 614 , and a communication component 616 .

[0205] The processing component 602 generally controls the overall operation of the electronic device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 may include one or more processors 620 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 602 may include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 may include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0206] The memory 604 is used to store various types of data to support the operation of the electronic device 600. Examples of such data include instructions for any application or method operating on the electronic device 600, contact data, phone book data, messages, pictures, multimedia, etc. The memory 604 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0207] The power supply component 606 provides power to the various components of the electronic device 600. The power supply component 606 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 600.

[0208] The multimedia component 608 includes an interface that provides an output interface between the electronic device 600 and the user. In some embodiments, the interface may include a liquid crystal display (LCD) and a touch panel (TP). If the interface includes a touch panel, the interface may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. When the electronic device 600 is in an operating mode, such as a shooting mode or a multimedia mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0209] The audio component 610 is used to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC), and when the electronic device 600 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is used to receive an external audio signal. The received audio signal can be further stored in the memory 604 or sent via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0210] The input / output I / O interface 612 provides an interface between the processing component 602 and the peripheral interface module, which may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0211] The sensor assembly 614 includes one or more sensors for providing various aspects of status assessment for the electronic device 600. For example, the sensor assembly 614 can detect the open / closed state of the electronic device 600, the relative positioning of components, such as the display and keypad of the electronic device 600, and the sensor assembly 614 can also detect the position change of the electronic device 600 or a component of the electronic device 600, the presence or absence of user contact with the electronic device 600, the orientation or acceleration / deceleration of the electronic device 600, and the temperature change of the electronic device 600. The sensor assembly 614 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 614 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 614 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0212] The communication component 616 is used to facilitate wired or wireless communication between the electronic device 600 and other devices. The electronic device 600 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 6G or 6G), or a combination thereof. In an exemplary embodiment, the communication component 616 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 616 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0213] In an exemplary embodiment, the electronic device 600 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to implement a control method for a smart door lock provided in an embodiment of the present application.

[0214] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 604 including instructions, and the instructions can be executed by a processor 620 of an electronic device 600 to perform the above method. For example, the non-transitory storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0215] Figure 7 is a block diagram of an electronic device 700 according to another embodiment of the present invention. For example, the electronic device 700 may be provided as a server. Figure 7 , the electronic device 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732, for storing instructions that can be executed by the processing component 722, such as an application. The application stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to execute a control method of a smart door lock provided in an embodiment of the present application.

[0216] The electronic device 700 may also include a power supply component 726 configured to perform power management of the electronic device 700, a wired or wireless network interface 750 configured to connect the electronic device 700 to a network, and an input / output (I / O) interface 757. The electronic device 700 may operate based on an operating system stored in the memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0217] In an embodiment of the present application, the memory 732 may be used to store software programs and various data. The memory 732 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 732 may include a volatile memory or a non-volatile memory, or the memory 732 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 732 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0218] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0219] The present application also provides a readable storage medium. When the instructions in the readable storage medium are executed by a processor of an electronic device, the electronic device can execute the control method of the smart door lock of the aforementioned embodiment.

[0220] An embodiment of the present application also provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the control method embodiment of the smart door lock as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0221] It should be noted that the various information and data obtained in the embodiments of the present application are obtained with the authorization of the information / data holder. All actions of obtaining signals, information or data in the present application are carried out in compliance with the relevant data protection laws and policies of the country of residence and with the authorization of the corresponding device owner.

[0222] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description of the specific language above is for the purpose of disclosing the best mode of implementation of the application.

[0223] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0224] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0225] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0226] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the sorting device according to the present application. The present application can also be implemented as a device or apparatus program for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0227] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0228] The user information (including but not limited to the user's device information, user personal information, etc.) and related data involved in this application are all information authorized by the user or by all parties.

[0229] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0230] The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application. The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A control method for a smart door lock, characterized in that: The method comprises: Obtain a pre-trained deep learning model stored locally, where the deep learning model is sent by the server after training is completed; When human body information is detected, obtaining a first video stream; Inputting the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data; Verifying the biometric information and the unlocking behavior data according to pre-stored user physiological information and user behavior data; When the verification is successful, an unlocking signal is generated and the unlocking is controlled according to the unlocking signal.

2. The method according to claim 1, characterized in that After generating an unlock signal and controlling the unlocking according to the unlock signal when the verification is passed, the method further includes: Encrypting the biometric information and the unlocking behavior data and storing them in a local database; The pruned deep learning model is updated according to the local database.

3. The method according to claim 2, characterized in that The method further comprises: Determining the contribution of multiple parameters in the deep learning model by minimum absolute error; Parameters whose contribution is less than a preset contribution threshold are regarded as parameters that should be clipped; Setting the parameters to be pruned to zero to obtain the pruned deep learning model; The underlying calculation function of the pruned deep learning model is modified so that the pruned deep learning model skips calculating the zeroed parameters during operation.

4. The method according to claim 1, characterized in that: The biometric information includes face information, the unlocking behavior data includes unlocking time and unlocking action data, and the verification of the biometric information and the unlocking behavior data based on the pre-stored user physiological information and user behavior data includes: Matching the facial information with the pre-stored user face to obtain a first matching result; Matching the unlocking time, the unlocking action data and the pre-stored behavior data to obtain a second matching result; A verification result is determined according to the first matching result and the second matching result.

5. The method according to claim 4, characterized in that The determining a verification result according to the first matching result and the second matching result includes: Determining a weight ratio between the first matching result and the second matching result according to the current ambient light; According to the weight ratio, weighted sum of the first matching result and the second matching result is obtained to obtain a target matching degree; When the target matching degree is greater than a preset matching degree threshold, determining the verification result as verification passed; When the target matching degree is less than or equal to the matching degree threshold, the verification result is determined to be a verification failure.

6. The method according to claim 1, characterized in that After verifying the biometric information and the unlocking behavior data according to the pre-stored user physiological information and user behavior data, the method further includes: Performing anomaly detection on the first video stream using a deep learning model to obtain an output detection result; If the detection result is abnormal, generate security data and log information according to the first video stream; The encrypted security data and log information are sent to a server, so that the server generates historical records and monitoring data according to the security data and log information.

7. The method according to claim 4, characterized in that The performing abnormality detection on the first video stream includes: Inputting the first video stream into the deep learning model to obtain output unlocking sound data; Get the current ambient temperature; When the unlocking behavior data, the unlocking sound data and the ambient temperature meet a preset abnormal condition, the detection result is determined to be abnormal.

8. The method according to claim 1, characterized in that The method further comprises: Obtaining the unlocking password input by the target object in the password input area of ​​the front panel; Determining the number of times the target object inputs the unlocking password; When the number of inputs is less than the preset number of times and the unlocking password is consistent with the default password, an unlocking signal is generated according to the received human physiological information of the target object and the coded data.

9. A control device for an intelligent door lock, characterized in that: The device comprises: A first acquisition module is used to acquire a pre-trained deep learning model stored locally, where the deep learning model is issued by the server after training is completed; A second acquisition module is used to acquire the first video stream when human body information is detected; A model application module, used for inputting the first video stream into the deep learning model to obtain output biometric information and unlocking behavior data; A verification module, used to verify the biometric information and the unlocking behavior data according to pre-stored user physiological information and user behavior data; The unlocking module is used to generate an unlocking signal and control the unlocking according to the unlocking signal when the verification is passed.

10. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the control method of the smart door lock as described in any one of claims 1 to 8 when executing the program.

11. A readable storage medium, characterized in that: When the instructions or transactions in the storage medium are executed by the processor of the electronic device, the electronic device is enabled to execute the control method of the smart door lock described in any one of claims 1 to 8.

12. A smart door lock, characterized in that: The smart door lock includes the control device of the smart door lock as claimed in claim 9, or the electronic device as claimed in claim 10.

Images