Intelligent lock system and remote control equipment and control method thereof

Abstract

The invention discloses an intelligent lock system and remote control equipment and a remote control method thereof. The smart lock system (100) comprises a lock device (10) and a remote control device (20). The lock device (10) comprises a smart lock (11) and a camera (12) arranged at the smart lock (11). The remote control device (20) comprises: a wireless communication module (21) configured to receive a video stream provided by the camera; a preprocessing module (22) configured to perform object detection on the video stream to obtain loitering objects in the monitoring space of the camera (12); the privacy protection module (23) is configured to determine a target object and a non-target object in the wandering objects based on a sensitivity threshold value so as to take the non-target object in the wandering objects as a privacy protection object to perform fuzzy processing, the sensitivity threshold value comprises a distance threshold value aiming at the distance of the wandering object relative to the camera (12) and a duration threshold value aiming at the wandering duration of the wandering object appearing in the monitoring space of the camera (12); and a user interface (24) configured to be capable of presenting a screen including the target object and the non-target object subjected to the fuzzification processing.

Description

Smart lock system and remote control device and control method thereof Technical Field

[0001] The present invention relates to the technical field of smart locks, and more particularly to a smart lock system and a remote control device and method thereof. Background Art

[0002] Compared with traditional mechanical locks, smart locks (also called electronic locks in some places) are more convenient to use and have many intelligent functions. Therefore, the smart lock market has developed rapidly worldwide.

[0003] Existing smart locks typically include cameras and use the video stream of the surrounding environment captured by the camera to control the smart lock, for example, unlocking, locking, or locking. However, the video stream may contain private information. Existing solutions do not yet exist for identifying and protecting private information in the video stream captured by the camera.

[0004] Summary of the Invention

[0005] In this context, the present invention aims to provide a smart lock solution with privacy protection function.

[0006] According to one embodiment of the present invention, a remote control device for a smart lock system is provided, wherein the smart lock system includes a smart lock and a camera arranged at the smart lock, and the remote control device includes: a wireless communication module configured to receive a video stream provided by the camera; a preprocessing module configured to perform object detection on the video stream to obtain wandering objects in the camera monitoring space; a privacy protection module configured to determine target objects and non-target objects among the wandering objects based on a sensitivity threshold, so as to fuzzy-process the non-target objects among the wandering objects as privacy protection objects, wherein the sensitivity threshold includes a distance threshold for the distance of the wandering object relative to the camera and a duration threshold for the wandering time during which the wandering object appears in the camera monitoring space; and a user interface configured to be able to present a screen containing the target object and the blurred non-target objects.

[0007] In one embodiment, determining target objects and non-target objects among wandering objects based on a sensitivity threshold includes: determining the distance of each wandering object relative to a camera and the duration of its wandering within the camera's monitoring space; determining a wandering object whose determined distance is less than or equal to a distance threshold and whose determined wandering duration is greater than or equal to a duration threshold as a target object; and determining a wandering object whose determined distance is greater than the distance threshold or whose determined wandering duration is less than the duration threshold as a non-target object.

[0008] In one embodiment, the distance threshold and the duration threshold are both preset values.

[0009] In one embodiment, the distance threshold has a preset range and is a value within the preset range set by the user of the smart lock system via the user interface; and the duration threshold has a preset range and is a value within the preset range set by the user of the smart lock system via the user interface.

[0010] In one embodiment, the distance threshold and the duration threshold have a preferred combination, and the preferred combination of the distance threshold and the duration threshold satisfies the following optimization function: y=ax+b,

[0011] Where x is the distance threshold, and the range of x is the domain of the function;

[0012] y is the duration threshold, and the range of y is the value range of the function;

[0013] Both a and b are configurable coefficients, and a>0.

[0014] In one embodiment, one of the distance threshold and the time threshold is set by a user of the smart lock system, and the other threshold adopts a recommended value calculated according to the optimization function for the set threshold or a value obtained by fine-tuning the recommended value.

[0015] In one embodiment, the privacy protection module is further configured to perform high-precision human body segmentation processing on the target object among the wandering objects, so that the target object included in the presented picture is processed with high-precision human body segmentation.

[0016] In one embodiment, the high-precision human body segmentation processing includes: performing human body segmentation on the target object among the wandering objects with the help of a human body segmentation model to distinguish the human body of the target object as the foreground from the background; and performing human body key point detection with the help of a human body pose estimation model to obtain the human body pose of the target object.

[0017] In one embodiment, the human body segmentation model and the human body posture estimation model are both deep neural network models that have been lightweight processed, and wherein the lightweight processing includes: reducing the parameter accuracy of the human body segmentation model and / or the human body posture estimation model; training a small model with a small number of parameters to learn the output distribution of a large model with a large number of parameters; and / or removing neurons with low responsiveness during the model inference process.

[0018] In one embodiment, the remote control device further comprises a behavior modeling module configured to obtain behavior information including the behavior of the target object by analyzing spatial features and temporal features related to the target object in the video stream.

[0019] In one embodiment, the wireless communication module is further configured to send the behavior information of the target object to a smart home system associated with the smart lock system, so as to assist the smart home system in executing smart home control that matches the behavior of the target object.

[0020] In one embodiment, the remote control device further comprises a security protection module configured to perform security detection on the video stream and send an alarm message to the user through the user interface once an abnormal situation is found.

[0021] In one embodiment, the obfuscation process is implemented by one of the following methods:

[0022] -Blurring according to the default blur level and / or default background pattern;

[0023] -Blurring is performed according to the blur level set by the user through the user interface and / or the background pattern set by the user.

[0024] In one embodiment, the remote control device is configured in an electronic product of a user of the smart lock system.

[0025] According to one embodiment of the present invention, a smart lock system is provided, which includes: a lock device, which includes: a lock installed on a door, and a camera and a human body sensor contained in the same device housing as the lock; and a remote control device as described above.

[0026] According to one embodiment of the present invention, a remote control method for a smart lock system is provided, comprising: receiving a video stream captured by a camera of the smart lock system; performing object detection on the video stream to obtain wandering objects in a camera monitoring space; determining target objects and non-target objects among the wandering objects based on sensitivity thresholds, wherein the sensitivity thresholds include a distance threshold for the distance of the wandering object from the camera and a duration threshold for the duration of the wandering object's presence in the camera monitoring space; blurring the non-target objects among the wandering objects as privacy protection objects; and presenting a picture containing the target object and the blurred non-target objects. Optionally, the remote control method is performed by the remote control device described above or the smart lock system described above.

[0027] According to one embodiment of the present invention, a multifunctional camera of a user smart lock system is provided, which includes: a camera configured to provide a video stream of its monitored space; a preprocessing module configured to perform object detection on the video stream to obtain wandering objects in the camera monitored space; a privacy protection module configured to determine target objects and non-target objects among the wandering objects based on a sensitivity threshold, so as to fuzzy-process the non-target objects among the wandering objects as privacy protection objects, wherein the sensitivity threshold includes a distance threshold for the distance of the wandering object relative to the camera and a duration threshold for the wandering time during which the wandering object appears in the camera monitored space; and a behavior modeling module configured to obtain behavior information including the behavior of the target object by analyzing spatial features and temporal features related to the target object in the video stream.

[0028] The foregoing provides a summary of the major aspects of the present invention to facilitate a basic understanding of these aspects. This summary is not intended to describe key or important elements of all aspects of the present invention, nor is it intended to limit the scope of any or all aspects of the present invention. The purpose of this summary is to provide some implementations of these aspects in a simplified form, serving as a prelude to the detailed description that will be provided later. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The technical solution of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings. It should be understood that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention.

[0030] FIG1 is a schematic block diagram of a smart lock system according to an embodiment of the present invention.

[0031] FIG2 is a flow chart of a remote control method for the smart lock system in FIG1 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The inventors discovered that although existing smart lock systems already have multiple intelligent functions, such as remote viewing, remote intercom and anomaly detection, they do not have privacy protection functions such as protecting personal information.

[0033] To address this issue, the present invention proposes a smart lock system with privacy protection. In this embodiment, objects that appear within the camera-monitored area of ​​the smart lock system but merely pass through it are considered non-target objects, and their personal information is protected. Because the personal information of non-target objects is considered private, it is neither appropriate nor appropriate to be clearly presented to users of the smart lock system.

[0034] In the embodiments of the present invention, the user of the smart lock system should be understood as a person who can operate the smart lock system and has certain permissions to customize parameters, especially a person who can set and adjust the sensitivity parameters of the smart lock system. For example, the users of the smart lock system may include: a user of the smart lock system (for example, a person who operates the remote control device of the smart lock system), a developer of the smart lock system (for example, a person who develops the remote control device of the smart lock system), a purchaser of the smart lock system (for example, a person who makes requirements on the performance of the smart lock system and purchases the system), and a central monitoring platform of the smart lock system (for example, a central monitoring platform that monitors and manages multiple smart lock systems at the same time).

[0035] According to embodiments of the present invention, behavioral modeling technology can also be used to obtain behavioral information representing the behavior of a target object, and this behavioral information can be transmitted to a smart home system, thereby assisting the smart home system in executing smart home controls that match the target object's behavior. For example, if the behavioral information indicates that the object has finished exercising and is returning home, the smart home control may include turning on the air conditioner.

[0036] According to an embodiment of the present invention, based on the duration of an object's presence within the camera's monitoring space (i.e., the loitering duration below) and a duration threshold, and the object's distance from the camera and a distance threshold, it is determined whether the object requires further high-precision human segmentation processing or an object that requires fuzzification as a privacy-protected object. In this way, while achieving the conventional functions of the smart lock, it also implements privacy protection functions.

[0037] According to an embodiment of the present invention, an optimization function is designed to assist in providing a preferred combination of distance thresholds and duration thresholds. Furthermore, this optimization function can also be used to recommend a preferred value for one of the two thresholds when the user enters a setting for the other. In this way, both thresholds can be optimized to meet the user's personalized settings while also being as reasonable as possible.

[0038] According to an embodiment of the present invention, a high-precision and lightweight neural network model is used to implement human body segmentation and human body posture estimation of the target object, which has the advantages of fast calculation speed and high calculation result accuracy.

[0039] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0040] 1 shows a smart lock system 100 according to an embodiment of the present invention, which includes a lock device 10 and a remote control device 20. The smart lock system 100 can be used in smart homes, smart buildings, smart hotels, smart construction sites, etc.

[0041] The lock device 10 is set on a door that requires security, such as a door of a residence, a door of a school dormitory, a door of a hotel, a door of an office, and the like. As shown in Figure 1, in one embodiment, the lock device 10 includes: a lock 11, a camera 12, and a human body sensor 13. The lock 11 is installed on the door and has a housing (not shown). The camera 12 can be set in the housing of the lock 11. In addition, the housing of the lock 11 has an opening or a transparent housing portion for exposing the lens of the camera 12. The human body sensor 13 can also be set in the housing of the lock 11. The lock device 10 can also include a battery 14 for providing power to the lock device 10. The battery 14 can be a disposable battery or a rechargeable battery. The lock device 10 can also include a wireless communication unit (not shown) for wirelessly connecting with the remote control device 20 and exchanging information.

[0042] Camera 12 is used to capture the surroundings of the door where lock 11 is installed and generate a video stream. Camera 12 can be an ultra-wide-angle camera. Camera 12 can include one or more cameras. Camera 12 can be implemented as an IoT camera, automatically establishing a connection with remote control device 20 and transmitting the video stream to remote control device 20.

[0043] The human sensor 13, also known as a human inductor, is used to sense the presence of a human around the door where the lock 11 is installed and to issue a trigger signal when a human is detected. This trigger signal can be implemented as a level signal. For example, the human sensor 13 outputs a low-level signal when no human is detected and a high-level signal when a human is detected. The human sensor 13 determines whether a human is detected by, for example, sensing human temperature, human activity, human voice, etc.

[0044] The human body sensor 13 sends the trigger signal to the camera 12. The camera 12 can be configured to normally be in a standby state and be activated to enter a shooting state in response to receiving the trigger signal. Furthermore, if the trigger signal is not received for a predetermined period of time, the camera 12 enters a standby state again and does not enter a shooting state and shoot again until the next trigger signal arrives. In the standby state, the camera 12 does not shoot, and only consumes a small amount of power to maintain a state in which it can be activated at any time. This can greatly save the power consumption of the camera 12, thereby reducing the frequency of replacing or charging the battery 14.

[0045] The remote control device 20 is capable of wirelessly connecting to the lock device 10. The remote control device 20 can be installed in a terminal device used by a user of the smart lock system 100. This terminal device can be, for example, a smartphone, smartwatch, tablet, laptop, or the like. The remote control device 20 receives the video stream from the camera 12, protects the privacy of non-target objects in the video stream, and obtains behavioral information about the target object in the video stream.

[0046] As shown in FIG. 1 , in one embodiment, a remote control device 20 includes a wireless communication module 21 , a pre-processing module 22 , a privacy protection module 23 , a user interface (UI) 24 , a behavior modeling module 25 , and a security protection module 26 .

[0047] The wireless communication module 21 can be implemented with the help of a wireless communication unit of a terminal device (for example, a wireless communication unit of a smartphone), and its communication mode includes one or more of Bluetooth, WiFi, and 5G.

[0048] The user interface 24 may be implemented by means of a display screen of the terminal device, such as a touch screen, which can receive user input and present an interface containing various information to the user.

[0049] The preprocessing module 22, the privacy protection module 23, the behavior modeling module 25 and the security protection module 26 can be implemented in the form of software or hardware or a combination of software and hardware. It should be understood that the naming of the preprocessing module 22, the privacy protection module 23, the behavior modeling module 25 and the security protection module 26 is logical (functional) and is not intended to limit the implementation method or physical location of these modules. One or more of these modules can be further divided into multiple sub-modules. These modules can also be merged into a single module. In one embodiment, these modules can be implemented in the form of APP, mini-programs, web pages, etc. For example, these modules are implemented as APPs installed on the user's smartphone, or mini-programs on the APP, or web pages on the computer, etc.

[0050] In one embodiment, the remote control device 20 is in communication with the smart home system 200 and transmits the target user's behavior information to the smart home system 200 to assist the smart home system 200 in executing smart home control that matches the target user's behavior. The smart lock system 100 can be implemented as part of the smart home system 200.

[0051] FIG2 illustrates a remote control method 200 for a smart lock system 100 according to an embodiment of the present invention. The following describes the method 200 using the remote control device 20 as an example. It is understood that the descriptions above regarding the smart lock system 100 and the remote control device 20 also apply here.

[0052] In block 202 , the wireless communication module 21 receives a video stream. The camera 12 at the lock device 10 captures the surrounding environment and generates the video stream.

[0053] At block 204, the pre-processing module 22 performs object detection on the video stream to obtain wandering objects in the space monitored by the camera 12. Wandering objects should be understood to include all people present in the space monitored by the camera, including, for example, people passing through the space in a hurry (i.e., non-target objects requiring blurring), people who linger in the space for an extended period of time (such objects may not be target objects, but should be considered suspicious objects requiring attention and visualization to the user), people who inadvertently appear in the space (e.g., neighbors across the street, such objects are non-target objects requiring blurring), and people who need to unlock the door to enter (i.e., target objects requiring visualization to the user). The present invention does not limit the specific method for obtaining wandering objects based on the video stream.

[0054] At block 206, the privacy protection module 23 determines a sensitivity threshold. The sensitivity thresholds include a distance threshold for the loitering object relative to the camera and a duration threshold for the duration of the loitering object's presence within the camera's monitoring space. It will be appreciated that the sensitivity thresholds according to embodiments of the present invention must comply with relevant laws and regulations. The following describes some embodiments for determining the sensitivity thresholds (blocks 2061-2064).

[0055] In one embodiment, in block 2061 , the distance threshold and the duration threshold are preset values, which can be stored in the remote control device 20 , for example, in the privacy protection module 23 , and read when needed.

[0056] In another embodiment, in block 2062, both the distance threshold and the duration threshold have a predetermined range and can be set by the user of the smart lock system 100 through the user interface 24 within the range. For example, the user interface may include a slider that can be dragged by the user and / or a text box for the user to enter a specific value, and when the user sets the distance threshold and the duration threshold, the user is constrained by the predetermined range of the distance threshold and the duration threshold, i.e., the user cannot set a threshold outside the predetermined range.

[0057] In another embodiment, in block 2063, the distance threshold and the duration threshold have a preferred combination, that is, the two thresholds have a correlation, and the correlation is reflected as a preferred combination of the two thresholds.

[0058] In one implementation of this embodiment, a certain numerical distance threshold corresponds to one or more duration thresholds or to a certain range of duration thresholds. Similarly, a certain numerical duration threshold corresponds to one or more distance thresholds or to a certain range of distance thresholds. Such a correspondence is preferred in setting the sensitivity threshold.

[0059] In another implementation of this embodiment, the distance threshold and the duration threshold are positively correlated, that is, the smaller the distance threshold, the smaller the duration threshold; conversely, the larger the distance threshold, the larger the duration threshold.

[0060] In another implementation of this embodiment, the preferred combination of the distance threshold and the duration threshold is determined using the following optimization function: y=ax+b,

[0061] Among them, x is the distance threshold, and the range of x is the domain of the function;

[0062] y is the duration threshold, and the range of y is the value range of the function;

[0063] Both a and b are preset coefficients and are adjustable, for example, based on the different environments (homes, schools, hotels, offices, etc.) in which the smart lock is used, the user's behavior, and the user's privacy requirements. The following describes some examples of setting and adjusting coefficients a and b.

[0064] The parameter a may be set to a>0, that is, the larger the distance threshold is set, the larger the corresponding duration threshold (that is, the duration threshold obtained according to the optimization function) will be.

[0065] When the user is more sensitive to the duration threshold y (i.e., the sensitivity is high), this can be achieved by adjusting the coefficient a to a larger value (i.e., increasing the coefficient a). This is because when the coefficient a is a large value, a small change in the distance threshold x will have a significant impact on the duration threshold y. Conversely, when the user is not sensitive to the duration threshold y (i.e., the sensitivity is low), this can be achieved by adjusting the coefficient a to a smaller value (i.e., decreasing the coefficient a). This is because when the coefficient a is a small value, a large change in the distance threshold x will not have a significant impact on the duration threshold y.

[0066] Similarly, different user sensitivities to the distance threshold x can also be achieved by adjusting the coefficient a.

[0067] The coefficient b can be set to the length of time the user can tolerate a stranger who is closest to the smart lock (i.e., the value of y when x is close to 0) staying at the door, or the length of time a stranger who may violate privacy according to laws and regulations stays at the door.

[0068] According to this optimization function, if the distance threshold x is smaller, the duration threshold y is also smaller. This indicates that the closer someone gets to the door, the shorter the acceptable dwell time. If they dwell even slightly longer, the image is displayed to the user, improving user safety. Conversely, if the distance threshold x is larger, the duration threshold y is also larger. This indicates that the farther someone gets from the door, the longer the acceptable dwell time. In this case, the image is displayed to the user only when the dwell time is longer. This setting reduces the number of unnecessary alerts that interrupt the user.

[0069] It will be understood that the above optimization function is merely exemplary. According to an embodiment of the present invention, the preferred combination of the distance threshold and the duration threshold may also be determined in other ways, for example, other forms of optimization functions (e.g., quadratic functions), drawing a curve representing the relationship between the distance threshold and the duration threshold, and making a table containing the relationship between the distance threshold and the duration threshold.

[0070] In yet another embodiment, in box 2064, the distance threshold and the duration threshold are determined based on user settings and according to the recommendation of the above-mentioned optimization function. For example, the user sets one of the duration threshold and the distance threshold through the user interface 24, and then the other threshold calculated according to the optimization function is presented to the user through the user interface 24 as a recommended value corresponding to the threshold set by the user, so that the user can set the other threshold according to the recommended value. In addition, the other threshold can also be implemented as a neighboring value of the recommended value. For example, the user can increase or decrease a bias value based on the recommended value to obtain another threshold. The range of the bias value is predetermined, for example, from zero to a smaller value. The purpose of such a setting is to allow the user to fine-tune the recommended value on the one hand, and to constrain the user from setting the other threshold to deviate too far from the recommendation on the other hand.

[0071] For clarity, the implementation of this embodiment is described using an example in which a user first sets a duration threshold. For example, the user first sets the duration threshold via user interface 24. Next, privacy protection module 23 calculates the distance threshold corresponding to the duration threshold based on the aforementioned optimization function. The calculated distance threshold is then presented to the user via user interface 24 as a recommended distance threshold value, allowing the user to set the distance threshold according to the recommended value.

[0072] At block 208, the privacy protection module 23 determines target objects and non-target objects among the wandering objects based on the sensitivity threshold. Target objects refer to objects that need to be visualized to the user of the smart lock system 100. Target objects include authorized users of the smart lock system 100 (e.g., house users, office workers, etc.). Non-target objects refer to objects that do not need to be visualized to the user of the smart lock system 100, such as objects that need to be blurred for privacy protection. Non-target objects include, for example, people passing through the monitored space of the camera.

[0073] In one embodiment, determining target and non-target objects among wandering objects based on a sensitivity threshold includes: determining the distance of each wandering object from the camera and the duration of its presence within the camera's monitoring space; determining as target objects those whose distance is less than or equal to the distance threshold and whose duration of presence is greater than or equal to the duration threshold; and determining as non-target objects those whose distance is greater than the distance threshold or whose duration of presence is less than the duration threshold. In other words, objects that are closer to the camera and remain there for a longer period of time are determined as target objects. Furthermore, objects that are farther from the camera or that only briefly pass through (flash) the camera's monitoring space are determined as non-target objects.

[0074] In box 210, the privacy protection module 23 performs blurring processing on the determined non-target objects. In one embodiment, the blurring processing may include blurring processing according to a default blurring degree (e.g., 0% to 100%) and / or a default background pattern. In another embodiment, the blurring processing may include blurring processing according to a blurring degree set by the user through the user interface 24 and / or a background pattern set by the user. Here, the user interface 24 may present a plurality of optional background patterns to the user so that the user can select a background pattern from them so as to perform blurring processing according to the background selected by the user. In addition, the user can also import a preferred background pattern by operating the terminal device, thereby using the background pattern imported by the user for blurring processing.

[0075] It is understandable that in the image presented to the user, the target object will be visually presented as the foreground, and the non-target object will be presented as part of the background, that is, blending into the background or slightly protruding from the background.

[0076] In block 212 , the privacy protection module 23 performs high-precision human segmentation processing on the target object among the wandering objects. This processing can be implemented with the help of a deep neural network model.

[0077] In one embodiment, high-precision human segmentation processing includes: using a human segmentation model to segment a target object among wandering objects, thereby distinguishing the target object's body as the foreground from the background; and using a human pose estimation model to detect key human points (e.g., head, shoulders, elbows, wrists, etc.) to obtain the target object's body pose, thereby achieving high-precision human segmentation. This allows for more precise analysis of the human body's position and pose, resulting in higher accuracy than a single human segmentation algorithm.

[0078] In this embodiment, the human body segmentation model and the human body posture estimation model are both deep neural network models and are lightweight. The lightweight processing is intended to reduce the video memory occupied by the model and increase the model reasoning speed, so that the model calculation has the advantages of high precision and fast efficiency. In one implementation of this embodiment, the lightweight processing includes at least one of the following: 1) reducing the parameter accuracy of the human body segmentation model and / or the human body posture estimation model, for example, converting the floating point 32 bits (i.e., float32) into integer 8 / 16 bits (i.e., int8 / int16); 2) through knowledge distillation, for example, training a small model (student model) with a small number of parameters to learn the output distribution of a large model (teacher model) with a large number of parameters; 3) model pruning, for example, removing neurons with low responsiveness during the model reasoning process or removing components and layers with less effect on the results, and sharing parameters. These methods can reduce the model memory occupancy and increase the model reasoning speed. Here, a small model refers to a neural network with a relatively small number of parameters, for example, the number of parameters is on the order of thousands to millions. Examples of small models include YOLO (You Only Look Once), EfficientNet (Efficient Neural Network), and SqueezeNet (a lightweight network using the Fire module for parameter compression). Large models are neural networks with relatively large parameters, for example, ranging from tens of millions to billions or even more. Examples of large models include SAM (Segment Anything Model), DIDO (DETR with Improved deNoising anch Or boxes), and ViT-based (Vision Transformer-based).

[0079] In block 214 , a screen is generated for presentation on the user interface 24 , which includes the target object processed with high-precision human body segmentation and the non-target objects processed with blurring.

[0080] At block 216, the screen generated in block 214 is presented on the user interface 24. This presentation can be based on system settings and can be presented on the user interface 24 at a predetermined time (e.g., every morning or evening) or at a predetermined interval (e.g., every hour). This presentation can also be presented on the user interface 24 in response to a user request. For example, if the user enters a request on the user interface 24 through voice input, key input, or text input to present a screen of information captured by the camera, the screen generated in block 214 will be presented on the user interface 24.

[0081] At block 218, the behavior modeling module 25 obtains behavior information representing the target subject's behavior based on the video stream. Examples of the target subject's behavior include: finishing exercise, returning from an outing, returning home from school, etc. This behavior information includes, for example, the current time, the subject's ID, and the subject's behavior. In one embodiment, the video stream is input into a trained behavior model, which analyzes the spatial and temporal features in the video stream that are relevant to the subject's behavior to derive and output the user's behavior. Furthermore, the high-precision human body segmentation results described above can also be input into the model as auxiliary information to improve the accuracy of the derived subject behavior.

[0082] In box 220, the wireless communication module 21 sends the behavior information of the target object obtained in box 218 to the smart home system 200 associated with the smart lock system 100, so as to assist the smart home system in performing smart home control that matches the behavior of the target object.

[0083] In addition, according to an embodiment of the present invention, after the remote control device 20 receives the video stream via the wireless communication module 21, the security protection module 26 can perform a security check on the video stream. Once an abnormal situation (e.g., an unsafe factor) is detected, an alarm message is issued to the user through the user interface 24. Examples of abnormal situations include: a person following the target object; the target object suddenly fainting; etc.

[0084] In addition, the user can set a customized abnormal situation through the user interface. Once the security protection module 26 detects a user-defined abnormal situation, an alert message is issued to the user through the user interface 24. In one embodiment, the customized abnormal situation may include: the user can edit customized conditions and customized actions in the user interface, which is similar to a regular rule. Once the current scene meets the customized conditions, the customized action is triggered. For example, once the current scene meets the target object A (for example, the target object A is an object of particular interest to the user) appears in the monitoring space of the camera (customized condition), a reminder message is broadcast to the user through the user interface 24, displayed in text, or displayed on the screen (customized action).

[0085] According to one embodiment of the present invention, a multifunctional camera for a smart lock system is provided. The camera has the aforementioned privacy protection function and the ability to obtain behavioral information of a target object. In one implementation, the multifunctional camera can be implemented as comprising: the aforementioned camera 12, the aforementioned preprocessing module 22, the aforementioned privacy protection module 23, and the aforementioned behavior modeling module 25. In this embodiment, the preprocessing module 22, the privacy protection module 23, and the behavior modeling module 25 can be provided in a chip packaged with the camera, thereby implementing the multifunctional camera.

[0086] The present invention also provides a machine-readable storage medium storing executable instructions. When the instructions are executed, one or more processors are enabled to perform the remote control method 200 .

[0087] It will be appreciated that processor can be implemented using electronic hardware, computer software or its any combination.Whether these processors are implemented as hardware or software will depend on specific application and the overall design constraint imposed on the system.As an example, the processor provided in the present invention, any part of processor or any combination of processors can be implemented as microprocessor, microcontroller, digital signal processor (DSP), field programmable gate array (FPGA), programmable logic device (PLD), state machine, gate logic, discrete hardware circuit and other suitable processing components configured for performing the various functions described in this disclosure.The function of the processor provided in the present invention, any part of processor or any combination of processors can be implemented as software performed by microprocessor, microcontroller, DSP or other suitable platforms.

[0088] Although some embodiments have been described above, these embodiments are given by way of example only and are not intended to limit the scope of the invention. The appended claims and their equivalents are intended to cover all modifications, substitutions and changes made within the scope and spirit of the invention.

Claims

1. A remote control device for a smart lock system, the smart lock system comprising a smart lock and a camera arranged at the smart lock, the remote control device comprising: A wireless communication module configured to receive a video stream provided by a camera; a pre-processing module configured to perform object detection on the video stream to obtain wandering objects in the camera monitoring space; a privacy protection module configured to determine target objects and non-target objects among the wandering objects based on a sensitivity threshold, so as to perform fuzzy processing on the non-target objects among the wandering objects as privacy protection objects, wherein the sensitivity threshold includes a distance threshold for the distance of the wandering object relative to the camera and a duration threshold for the wandering duration of the wandering object appearing in the monitoring space of the camera; as well as A user interface is configured to present a screen including a target object and non-target objects that have been blurred.

2. The remote control device according to claim 1, wherein: Determining target objects and non-target objects among wandering objects based on the sensitivity threshold includes: Determine the distance of each wandering object relative to the camera and the wandering time it appears in the camera monitoring space; Determine a wandering object whose determined distance is less than or equal to a distance threshold and whose determined wandering duration is greater than or equal to a duration threshold as a target object; and A wandering object whose determined distance is greater than the distance threshold or whose determined wandering duration is less than the duration threshold is determined as a non-target object.

3. The remote control device according to claim 1 or 2, wherein: The distance threshold and the duration threshold are both preset values.

4. The remote control device according to claim 1 or 2, wherein: The distance threshold has a preset range and is a value within the preset range set by a user of the smart lock system via a user interface; and The duration threshold has a preset range and is a value within the preset range set by the user of the smart lock system via the user interface.

5. The remote control device according to claim 1 or 2, wherein: The distance threshold and the duration threshold have a preferred combination, and the preferred combination of the distance threshold and the duration threshold satisfies the following optimization function: y=ax+b, Where x is the distance threshold, and the range of x is the domain of the function; y is the duration threshold, and the range of y is the value domain of the function; Both a and b are configurable coefficients, and a>0.

6. The remote control device according to claim 5, wherein: One of the distance threshold and the duration threshold is set by a user of the smart lock system, and the other threshold adopts a recommended value calculated according to the optimization function for a set threshold or a value obtained by fine-tuning the recommended value.

7. The remote control device according to any one of claims 1 to 6, wherein the privacy protection module is further configured to perform high-precision human body segmentation processing on the target object among the wandering objects, so that the target object included in the presented picture is processed by high-precision human body segmentation.

8. The remote control device according to claim 7, wherein: The high-precision human body segmentation process comprises: performing human body segmentation on the target object among the wandering objects by means of a human body segmentation model, so as to distinguish the human body of the target object as a foreground from the background; and Human body key point detection is performed with the help of a human body pose estimation model to obtain the human body pose of the target object.

9. The remote control device according to claim 8, wherein: The human body segmentation model and the human body posture estimation model are both lightweight deep neural network models, and The lightweight processing includes: Reduce the parameter accuracy of the human segmentation model and / or the human pose estimation model; Training a small model with a small number of parameters to learn the output distribution of a large model with a large number of parameters; and / or Remove neurons that have low response during model inference.

10. The remote control device according to any one of claims 1 to 9, further comprising a behavior modeling module configured to obtain behavior information including the behavior of the target object by analyzing spatial features and temporal features related to the target object in the video stream.

11. The remote control device according to claim 10, wherein: The wireless communication module is also configured to send the behavior information of the target object to a smart home system associated with the smart lock system, so as to assist the smart home system in executing smart home control that matches the behavior of the target object.

12. The remote control device according to any one of claims 1 to 11, further comprising a security protection module configured to perform security detection on the video stream and send an alarm message to the user through the user interface once an abnormal situation is found.

13. The remote control device according to any one of claims 1 to 12, wherein: Obfuscation is accomplished in one of the following ways: -blurring according to a default blur level and / or a default background pattern; -Blurring is performed according to the blur level set by the user through the user interface and / or the background pattern set by the user.

14. The remote control device according to any one of claims 1 to 13, wherein: The remote control device is configured in an electronic product of a user of the smart lock system.

15. A smart lock system, comprising: A locking device, comprising: a lock mounted on a door, and a device contained in the same device housing as the lock cameras and body sensors in the body; and A remote control device as claimed in any one of claims 1 to 14.

16. A remote control method for a smart lock system, comprising: Receive the video stream collected by the camera of the smart lock system; Perform object detection on the video stream to obtain wandering objects in the camera monitoring space; Determine the target object and the non-target object among the wandering objects based on the sensitivity threshold, wherein the sensitivity threshold includes a distance threshold for the distance of the wandering object relative to the camera and a duration threshold for the wandering time of the wandering object in the camera monitoring space; The non-target objects among the wandering objects are treated as privacy-protected objects and fuzzy processed; as well as A picture containing the target object and non-target objects that have been blurred is presented. Optionally, the remote control method is performed by a remote control device as described in any one of claims 1 to 14 or a smart lock system as described in claim 15.

17. A multifunctional camera for a user smart lock system, comprising: A camera configured to provide a video stream of the space it monitors; a pre-processing module configured to perform object detection on the video stream to obtain wandering objects in the camera monitoring space; a privacy protection module configured to determine target objects and non-target objects among the wandering objects based on a sensitivity threshold, so as to perform fuzzy processing on the non-target objects among the wandering objects as privacy protection objects, wherein the sensitivity threshold includes a distance threshold for the distance of the wandering object relative to the camera and a duration threshold for the wandering duration of the wandering object appearing in the monitoring space of the camera; as well as The behavior modeling module is configured to obtain behavior information including the behavior of the target object by analyzing the spatial features and temporal features related to the target object in the video stream.

Images