Automatic feedback method and system of intelligent door lock system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an automatic feedback method and system for an intelligent door lock system, electronic equipment and a storage medium, and belongs to the technical field of smart home. According to the method, the call connection state of target equipment for the intelligent door lock system is monitored, when the call connection state is a non-connection state, a visitor video collected by the intelligent door lock system is obtained, then the visitor video is analyzed by using a video detection technology, and the intention of a visitor is determined; and matching corresponding feedback data according to the intention of the visitor, and sending the feedback data to the intelligent door lock system for playing or displaying. According to the method, under the condition that the visitor cannot contact the householder, the intention of the visitor can be automatically analyzed according to the visitor video of the intelligent door lock system, and then the appropriate feedback data is determined, so that the reply of the householder is automatically fed back to the visitor, and the residence convenience of the householder and the visiting convenience of the visitor are improved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to an automatic feedback method, system, electronic device and storage medium for a smart door lock system. Background Art

[0002] The smart door lock system is an important part of the smart home system. It provides higher security and more functions through the Internet and mobile devices. In the related technology, the smart door lock system includes door locks, doorbells, cameras, voice interaction devices and communication platforms. When a visitor presses the doorbell, the visitor's video can be transferred to the homeowner's terminal. However, if the homeowner's terminal does not connect to the visitor's video, the visitor cannot complete the purpose of the visit, which brings inconvenience to the homeowner and the visitor. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an automatic feedback method, system, electronic device and storage medium for a smart door lock system, which aims to automatically feedback the homeowner's reply to the visitor, thereby improving the convenience of the homeowner's residence and visitors' visits.

[0004] To achieve the above object, one aspect of an embodiment of the present application proposes an automatic feedback method for a smart door lock system, comprising the following steps:

[0005] Monitor the call connection status of the target device to the smart door lock system;

[0006] When the call connection state is an unconnected state, obtaining a visitor video collected by the smart door lock system;

[0007] Analyze the visitor video using video detection technology to determine the visitor's intention;

[0008] The corresponding feedback data is matched according to the visitor's intention, and the feedback data is sent to the smart door lock system for playback or display.

[0009] In some embodiments, monitoring the call connection status of the target device to the smart door lock system includes the following steps:

[0010] Receive access requests from smart door lock systems;

[0011] Matching a corresponding target device according to the incoming request;

[0012] Sending the device number of the target device to the smart door lock system, so that the smart door lock system initiates a call to the target device according to the device number;

[0013] Get the call connection status based on the call feedback of the target device.

[0014] In some embodiments, the use of video detection technology to analyze the visitor video to determine the visitor's intention includes the following steps:

[0015] Using a moving target detection algorithm to detect and segment the visitor video to obtain target object data;

[0016] Feature extraction and feature classification are performed on the target object data to obtain the visitor's intention.

[0017] In some embodiments, extracting and classifying the target object data to obtain the visitor's intention includes the following steps:

[0018] Identify key feature points in the target object data to obtain human motion features;

[0019] Calculate the similarity between the human motion feature and the preset features of each preset action, and determine the preset action matching the human motion feature;

[0020] The corresponding visitor intention is determined according to the preset action.

[0021] In some embodiments, matching corresponding feedback data according to the visitor's intention includes the following steps:

[0022] Determining whether the detection accuracy of the visitor's intention is greater than a first threshold;

[0023] When the accuracy of the visitor's intention is greater than a first threshold, the corresponding feedback data is matched from the intention reply table according to the semantic information of the visitor's intention, wherein the intention reply table is used to store the mapping relationship between each visitor's intention and the preset reply content;

[0024] When the accuracy of the visitor's intention is less than or equal to a first threshold, the first prompt content is determined as feedback data, wherein the first prompt content is used to prompt the visitor to leave a message about the visit intention through the smart door lock system.

[0025] In some embodiments, the automatic feedback method of the smart door lock system further includes the following steps:

[0026] Obtaining the visitor's message audio from the smart door lock system;

[0027] Encoding the audio of the visitor's message to obtain an audio feature graph;

[0028] Matching the audio feature graph with preset audio features to extract key audio features in the audio feature graph;

[0029] Decoding the key audio features to obtain key audio;

[0030] Convert the key audio into text data, and write the text data into a message template to obtain a message text;

[0031] The message text is sent to the target device for display.

[0032] In some embodiments, the automatic feedback method of the smart door lock system further includes the following steps:

[0033] Determine whether the visitor's intention is dangerous;

[0034] When the visitor's intention is dangerous, a start instruction is sent to the alarm device of the smart door lock system, and an alarm message is sent to the security platform.

[0035] To achieve the above purpose, another aspect of the embodiment of the present application provides an automatic feedback system for a smart door lock system, comprising:

[0036] The first module is used to monitor the call connection status of the target device to the smart door lock system;

[0037] The second module is used to obtain the visitor video collected by the smart door lock system when the call connection state is not connected;

[0038] The third module is used to analyze the visitor video using video detection technology to determine the visitor's intention;

[0039] The fourth module is used to match the corresponding feedback data according to the visitor's intention, and send the feedback data to the smart door lock system for playback or display.

[0040] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program implements the method described in the above embodiment when executed by the processor.

[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the above embodiment.

[0042] The automatic feedback method, system, electronic device and storage medium of the smart door lock system proposed in this application monitors the call connection status of the target device to the smart door lock system. When the call connection status is not connected, the visitor video collected by the smart door lock system is obtained, and then the visitor video is analyzed by video detection technology to determine the visitor's intention, and then the corresponding feedback data is matched according to the visitor's intention, and the feedback data is sent to the smart door lock system for playback or display. In the case where the visitor cannot contact the householder, this application can automatically analyze the visitor's intention based on the visitor video of the smart door lock system, and then determine the appropriate feedback data, so as to automatically feedback the householder's reply to the visitor, thereby improving the convenience of the householder's residence and visitors. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of an automatic feedback method for a smart door lock system provided in an embodiment of the present application;

[0044] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0045] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0046] Figure 4 yes Figure 3 Flow chart of step S302 in FIG.

[0047] Figure 5 yes Figure 1 Flow chart of step S104 in FIG.

[0048] Figure 6 is a flow chart of an automatic feedback method for an intelligent door lock system provided by another embodiment of the present application;

[0049] Figure 7 is a flow chart of an automatic feedback method for an intelligent door lock system provided by another embodiment of the present application;

[0050] Figure 8 It is a schematic diagram of an automatic feedback system of a smart door lock system provided in an embodiment of the present application;

[0051] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application.

[0052] Fig.10 This is the overall flow chart of automatic feedback of the smart door lock system provided in the embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] It should be noted that, although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] The embodiments of the present application provide an automatic feedback method, system, electronic device and storage medium for a smart door lock system, which are intended to automatically feedback the householder's reply to the visitor, thereby improving the convenience of the householder's residence and the visitor's visit.

[0057] The automatic feedback method, system, electronic device and storage medium of the smart door lock system provided in the embodiments of the present application are specifically illustrated by the following embodiments. First, the automatic feedback method of the smart door lock system in the embodiments of the present application is described.

[0058] The embodiments of the present application can acquire and process relevant data based on smart home technology. Among them, smart home is a modern residential system that integrates a variety of advanced technologies. It uses these technologies to realize the intelligent, automated and networked management of home facilities, thereby improving the safety, convenience, comfort and artistry of the home, and realizing an environmentally friendly and energy-saving living environment. Smart home technology integrates a variety of technologies, including but not limited to network communication technology, security technology, automatic control technology, Internet of Things technology, environmental perception technology, artificial intelligence technology, and audio and video technology. Based on user needs, the smart home system uses relevant technologies for intelligent development to improve living convenience.

[0059] The automatic feedback method of the smart door lock system provided in the embodiment of the present application relates to the field of smart home technology. The automatic feedback method of the smart door lock system provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the automatic feedback method of the smart door lock system, etc., but is not limited to the above forms.

[0060] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0061] In some embodiments, the automatic feedback method of the smart door lock system of the embodiment of the present application can be applied to a smart home platform. The smart home platform can interact with the smart door lock systems and homeowner devices of each household. The smart door lock system uploads the video or audio data to the smart home platform. The smart home platform processes the data according to the method of the embodiment of the present application and feeds back the data to the homeowner's target device to achieve smart home linkage.

[0062] Figure 1 is an optional flow chart of the automatic feedback method of the smart door lock system provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0063] Step S101, monitoring the call connection status of the target device to the smart door lock system;

[0064] Step S102, when the call connection state is not connected, obtaining the visitor video collected by the smart door lock system;

[0065] Step S103, analyzing the visitor video using video detection technology to determine the visitor's intention;

[0066] Step S104, matching corresponding feedback data according to the visitor's intention, and sending the feedback data to the smart door lock system for playback or display.

[0067] Steps S101 to S104 shown in the embodiment of the present application monitor the call connection status of the target device to the smart door lock system. When the call connection status is not connected, the visitor video collected by the smart door lock system is obtained, and then the visitor video is analyzed using video detection technology to determine the visitor's intention, and then the corresponding feedback data is matched according to the visitor's intention, and the feedback data is sent to the smart door lock system for playback or display. In the case where the visitor cannot contact the householder, the present application can automatically analyze the visitor's intention based on the visitor video of the smart door lock system, and then determine the appropriate feedback data, so as to automatically feedback the householder's reply to the visitor, thereby improving the convenience of the householder's residence and the visitor's visit.

[0068] In step S101 of some embodiments, the smart door lock system refers to a series of communicative devices installed at the door of the homeowner, including but not limited to smart door locks, smart doorbells, monitoring equipment, microphones and voice players, etc. The smart door lock is used to lock the door and provide security protection functions; the smart doorbell is used to remind the homeowner that there is a visitor indoors. The smart doorbell can also be linked with the monitoring equipment. When a visitor presses the doorbell, the monitoring equipment is triggered to collect the visitor's monitoring data; the monitoring equipment can be a camera for shooting visitor video data to monitor the visitor's behavior; the microphone is used to collect the visitor's voice data so that the visitor can talk to the homeowner or leave a message; the voice player is used to play voice from the smart home platform or the homeowner's device.

[0069] The target device refers to the device configured by the householder that can communicate with the smart door lock system. For example, the target device can be the householder's mobile phone, tablet computer, or a fixed screen set at the indoor door. The target device can also be an indoor smart speaker with remote communication function, or a smart speaker with a screen. The fixed screen can generally only be fixed at the door, and it is not possible to answer the call in a timely and convenient manner in the bedroom and kitchen scenes. The mobile terminal APP pushes and pulls up the video. There are also problems such as large push delay and long time to open the mobile phone APP. In this embodiment, the smart door lock system and the smart speaker with a screen are linked. When someone presses the doorbell outside the door, or opens the smart video door lock, or stays at the door, the smart door lock system reports to the smart home platform. The smart home platform can transfer the smart door lock system to the smart speaker, so that the visitor on the smart door lock system side can talk to the householder on the smart speaker side. In addition, the smart home platform can also judge the visitor's intention based on the captured visitor video or recorded voice, and simultaneously transmit the visitor information and visitor intention to the householder's smart speaker.

[0070] The smart home platform can monitor the call connection status of the target device to the smart door lock system. If the target device connects the call request of the smart door lock system, the target device can feedback the connected call connection status to the platform; if the target device rejects or does not respond to the call request of the smart door lock system, the target device can feedback the unconnected call connection status to the platform.

[0071] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S204:

[0072] Step S201, receiving a visit request from the smart door lock system;

[0073] Step S202, matching a corresponding target device according to the incoming request;

[0074] Step S203, sending the device number of the target device to the smart door lock system, so that the smart door lock system initiates a call to the target device according to the device number;

[0075] Step S204, obtaining the call connection status according to the call feedback of the target device.

[0076] In this embodiment, when a visitor presses the doorbell or knocks on the door, the smart door lock system can be triggered to generate a visit request and send the visit request to the smart home platform. After receiving the visit request, the smart home platform matches the corresponding target device configured by the householder from the database according to the system identifier of the smart door lock system carried in the visit request, and then sends the device number sent by the target device to the smart door lock system. The smart door lock system initiates a call to the target device according to the device number.

[0077] In one example, the smart home platform is integrated with a call service function, and the smart door lock system initiates a call request to the smart home platform according to the device number, and the smart home platform links the call request to the target device. If the target device connects to the call request, the connection status information is fed back to the smart home platform, and the smart home platform establishes a call connection between the target device and the smart door lock system. If the target device does not connect to the call request, the unconnected status information is fed back to the smart home platform, and the smart home platform executes steps S102 to S104.

[0078] In another example, the smart home platform generally does not have a call service function. After the smart door lock system obtains the target device number, it initiates a call request to the call platform according to the device number, and the call platform links the call request to the target device. If the target device connects to the call request, the connection status information is fed back to the call platform, and the call platform establishes a call connection between the target device and the smart door lock system. If the target device does not connect to the call request, the disconnection status information is fed back to the call platform, and the call platform then feeds back to the smart home platform. In response to the disconnection status information, the smart home platform executes steps S102 to S104.

[0079] In step S102 of some embodiments, the smart home platform can record the surveillance video of the smart door lock system in real time. After determining that the smart door lock system dials the target device and the target device does not answer the call, the video containing the visitor can be extracted from the surveillance video to obtain the visitor video. The method of extracting the visitor video from the surveillance video can be based on a certain time before and after the doorbell or dialing operation, or it can be extracted by using an image to detect the visitor, or it can be extracted by linking the infrared sensor at the door to detect the visitor.

[0080] In step S103 of some embodiments, video detection technology is an advanced application that combines video images and computer pattern recognition technology, mainly based on computer vision and image processing technology. Video detection technology mainly includes the following steps: obtaining video data through a camera or other video acquisition device, and performing preprocessing, such as adjusting the brightness and contrast of the picture, to improve the accuracy and effect of subsequent processing; extracting features of the preprocessed video, including key features such as color, texture, and shape, and identifying the target in the video according to a defined template or feature library; comparing the extracted video target features with the preset standard features to identify whether the video content contains preset content. The preset content of this embodiment corresponds to a visitor's intention. By analyzing the visitor video using video detection technology, it can be determined whether the visitor video contains preset content, and the visitor's intention is determined based on the existing preset content. Exemplarily, the visitor video may contain the action of the visitor holding an item in his hand, which may correspond to the action of delivering a courier. After receiving the video, the smart home platform regards the video as a sequence of images, dynamically adjusts the frame rate according to the platform performance, analyzes each frame, performs partition detection on human posture, hands, distance from the door lock, etc., and then compares it with the preset model.

[0081] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S302:

[0082] Step S301, using a moving target detection algorithm to detect and segment moving targets in the visitor video to obtain target object data;

[0083] Step S302: extract and classify the target object data to obtain the visitor's intention.

[0084] In this embodiment, moving target detection is an important task in the field of computer vision and image processing, which aims to detect and track moving target objects from video or image sequences. The goal of moving target detection is to accurately locate, track and segment the target object, and determine its position, size, shape and motion state in the video frame or image sequence. The moving target detection algorithm includes foreground detection, which aims to separate the foreground (i.e., the moving target) from the background in the video or image sequence. Foreground detection can use detection methods, optical flow methods or background subtraction methods to identify foreground pixels. After identifying the foreground pixels, an edge detection algorithm can be used to segment the moving target from the original image. This embodiment uses a moving target detection algorithm to perform moving target detection and segmentation on the visitor video, which can identify the visitor in the video image and segment the visitor from the image to obtain the target object data, so that the image data of the target object can be analyzed later, ignoring the influence of background pixels, thereby improving the accuracy and efficiency of subsequent visitor intention recognition.

[0085] In this embodiment, the target object data is a kind of image sequence data that represents the visitor. After obtaining the target object data, the trained neural network model can be used to extract and classify the target object data and output the visitor's intention. The training data set of the neural network model is a large amount of visitor video data marked with visitor intention labels. In one example, the neural network model is a label classification model, and the output of the model can be the probability of the feature of the target object data corresponding to each visitor's intention. The model can also select the visitor's intention with the highest probability for output.

[0086] In this embodiment, the feature types extracted from the target object data may include action features, facial features, item features, and clothing features, etc., and multiple feature types may be used to perform intention recognition. For example, the facial features of the visitor may be compared with the facial features of an acquaintance. If the facial features of the visitor can match the features of the acquaintance, it can be confirmed as a visit intention; if they do not match, the action features can be identified to determine whether the visitor is carrying items. If the visitor is carrying items, the action, item features, and clothing features of the items can be combined to further classify and identify the obvious first intention types such as the intention to visit with gifts, the intention to deliver takeout, the intention to deliver express, or the intention to harm with a weapon; if the visitor is not carrying items, the action features and clothing features can be further identified as the less obvious second intention type such as the courier picking up the item at the door, bending over to peek, or the third intention type that cannot be identified.

[0087] See also Figure 4 In some embodiments, step S302 may include but is not limited to steps S401 to S403:

[0088] Step S401, identifying key feature points in the target object data to obtain human motion features;

[0089] Step S402, calculating similarity between the human body motion feature and the preset features of each preset action, and determining the preset action matching the human body motion feature;

[0090] Step S403, determining the corresponding visitor intention according to the preset action.

[0091] In this embodiment, analysis can be performed based on the motion features of the target object data to determine the corresponding visitor's intention. Specifically, the feature points (i.e., key feature points) of the key parts of the human body (such as the palm, arm, waist) in the target object data can be identified, and the human motion features that characterize the visitor's posture or action can be obtained. Then, the human motion features are calculated for similarity with the preset features of each preset action (such as the action of holding a box, holding a weapon, or bending over, etc.), and the preset action that matches the human motion features is determined. The correspondence between the preset actions and the visitor's intentions is stored in the smart home platform. After determining the preset action that matches the visitor's human motion features, the corresponding visitor's intention can be determined by querying.

[0092] Exemplarily, the human body in the video can be located using foreground-background segmentation, motion detection or pedestrian detection and tracking algorithms, and then the body parts of the human body can be described. The body description is specifically based on sampling of the visitor's body parts, and through a posture estimation method or a depth map posture estimation method, the positions of various parts of the human body, the positions of the joints, and the motion information of the joints are obtained to characterize the behavior. Then, the postures such as whether the person is bending over, whether he is holding or holding objects, and whether he is holding tools close to the door lock can be classified and identified. After extracting features from the video to obtain a feature set, this embodiment can use K-means clustering and a Gaussian mixed model for clustering to achieve classification and identification. The Gaussian mixture model is expressed as The data point x is generated from K different Gaussian distributions, each with its own mean μk, covariance matrix Σk and mixing coefficients (or weights) π k .

[0093] In step S104 of some embodiments, the smart home platform stores the correspondence between the visitor's intention and the feedback data. The feedback data can be configured by the householder. The feedback data is used to characterize the preset reply content. The feedback data can be in audio form, audio plus video form, or text form. For example, if the visitor intends to send a courier, the reply content of the feedback data configured by the householder can indicate the location of the courier. After determining the visitor's intention, the smart home platform can query the corresponding feedback data and then send the feedback data to the smart door lock system. Based on the form of the feedback data, the smart door lock system can select a player for audio playback or text display.

[0094] See also Figure 5 In some embodiments, the step of matching corresponding feedback data according to the visitor's intention in step S104 may include, but is not limited to, steps S501 to S503:

[0095] Step S501, determining whether the detection accuracy of the visitor's intention is greater than a first threshold;

[0096] Step S502: when the accuracy of the visitor's intention is greater than a first threshold, the corresponding feedback data is matched from the intention reply table according to the semantic information of the visitor's intention, wherein the intention reply table is used to store the mapping relationship between each visitor's intention and the preset reply content;

[0097] Step S503, when the accuracy of the visitor's intention is less than or equal to the first threshold, the first prompt content is determined as feedback data, wherein the first prompt content is used to prompt the visitor to leave a message of the visit intention through the smart door lock system.

[0098] In this embodiment, in the method of using a trained neural network model to extract and classify the target object data, calculating the probability of the feature corresponding to each visitor's intention, and outputting the visitor's intention with the highest probability, the accuracy of the output visitor's intention can be measured by its probability. The greater the probability, the greater the accuracy of the analyzed visitor's intention. In the method of calculating the similarity between the human body motion features and the preset features of each preset action, determining the preset action that matches the human body motion features to determine the visitor's intention, the accuracy of the visitor's intention can be measured by the similarity between the features of the matched preset action and the visitor's action features. The greater the similarity, the greater the accuracy of the analyzed visitor's intention.

[0099] In this embodiment, when the accuracy of the visitor's intention is greater than the first threshold, indicating that the visitor has obvious characteristics and the identification result is relatively reliable, the corresponding feedback data is matched from the intention reply table according to the semantic information of the visitor's intention (such as delivering food). The intention reply table is used to store the mapping relationship between each visitor's intention and the reply content preset by the householder.

[0100] In this embodiment, when the accuracy of the visitor's intention is less than the first threshold, indicating that the visitor has no obvious features and the system is actually unable to identify the visitor's intention, the first prompt content is determined as feedback data, wherein the first prompt content is used to prompt the visitor to leave a message of the visit intention through the smart door lock system. For example, if the visit intention cannot be intelligently determined, the first prompt content can be broadcasted by intelligent voice inquiry: "Please explain the reason for your visit", "Please say your name and contact information", etc. Based on the first prompt content, the visitor can feedback the message in audio or text form to the smart platform through the smart door lock system, and the smart home platform can send the message to the homeowner terminal such as the smart speaker with screen.

[0101] See also Figure 6 In some embodiments, the automatic feedback method of the smart door lock system of the embodiment of the present application may also include but is not limited to steps S601 to S606:

[0102] Step S601, obtaining the visitor's message audio from the smart door lock system;

[0103] Step S602, encoding the audio of the visitor's message to obtain an audio feature graph;

[0104] Step S603, matching the audio feature graph with preset audio features to extract key audio features in the audio feature graph;

[0105] Step S604, decoding the key audio features to obtain the key audio;

[0106] Step S605, converting the key audio into text data, and writing the text data into a message template to obtain a message text;

[0107] Step S606, sending the message text to the target device for display.

[0108] In this embodiment, after the smart home platform obtains the visitor message audio from the smart door lock system, it can extract the main content of the visitor message audio (such as the purpose of the visit, name and contact), and then form a brief message text to feed back to the target device for the homeowner to read. Specifically, the visitor message audio is sampled, quantized and feature extracted (such as short-time Fourier transform or Mel spectrum and Mel cepstrum) to obtain an audio feature graph. The audio feature graph is matched with preset audio features (such as audio features of the visit intention of "I am", "contact information" or "delivering express") to extract the key audio features in the audio feature graph. For example, the features that match the latter part of the features of "I am" in the audio feature graph are key audio features, and the preset audio features that match a certain visit intention are key audio features. The extracted key audio features are decoded to restore the key audio fragments in audio form, and then the key audio is converted into text data using speech recognition technology, and the text data is written into a message template to obtain the message text. For example, the message template can be "The visitor is [name], the contact information is [contact number], and the visitor wants [visit intention]". The text data is filled in the corresponding position to obtain the message text, and then the message text is sent to the homeowner's target device for display, so that the homeowner can read the message conveniently.

[0109] In an exemplary embodiment, the smart home platform can convert audio into text analysis. The smart home platform inputs the audio into a pre-trained feature generation model for analysis. The feature similarity information generation model can be a neural network model that generates feature similarity information, or can be a convolutional neural network model with multiple layers connected in series. The feature similarity information can represent the similarity information between the feature semantics corresponding to the adjusted feature map and the feature semantics corresponding to the preset feature map. The semantics of the input audio can be identified according to the preset similarity condition. The preset similarity condition can be the similarity information with the maximum feature similarity information, which is used to determine the feature map range corresponding to the target feature in the input audio feature map, thereby realizing key feature extraction and text conversion.

[0110] See also Figure 7 In some embodiments, the automatic feedback method of the smart door lock system of the embodiment of the present application may also include but is not limited to steps S701 to S702:

[0111] Step S701, determining whether the visitor's intention is dangerous;

[0112] Step S702: When the visitor's intention is dangerous, a start instruction is sent to the alarm device of the smart door lock system, and an alarm message is sent to the security platform.

[0113] In this embodiment, in the process of identifying the visitor's intention, the preset actions for feature matching may include dangerous actions such as holding weapons such as knives, holding other tools close to the door lock, using devices such as hoods to deliberately cover the face, prying the door, marking the door, destroying other items, etc. If a dangerous action is matched, it indicates that the visitor's intention is dangerous. At this time, a safety protection operation can be initiated. The safety protection operation can be the alarm device of the smart door lock system, sending an alarm message to security platforms such as community security and public security.

[0114] Furthermore, corresponding danger information can be determined according to different dangerous actions. Danger information can be represented by a danger coefficient, which can be in the form of a score. For example, the higher the score, the higher the corresponding danger coefficient and the lower the safety factor. Danger information can be used to divide different danger levels, and different response operations can be performed for different danger levels. For example, for a high-risk action such as prying a door, an alarm can be sounded at the door, a call can be made to the user's mobile phone, and an alarm can be reported to the community and property platform; high-risk actions can be directly linked to the public security system for alarm; for general dangerous actions such as marking at the door, the Q&A audio and the above-mentioned personnel image set can be stored in the cloud and the homeowner can be notified.

[0115] According to some embodiments of the present application, a black and white list can be preset on the smart home platform. For example, if a housekeeper is known to be on the white list, the door lock can be directly opened through face recognition and the image and sound entry records can be stored for reference. At the same time, a message is sent to inform the user that the known housekeeper has entered the house. The notification is not limited to mobile phone APP, text messages, phone calls, etc.

[0116] According to some embodiments of this application, please refer to Fig.10 , the overall process of automatic feedback of the smart door lock system in the embodiment of the present application is described as follows:

[0117] The smart door lock system records audio and video of visitors and reports the recorded audio and video to the smart home platform;

[0118] When the smart home platform detects a visitor, it searches for the speaker that the smart door lock system corresponds to and then returns the speaker information to the smart door lock system;

[0119] The smart door lock system connects to the video call platform and calls the target speaker based on the speaker information;

[0120] The video call platform and the speaker are linked to conduct video calls, and monitor the successful connection of the speaker video or the call connection status when no one answers, and feedback the call connection status to the smart home platform;

[0121] Based on the call connection status, the smart home platform identifies and records the visitor's intention through the visitor's video when no one answers the call, searches for feedback data corresponding to the visitor's intention, and sends the feedback data to the smart door lock system. In addition, when the smart home platform identifies dangerous actions of the visitor, it can report to the community platform or property platform.

[0122] See also Figure 8 The embodiment of the present application also provides an automatic feedback system for a smart door lock system, including:

[0123] The first module is used to monitor the call connection status of the target device to the smart door lock system;

[0124] The second module is used to obtain the visitor video collected by the smart door lock system when the call connection state is not connected;

[0125] The third module is used to analyze the visitor video using video detection technology to determine the visitor's intention;

[0126] The fourth module is used to match the corresponding feedback data according to the visitor's intention, and send the feedback data to the smart door lock system for playback or display.

[0127] It can be understood that the contents of the above-mentioned smart door lock system automatic feedback method embodiment are applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above-mentioned smart door lock system automatic feedback method embodiment, and the beneficial effects achieved are also the same as those achieved by the above-mentioned smart door lock system automatic feedback method embodiment.

[0128] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the automatic feedback method of the smart door lock system. The electronic device can be any smart terminal including a tablet computer, a vehicle-mounted computer, etc.

[0129] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0130] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0131] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the automatic feedback method of the smart door lock system in the embodiment of this application;

[0132] Input / output interface 903, used to implement information input and output;

[0133] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0134] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0135] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0136] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned automatic feedback method of the smart door lock system.

[0137] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0139] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0140] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0141] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0142] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0143] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0145] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0147] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0148] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An automatic feedback method for an intelligent door lock system, characterized in that: The following steps are involved: Monitor the call connection status of the target device to the smart door lock system; When the call connection state is an unconnected state, obtaining a visitor video collected by the smart door lock system; Analyze the visitor video using video detection technology to determine the visitor's intention; The corresponding feedback data is matched according to the visitor's intention, and the feedback data is sent to the smart door lock system for playback or display.

2. The automatic feedback method of the smart door lock system according to claim 1, characterized in that: The monitoring target device's call connection status for the smart door lock system includes the following steps: Receive access requests from smart door lock systems; Matching a corresponding target device according to the incoming request; Sending the device number of the target device to the smart door lock system, so that the smart door lock system initiates a call to the target device according to the device number; Get the call connection status based on the call feedback of the target device.

3. The automatic feedback method of the smart door lock system according to claim 1, characterized in that: The method of analyzing the visitor video by using video detection technology to determine the visitor's intention includes the following steps: Using a moving target detection algorithm to detect and segment the visitor video to obtain target object data; Feature extraction and feature classification are performed on the target object data to obtain the visitor's intention.

4. The automatic feedback method of the smart door lock system according to claim 3, characterized in that: The feature extraction and feature classification of the target object data to obtain the visitor's intention includes the following steps: Identify key feature points in the target object data to obtain human motion features; Calculate the similarity between the human body motion feature and the preset features of each preset action, and determine the preset action matching the human body motion feature; The corresponding visitor intention is determined according to the preset action.

5. The automatic feedback method of the intelligent door lock system according to claim 1, characterized in that: The matching of corresponding feedback data according to the visitor's intention includes the following steps: Determining whether the detection accuracy of the visitor's intention is greater than a first threshold; When the accuracy of the visitor's intention is greater than a first threshold, the corresponding feedback data is matched from the intention reply table according to the semantic information of the visitor's intention, wherein the intention reply table is used to store the mapping relationship between each visitor's intention and the preset reply content; When the accuracy of the visitor's intention is less than or equal to a first threshold, the first prompt content is determined as feedback data, wherein the first prompt content is used to prompt the visitor to leave a message about the visit intention through the smart door lock system.

6. The automatic feedback method of the intelligent door lock system according to claim 5, characterized in that: The automatic feedback method of the intelligent door lock system further comprises the following steps: Obtaining the visitor's message audio from the smart door lock system; Encoding the audio of the visitor's message to obtain an audio feature graph; Matching the audio feature graph with preset audio features to extract key audio features in the audio feature graph; Decoding the key audio features to obtain key audio; Convert the key audio into text data, and write the text data into a message template to obtain a message text; The message text is sent to the target device for display.

7. The automatic feedback method of the smart door lock system according to claim 1, characterized in that: The automatic feedback method of the intelligent door lock system further comprises the following steps: Determine whether the visitor's intention is dangerous; When the visitor's intention is dangerous, a start instruction is sent to the alarm device of the smart door lock system, and an alarm message is sent to the security platform.

8. An automatic feedback system for an intelligent door lock system, characterized in that: include: The first module is used to monitor the call connection status of the target device to the smart door lock system; The second module is used to obtain the visitor video collected by the smart door lock system when the call connection state is not connected; The third module is used to analyze the visitor video using video detection technology to determine the visitor's intention; The fourth module is used to match the corresponding feedback data according to the visitor's intention, and send the feedback data to the smart door lock system for playback or display.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 7.

Images