Monitoring method, device, equipment, storage medium and computer program product

Abstract

The embodiment of the invention discloses a monitoring method and device, equipment and a storage medium, and the method comprises the steps: obtaining first image information shot by a first camera and second image information shot by a second camera in response to a condition that a visited object is located in a triggering region, the first camera and the second camera are cameras at different shooting angles; the monitoring area corresponding to the second camera is monitored on the basis of the first image information shot by the first camera and the second image information shot by the second camera, and the problem that a monitoring scheme is low in safety due to the fact that an existing intelligent door lock has a monitoring blind area can be solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a monitoring method, apparatus, intelligent device and storage medium. Background Art

[0002] Smart door locks are improved from traditional mechanical locks and are more intelligent and simple in terms of user security, identification, and management. Smart door locks are the executive components of door locks in access control systems. Smart door locks are mainly used in smart homes, smart hotels, smart buildings and other systems.

[0003] With the continuous development and popularization of technology, more and more smart door locks are integrated with cat-eye camera functions. In the current monitoring solution, users can use the cat-eye camera to remotely monitor the situation outside the door. However, due to the limitation of the camera's field of view, there are generally blind spots in the cat-eye camera's field of view, which makes it impossible for users to monitor the blind spots of the cat-eye camera during remote monitoring. Criminals may use the blind spots to pry locks without being captured by the camera, or they may use the blind spots to steal express or takeout in front of the user's door. It can be seen that the current monitoring solution has the problem of low security. Summary of the invention

[0004] Based on this, it is necessary to provide a monitoring method, apparatus, intelligent device and storage medium to address the above technical problems, which can solve the problem that current smart door locks have monitoring blind spots, resulting in low security of monitoring solutions.

[0005] In a first aspect, the present application provides a monitoring method, comprising:

[0006] In response to the visiting object being located in the triggering area, obtaining first image information taken by the first camera and second image information taken by the second camera, wherein the first camera and the second camera are cameras at different shooting angles;

[0007] Based on the first image information taken by the first camera and the second image information taken by the second camera, a monitoring area corresponding to the second camera is monitored.

[0008] In a second aspect, the present application provides a smart door lock, comprising:

[0009] A door lock body, and a control module, a first camera and a second camera arranged on the door lock body, wherein the first camera and the second camera are cameras at different shooting angles, the first camera is used to shoot first image information, and the second camera is used to shoot second image information, and the control module is used to monitor the monitoring area corresponding to the second camera according to the first image information and the second image information.

[0010] In a third aspect, the present application provides a monitoring device, comprising:

[0011] An acquisition module, configured to acquire, in response to the visitor being located in a triggering area, first image information captured by the first camera and second image information captured by the second camera, wherein the first camera and the second camera are cameras at different angles;

[0012] The monitoring module is used to monitor the transfer object of the monitoring area corresponding to the second camera based on the first image information taken by the first camera and the second image information taken by the second camera.

[0013] In some embodiments of the present application, the monitoring module includes:

[0014] a first determining unit, configured to determine whether the first image information contains an object image of the visiting object;

[0015] a second determining unit, configured to determine, if the first image information includes the object image of the visited object, a transferred object existing in the monitoring area corresponding to the second camera according to the second image information captured by the second camera;

[0016] A generating unit is used to generate monitoring information corresponding to the transfer object according to the object image and the second image information.

[0017] In some embodiments of the present application, the generating unit includes:

[0018] a determination subunit, configured to determine a transfer type of the transfer object according to the second image information;

[0019] A first generating subunit, configured to generate a placement reminder corresponding to the transfer object if the transfer type is a placement type;

[0020] The second generating subunit is configured to generate a moving reminder corresponding to the transfer object based on the object image if the transfer type is a moving type.

[0021] In some embodiments of the present application, the generating unit further includes:

[0022] a determination subunit, configured to identify the object identity of the visiting object according to the first image information if there is a transferred object in the monitoring area, and determine the object information of the transferred object according to the second image information;

[0023] The generating subunit is used to generate the target speech according to the object identity and the object information.

[0024] In some embodiments of the present application, the generating subunit is specifically used for:

[0025] If the object identity is a preset identity, generating an object transfer voice of the visiting object according to the object information;

[0026] If the object identity is not a preset identity, an alarm voice of the visiting object is generated according to the object information.

[0027] In some embodiments of the present application, the generating unit is further configured to:

[0028] If it is determined according to the second image information that there is no transferred object in the monitoring area, and the number of visits of the visited object within the preset time period is greater than the preset number, the visited object is marked, and first risk warning information is generated based on the marked visited object;

[0029] or,

[0030] If it is determined according to the second image information that there is no transferred object in the monitored area, and the stay time of the visited object is greater than the preset time, second risk warning information is generated based on the visited object.

[0031] In some embodiments of the present application, the generating unit is further configured to:

[0032] If the transfer type is a placement type, determining the placement time of the transfer object according to the second image information;

[0033] If the placement time of the transferred object is greater than a preset value, an object retention reminder is generated.

[0034] In some embodiments of the present application, a generation module is further included, and the generation module is specifically used to:

[0035] Displaying the second monitoring screen corresponding to the second camera;

[0036] In response to the area generation operation for the second monitoring screen, a monitoring area is generated in the second monitoring screen.

[0037] In a fourth aspect, the present application provides an intelligent device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the monitoring steps of each embodiment of the present application when executing the computer program.

[0038] Fifth aspect:

[0039] The present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the monitoring steps in each embodiment of the present application are implemented.

[0040] The sixth aspect:

[0041] The present application provides a computer program product, including a computer program, which implements the steps of the monitoring method in each embodiment of the present application when executed by a processor.

[0042] Implementing the embodiments of the present application will have the following beneficial effects:

[0043] In response to a visiting object being located in a triggering area, first image information taken by a first camera and second image information taken by a second camera are obtained, and a monitoring area corresponding to the second camera is monitored based on the first image information taken by the first camera and the second image information taken by the second camera. By monitoring the monitoring area corresponding to the second camera through images taken by cameras at different shooting angles, the problem of blind spots in monitoring of current smart door locks can be solved, thereby improving the security of the monitoring solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] in:

[0046] Figure 1 An application environment diagram of the monitoring method provided in the embodiment of the present application;

[0047] Figure 2 A flow chart of a monitoring method provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a monitoring method according to an embodiment of the present invention;

[0049] Figure 4 Another schematic diagram of a monitoring method according to an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of the structure of the smart door lock provided in the embodiment of the present application;

[0051] Figure 6 A schematic diagram of the structure of a monitoring device provided in an embodiment of the present application;

[0052] Figure 7 Another schematic diagram of the structure of the monitoring device provided in the embodiment of the present application;

[0053] Figure 8An internal structure diagram of a smart device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] The monitoring method provided in this application can be applied to Figure 1 In the application environment shown in the figure, Figure 1 A monitoring system is provided, which includes a smart device 100, a gateway 200 communicatively connected to the smart device 100, a router 300, a server 400, and a terminal 500.

[0056] The smart device 100 accesses the gateway 200 in the monitoring system, and communicates with the gateway 200 through its own configured communication module, and is then controlled by the gateway 200. In one embodiment, the smart device 100 accesses the gateway 200 through a local area network path or a wide area network path, and is deployed in the gateway 200. The local area network may include ZIGBEE or Bluetooth, and the wide area network may include 2G / 3G / 4G / 5G / WIFI, etc.

[0057] The gateway 200 establishes a network connection with the terminal 500 or the server 400 through the router 300. In one embodiment, the gateway 200 and the terminal 500 may establish a network connection through a local area network or a wide area network path. The network 200 is connected to interact with the terminal 500, so that the user can control the smart home device connected to the gateway to perform corresponding actions with the help of the terminal 500 or the smart device 100.

[0058] Among them, the smart device 100 can be a smart camera, a smart curtain motor, a smart air conditioner, a smart door lock, a smart light, a smart TV, a smart control panel, a smart printer, a smart sweeper, a smart robot, and can also be an electronic device such as a human body sensor, a door and window sensor, a temperature and humidity sensor, a water immersion sensor, a natural gas alarm, a smoke alarm, a wall switch, a wall socket, a wireless switch, a wireless wall switch, a magic cube controller, etc. equipped with a communication module. The terminal 500 can be a smart phone, a laptop, a personal computer, a tablet computer, a smart control panel or other smart device that can implement network connection, which is not limited here. The server can be implemented as an independent server or a server cluster composed of multiple servers.

[0059] In addition, the present application can also be implemented through hardware circuits or hardware circuits combined with software instructions. Therefore, the implementation of the present application is not limited to any specific hardware circuits, software, or a combination of the two.

[0060] It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

[0061] See also Figure 2 A monitoring method is provided, and the method is applied to an electronic device as an example for explanation. The electronic device may be Figure 1 The method specifically includes the following steps:

[0062] S101. In response to a visiting object being located in a triggering area, obtaining first image information captured by a first camera and second image information captured by a second camera.

[0063] Among them, the first camera and the second camera are cameras at different shooting angles, and the trigger area is an area for activating the monitoring function. When a visiting object (such as a person or an object) enters this trigger area, the system will respond and start to perform a series of monitoring operations. For example, in the scenario of smart door lock monitoring, the trigger area refers to the set area within the monitoring range of the smart door lock camera. When an activity or object enters this area, a series of actions of the monitoring system will be triggered. Exemplarily, detection components such as PIR (Passive Infrared), IR (Infrared), ToF (Time of Flight), RFID (Radio-frequency identification), radar, etc. can be used on the smart door lock to detect the trigger area. When a visiting object such as a living thing or an object enters the trigger area, the detection component can detect the corresponding trigger signal, thereby triggering the first camera and the second camera of the monitoring system to shoot and monitor. When the detection component does not detect a trigger signal, the first camera and the second camera of the monitoring system can be in a sleep state or a power-off state to save power consumption. When the detection component detects that a living thing or an object enters the trigger area, the first camera and the second camera can be woken up / started to shoot.

[0064] When the visiting object enters the trigger area, the first and second cameras of the smart door lock will start to capture image information. The image information can not only be used to determine whether there are transfer objects (such as express delivery or takeout) in the monitoring area, and the status of these objects (placed or moved). It can also be used to identify the identity of the visiting object and the information of the transfer object.

[0065] For example, the user can customize the monitoring area and trigger area through the mobile phone application (APP). The first camera is located on the front of the door lock, facing horizontally towards the area in front of the door, and is used to capture images directly in front of the door. When a person or object enters the trigger area (such as an area that can be sensed by a human body sensing sensor), the first camera will capture the relevant image. The second camera is located at the bottom of the front of the door lock panel, tilted downward, and is used to monitor the area below the door, including the ground area that is usually located in the blind spot of the field of vision. When a person or object enters the trigger area, the human body sensing sensor (such as an ultrasonic sensor) will detect this activity, and at this time, the two cameras will start to capture video or real-time images.

[0066] S102: Based on the first image information captured by the first camera and the second image information captured by the second camera, monitor a monitoring area corresponding to the second camera.

[0067] The first camera is used to capture the main field of view in front of the door, including recognizing faces and monitoring activities in front of the door. The second camera is used to cover the low area in front of the door, such as near the ground, to reduce blind spots. When a visiting object (such as a courier, delivery man or other visitor) enters the monitoring area, the two cameras start working at the same time to capture their respective image information. The monitoring system will analyze the image information captured by the first camera to identify the identity or behavior of the visiting object. At the same time, the monitoring system will also analyze the image information captured by the second camera, especially to monitor whether there are any objects (such as express packages) placed or moved in the ground area in front of the door.

[0068] For example, specifically, the user has set up a monitoring area through the smart door lock APP. This area is the ground part in front of the door, which is used to specifically monitor the placement of couriers and takeouts. When a courier or takeout deliveryman approaches the door lock, the first camera starts working, capturing the image directly in front of the door and identifying the identity or characteristics of the visitor. The second camera works at the same time, focusing on the area below the door, especially the monitoring area set by the user. The first camera captures the image of the courier, and the second camera captures the courier's action of putting down the package. The system records the location of the package and the courier's action. The system combines the image information of the two cameras to identify the event of the courier putting down the package. The main camera recognizes the visitor, and the secondary camera recognizes the placement of the item.

[0069] Therefore, through the images taken by the cameras at different shooting angles, the monitoring area corresponding to the second camera can be monitored, which can solve the problem of blind spots in monitoring of current smart door locks, thereby improving the security of the monitoring solution.

[0070] In some embodiments of the present application, the step of “monitoring a monitoring area corresponding to the second camera based on the first image information captured by the first camera and the second image information captured by the second camera” may specifically include:

[0071] determining whether the first image information includes an object image of a visiting object;

[0072] If the first image information includes an object image of a visited object, determining a transferred object existing in a monitoring area corresponding to the second camera according to the second image information captured by the second camera;

[0073] Monitoring information corresponding to the transfer object is generated according to the object image and the second image information.

[0074] An object image refers to an image captured by a camera that can represent or indicate a specific object (such as a person, vehicle, package, etc.). A transfer object refers to an item that has changed position within the monitoring area. This change may be caused by placement, movement, or removal, such as express packages, takeout, etc. Monitoring information refers to detailed information generated when a transfer object in the monitoring area is detected to have changed position. The monitoring information can clearly indicate the type of event that occurred, such as "item placement", "item movement", or "item removal". In some embodiments of the present application, the monitoring information can be accompanied by relevant image screenshots or video clips as visual evidence of the event. If the event of the transfer object is related to a specific visitor, the monitoring information will include the visitor's identity or feature description.

[0075] For example, the user sets a monitoring area through the APP of the smart door lock. The monitoring area is the ground part in front of the door, which is specifically used to monitor the placement of express and takeaway. When a courier or takeaway approaches the door lock, the first camera (main camera) starts working and captures the image directly in front of the door. The system analyzes these images to identify the identity or characteristics of the visitor. At the same time, the second camera (secondary camera) focuses on the area below the door, especially the monitoring area set by the user, to cover possible blind spots. The first camera captures the image of the courier, and the monitoring system confirms the identity of the visitor through face recognition or other recognition technology. The second camera captures the action of the courier putting down the package, and the monitoring system records the location of the package and the courier's action. The monitoring system combines the image information of the two cameras and identifies the event of the courier putting down the package. This indicates that a transfer object (express package) has changed its state in the monitoring area. The monitoring system generates monitoring information about the transfer object (express package) based on the object image (the image of the courier) and the second image information (the placement action of the package). The monitoring information can be: the type and appearance description of the package, the time and location of the package placement, and the characteristics or identity information of the courier.

[0076] In this way, the surveillance system is able to provide real-time monitoring and notification capabilities, intelligently identifying and responding to events occurring at the door, thereby improving safety and convenience.

[0077] In some embodiments of the present application, the step of “generating monitoring information corresponding to the transfer object according to the object image and the second image information” may specifically include:

[0078] determining a transfer type of the transfer object according to the second image information;

[0079] If the transfer type is a placement type, a placement reminder corresponding to the transfer object is generated;

[0080] If the transfer type is a moving type, a moving reminder corresponding to the transfer object is generated based on the object image.

[0081] Among them, the transfer type refers to the classification of specific movement or change behavior of an object in the monitoring area. The transfer type can include placement type and removal type. The placement type refers to a new event in which the target object is placed in the monitoring area. The monitoring system will analyze the video shot by the first camera and the second camera frame by frame or segment by segment. If it is analyzed that there is no transfer object in the previous video frame in the monitoring area in the video shot by the second camera, but there is a transfer object in the subsequent video frame, then the transfer object is judged to be newly placed, and it can be determined as a placement type; for example, a courier places a package in front of the user's door, and the monitoring system can identify the transfer type of the package as a prevention type; the movement type refers to the change in the position of the object in the monitoring area, that is, the object disappears from the monitoring area. If the monitoring system analyzes that there is a transfer object in the previous video frame in the monitoring area in the video shot by the second camera, but there is no transfer object in the subsequent video frame, then it is judged that the transfer object is taken away or moved, and it can be determined as a movement type, such as the courier taking the package away. Placement reminder is a function in the monitoring system, which is a series of notifications automatically triggered based on the placement events in the monitoring area identified by the monitoring system. Reminders can be sent to users in a variety of ways, including mobile APP push notifications, text messages, emails, etc. Mobile reminder is a notification mechanism. When the monitoring system detects that the position of an object in the monitoring area has changed, the monitoring system will automatically send a reminder to the user.

[0082] For example, specifically, the user sets up a monitoring area through the smart door lock APP to specifically monitor the ground part in front of the door, especially the area used to place express deliveries and takeouts. When the detection component detects that someone is approaching the door lock, the two cameras work at the same time. The first camera identifies the visitor, and the second camera monitors the ground area in front of the door. The system analyzes the image information of the first camera and the second camera to determine whether there is a transfer object and the transfer type of the transfer object. If the monitoring system recognizes that there is a transfer object, and the type of the transfer object is a placement type, a placement reminder is generated. For example: push notification through the APP: "You have a new express delivery placed in front of the door."

[0083] If the monitoring system detects that the same express package has been moved, for example, if it analyzes that the transfer object in the monitoring area disappears, a move reminder is generated. For example, a notification can be pushed through the APP: "The express in front of your door has been moved." At the same time, the location information and movement time of the items before and after the move are provided. At the same time, the system can also capture the visitor of the mobile transfer object based on the first camera, and can also provide a description of the visitor in the move reminder. For example, if the monitoring system analyzes that the transfer object in the monitoring area disappears, the relevant visitors before or after the disappearance of the transfer object can be obtained based on the first image information captured by the first camera, and the corresponding image information can be sent to the APP to notify the user.

[0084] After receiving the reminder, the user can view real-time video or image evidence through the APP, confirm the status of the item, and take corresponding measures.

[0085] In this way, the monitoring system can provide detailed monitoring information and timely feedback to help users understand the dynamics of the area in front of the door, ensure the safety of items, and respond quickly when necessary.

[0086] In some embodiments, when the monitoring system analyzes that the transfer object in the monitoring area disappears, the transfer type of the transfer object can also be determined based on the door lock event triggered by the door lock body. If the type of the transfer object is a retrieval type, no reminder information may be generated. The retrieval type refers to the transfer object in the monitoring area being retrieved by a trusted person. For example, when the transfer object disappears or within a preset time before it disappears, if a door opening event is detected at the door lock end or the first camera recognizes a registered user, the transfer type of the transfer object is considered to be retrieval. This behavior may be a behavior that the user is aware of. Therefore, in order to avoid the APP receiving too much information push, no reminder information may be sent to the APP in this case.

[0087] In some embodiments of the present application, the step of “generating monitoring information corresponding to the transfer object according to the object image and the second image information” may also include:

[0088] If there is a transferred object in the monitoring area, identifying the object identity of the visited object according to the first image information, and determining the object information of the transferred object according to the second image information;

[0089] Generate target speech based on the object identity and object information.

[0090] Among them, object identity refers to the identity information about the visitor or object extracted from the image captured by the camera through image recognition technology; object information refers to detailed information related to the transferred object that moves or changes position within the monitoring area, such as the size, shape, color, type (such as express parcels, takeout, etc.) of the object and its location within the monitoring area.

[0091] The system identifies the identity of the visiting object, such as a courier, a delivery person, or a specific person, based on the image information from the first camera. At the same time, the system determines the specific information of the transferred object (such as a courier package) based on the image information from the second camera, such as the size, shape, and position change of the package. The system combines the identity of the visiting object and the information of the transferred object to generate the corresponding target voice. If the visitor is identified as a courier, the system may generate a voice: "Thank you for your express service, please place the package in the designated area."

[0092] For example, when the courier arrives at the door, the first camera captures his facial image and uses face recognition technology to identify the courier. At the same time, the second camera (secondary camera) detects that the courier places a package in the monitoring area, that is, on the ground in front of the door. The image information of the first camera helps the system identify the identity of the visiting object, such as confirming that the courier belongs to a specific courier company. The image information of the second camera helps the system determine the information of the transferred object (package), such as the size, shape and location of the package. Combined with the identity information of the visiting object and the information of the transferred object, the corresponding target voice is generated. For example: If the courier is identified as an authorized courier company employee, a voice will be generated: "Thank you, courier from XX company, please place the package in a safe place." The generated target voice is played through the speaker of the smart door lock to communicate directly with the courier. At the same time, a notification can also be sent to the user through the APP to inform that a courier has arrived, and provide the courier's identity information and a description of the package.

[0093] In this way, not only can real-time monitoring and notification functions be provided, but also intelligent interaction with visitors can be carried out according to actual conditions to improve user experience and security.

[0094] In some embodiments of the present application, the step of “generating a target speech according to the object identity and object information” may specifically include:

[0095] If the object identity is a preset identity, generating an object transfer voice of the visiting object according to the object information;

[0096] If the object identity is not the preset identity, the alarm voice of the visiting object is sounded according to the object information.

[0097] Among them, object transfer voice is an automatic voice feedback function used to communicate with visitors. When an object transfer behavior is identified in the monitoring area and the visitor is a preset identity (such as a family member, a frequent service staff, etc.), a corresponding voice message will be generated to interact with the visitor. Alarm voice is used to issue a warning when suspicious behavior or unauthorized access is detected. The purpose is to remind users to pay attention to abnormal conditions in front of the door, and it may also serve as a deterrent to potential intruders.

[0098] Users can preset some specific identity information in the smart door lock APP, such as the identity information of family members, frequent couriers or friends. When someone approaches the door, the first camera captures the image and tries to identify the visitor. Then, the captured image is compared with the preset identity information to determine whether the visiting object is the preset identity. If the visiting object is identified as the preset identity, an object transfer voice will be generated based on the transfer object information (such as a courier package) captured by the second camera. For example: Please place the package in the designated area in front of the door. If the visiting object is not identified as the preset identity, the system will generate an alarm voice based on the transfer object information to remind the user or serve as a deterrent.

[0099] For example, specifically, when someone approaches the door, the first camera captures an image and identifies the visitor, such as by comparing the captured image with preset identity information. If the visitor is identified as a preset identity, such as a known courier, an object transfer voice is generated based on the transfer object information (such as a courier package) captured by the second camera. The object transfer voice can be: Please place the courier in the designated area. "If the visitor is not identified as the preset identity, an alarm voice is generated based on the transfer object information, such as: Warning, please leave. The object transfer voice or alarm voice is played through the speaker of the smart door lock to communicate with the visitor or potential intruder.

[0100] In this way, the monitoring system of the smart door lock can intelligently identify activities in front of the door and provide corresponding voice feedback based on the identity of the visitor, while ensuring the safety and convenience of the user.

[0101] In some embodiments of the present application, the monitoring method may further include:

[0102] If it is determined according to the second image information that there is no transferred object in the monitoring area, and the number of visits of the visiting object within the preset time period is greater than the preset number, the visiting object is marked, and the first risk warning information is generated based on the marked visiting object;

[0103] and / or,

[0104] If it is determined based on the second image information that there is no object transferred to the monitored area, and the visiting object's stay time is greater than the preset time, the second risk warning information is output.

[0105] The number of visits refers to the number of times the visiting object appears in the monitoring area within a specific time period. The first risk warning information refers to the warning message generated when the visiting object in the monitoring area exhibits abnormal behavior. The length of stay refers to the length of time the visitor stays in the monitoring area.

[0106] The user pre-sets a preset time period and a visit count threshold. If the number of visits of a certain visitor exceeds the threshold within the preset time period, the visitor is marked, and may be marked as a key focus object. The first risk warning information is generated based on the marking, such as generating a notification to inform the user that someone frequently appears in front of the door, which may pose a security risk.

[0107] Similarly, users can also set a maximum allowed stay time. If the visitor stays in front of the door for longer than the preset time, it indicates a potential safety risk. The monitoring outputs a second risk warning message to remind the user that someone has stayed in front of the door for too long and needs attention.

[0108] This can provide users with a more proactive and intelligent security protection measure, helping them to promptly detect and respond to potential security threats.

[0109] In some embodiments of the present application, the step of “generating monitoring information corresponding to the transfer object according to the object image and the second image information” may also include:

[0110] If the transfer type is a placement type, determining the placement time of the transfer object according to the second image information;

[0111] If the placement time of the transferred object is greater than the preset value, an object retention reminder is generated.

[0112] The placement time refers to the specific time point recorded after an item is placed in the monitoring area. If the transfer type is identified as the placement type, that is, the item is intentionally placed in the monitoring area, the specific time point (i.e., the placement time) when the item is placed can be recorded. If the time counted from the placement time exceeds the preset value set by the user, the system will determine that the item is stranded and generate an object detention reminder, which will include information such as the description of the item, the placement time, and the length of the detention. In some embodiments of the present application, a detention reminder can be sent to the user through a preset notification method (such as APP push, SMS, email, etc.), informing the user that there are items in front of the door that have not been taken away for a long time.

[0113] For example, the user sets a monitoring area in the smart door lock APP, which is specifically used to monitor the ground in front of the door for express delivery and take-out. At the same time, the user sets a preset value, such as if any item is left in front of the door for more than 30 minutes without being taken away, the system should generate a detention reminder. A courier arrives at the door and places a package in the monitoring area. The second camera (secondary camera) captures this behavior. The monitoring system records the specific time when the package is placed. The monitoring system continuously monitors the placement time of the package. Once it exceeds the user-preset 30 minutes, the system determines that the package is stranded and generates a detention reminder, including the image of the package, the placement time, and the time it has been stranded. The monitoring system pushes a notification to the user through the APP: The express package in front of your door has been placed for more than 30 minutes, please pay attention to pick it up.

[0114] Through this mechanism, the smart door lock system can provide users with more proactive and intelligent security protection, helping users to promptly detect and respond to items left in the area in front of the door, and improving the convenience and safety of life.

[0115] In some embodiments of the present application, before the step of “before monitoring the monitoring area corresponding to the second camera”, the monitoring method may further include:

[0116] Display the second monitoring screen corresponding to the second camera;

[0117] In response to the area generation operation for the second monitoring screen, a monitoring area is generated in the second monitoring screen.

[0118] The user can access the monitoring screen through the smart door lock APP or a connected indoor display device (such as a smart TV or visual speaker). The smart door lock APP can display the real-time monitoring screen captured by the second camera for the user to view the specific area in front of the door. The area targeted by the second monitoring screen corresponding to the second camera may be relatively wide, while the area where users usually want to take care of items such as express delivery and takeout is relatively small. For example, the second monitoring screen may include areas such as corridors and neighbor's doorsteps, while the area that users want to focus on is only their own doorstep. Therefore, this application can facilitate users to operate on the APP interface and select a specific part of the second monitoring screen as the monitoring area. Specifically, the user defines an area on the second monitoring screen, which can be selected by dragging the screen, or setting specific coordinate points to determine the boundaries of the area. During the monitoring process, the monitoring system can only monitor the transferred objects within the monitoring area, and ignore the information outside the monitoring area to avoid too much interference information that is irrelevant to the user.

[0119] In some embodiments of the present application, the user can also personalize the detection period of the monitoring system through the smart door lock APP or the connected indoor display device. For example, if the user is usually not at home from 9:00 to 19:00 during the day, the user can set this period as the monitoring period according to their needs, so that the user can remotely monitor the status of items in front of the house. When the user is at home, the monitoring function of the monitoring system can be turned off to reduce the power consumption of the smart door lock.

[0120] In some embodiments of the present application, the smart door lock APP can also regularly count the transfer of items within a preset period, and identify the total amount of item transfer within the period based on all second image information captured by the second camera within the preset period. For example, it can count the user's daily or weekly express delivery and takeout delivery.

[0121] The above is the monitoring process of this application.

[0122] In summary, an embodiment of the present application provides a monitoring method, which, in response to a visiting object being located in a trigger area, obtains first image information taken by a first camera and second image information taken by a second camera, and then monitors a monitoring area corresponding to the second camera based on the first image information taken by the first camera and the second image information taken by the second camera. In the monitoring scheme provided in the present application, the monitoring area corresponding to the second camera is monitored by using images taken by cameras at different shooting angles, which can solve the problem of blind spots in monitoring in current smart door locks, thereby improving the security of the monitoring scheme.

[0123] In order to further understand the monitoring solution of this application, the following takes the monitoring scenario of a smart door lock as an example for further explanation. Figure 3In this solution, the smart door lock will have a main camera, which is mainly responsible for shooting the area directly in front of the door, mainly used to shoot faces in front of the door (user recognition and face unlocking, etc.); and a secondary camera tilted downward along the horizontal front, which is specifically responsible for monitoring the area in front of the door, especially the area below the door without blind spots. At the same time, the downward-tilted camera can be used to realize functions such as item care and express delivery. In the monitoring area set by the secondary camera, when a courier is placed or picked up, a video is captured, and the video and message are pushed to the user's mobile phone. Whenever an object moves in front of the door, the main camera and the secondary camera will be triggered to capture the video, and then the system will analyze the captured video to determine whether a courier has been placed or taken away. Figure 4 As shown. When someone stays in front of the door for too long, the door lock will automatically play the selected automatic reply audio to give feedback (this function can be pre-set on the door lock or APP). At the same time, this cat's eye camera can determine the status of express delivery and takeout in front of the door. If the express is taken away, a voice alarm will be issued (for example: "The package has been taken away, the owner has been notified") and the App will be pushed. The system will analyze the video captured by the cat's eye camera segment by segment. When it is analyzed that a courier is detected at the beginning of the video captured by the secondary camera, and then the courier disappears, it is determined that the courier is taken away. (Or when the courier disappears, and the main camera determines that the courier is taken away from the door lock based on the direction of the person in front of the door when the courier disappears; if the courier disappears, and the door lock detects a door opening event or recognizes the information of a registered user, it is determined that the courier is taken away by the owner, and no notification is sent.

[0124] See also Figure 5 The present application also provides a smart door lock, which includes a door lock body 201, and a control module 202, a first camera 203 and a second camera 204 arranged on the door lock body 201. The first camera 203 and the second camera 204 are cameras at different shooting angles. The first camera 203 is used to shoot first image information, and the second camera 204 is used to shoot second image information. The control module 202 is used to monitor the monitoring area corresponding to the second camera according to the first image information and the second image information. Exemplarily, the first camera 203 is located on the front of the outdoor panel of the door lock body 201, and is horizontally oriented. It is mainly responsible for shooting the area directly in front of the door, and is used to shoot the face in front of the door (for identifying users and face unlocking, etc.). The second camera 204 is located at the bottom of the outdoor panel of the door lock body 201, and is horizontally tilted downward. It can shoot the area on the ground outside the door, and is responsible for monitoring the area in front of the door, especially the area below the door, to realize the functions of item care, receiving express delivery, etc. By combining the shooting images of the two, blind spots in front of the door can be avoided.

[0125] In summary, the embodiment of the present application provides a smart door lock, which can solve the problem of blind spots in monitoring of current smart door locks by combining two cameras with different perspectives to monitor the monitoring area corresponding to the second camera, thereby improving the security of the monitoring solution.

[0126] It should be understood that although Figure 2 , Figure 4 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , Figure 4 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0127] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of the monitoring device provided in the embodiment of the present application. The target detection processing device includes an acquisition module 301 and a monitoring module 302, which can be specifically as follows:

[0128] The acquisition module 301 is used to acquire first image information taken by the first camera and second image information taken by the second camera in response to the visiting object being located in the triggering area.

[0129] The first camera and the second camera are cameras at different angles.

[0130] The monitoring module 302 is used to monitor the transfer object of the monitoring area corresponding to the second camera based on the first image information taken by the first camera and the second image information taken by the second camera.

[0131] In some embodiments of the present application, the monitoring module 302 includes:

[0132] A first determining unit, configured to determine whether the first image information contains an object image of a visiting object;

[0133] a second determining unit, configured to determine, if the first image information includes an object image of a visited object, a transferred object existing in a monitoring area corresponding to the second camera according to the second image information captured by the second camera;

[0134] A generating unit is used to generate monitoring information corresponding to the transfer object according to the object image and the second image information.

[0135] In some embodiments of the present application, the generating unit includes:

[0136] a determination subunit, configured to determine a transfer type of the transfer object according to the second image information;

[0137] A first generating subunit, configured to generate a placement reminder corresponding to the transfer object if the transfer type is a placement type;

[0138] The second generating subunit is used for generating a moving reminder corresponding to the transfer object based on the object image if the transfer type is a moving type.

[0139] In some embodiments of the present application, the generating unit further includes:

[0140] a determination subunit, configured to identify the object identity of the visited object according to the first image information if there is a transferred object in the monitoring area, and determine the object information of the transferred object according to the second image information;

[0141] The generating subunit is used to generate the target speech according to the object identity and object information.

[0142] In some embodiments of the present application, the generating subunit is specifically used for:

[0143] If the object identity is a preset identity, generating an object transfer voice of the visiting object according to the object information;

[0144] If the identity of the object is not the preset identity, the alarm voice of the visiting object is used according to the object information.

[0145] In some embodiments of the present application, the generating unit is further configured to:

[0146] If there is no transferred object in the monitoring area, and the number of visits of the visiting object within the preset time period is greater than the preset number, the visiting object is marked, and the first risk warning information is generated based on the marked visiting object;

[0147] If there is no object transferred in the monitoring area and the visiting object's stay time is greater than the preset time, the second risk warning information is output.

[0148] In some embodiments of the present application, the generating unit is further configured to:

[0149] If the transfer type is a placement type, determining the placement time of the transfer object according to the second image information;

[0150] If the placement time of the transferred object is greater than the preset value, an object retention reminder is generated.

[0151] In some embodiments of this application, please refer to Figure 7 The monitoring device of the present application may further include a generation module 303, which is specifically used to: display a second monitoring screen corresponding to the second camera; and generate a monitoring area in the second monitoring screen in response to an area generation operation for the second monitoring screen.

[0152] In summary, an embodiment of the present application provides a monitoring device, in which the acquisition module 301 obtains first image information taken by a first camera and second image information taken by a second camera in response to a visiting object being located in a trigger area. The monitoring module 302 monitors the monitoring area corresponding to the second camera based on the first image information taken by the first camera and the second image information taken by the second camera. In the monitoring scheme provided in the present application, the monitoring area corresponding to the second camera is monitored by using images taken by cameras at different shooting angles, which can solve the problem of blind spots in monitoring in current smart door locks, thereby improving the security of the monitoring scheme.

[0153] In one embodiment, a smart device is provided. The smart device may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown. The smart device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the smart device is used to provide computing and control capabilities. The memory of the smart device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the smart device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an intention recognition method is implemented. The display screen of the smart device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the smart device can be a touch layer covered on the display screen, or it can be a smart device.

[0154] Those skilled in the art will understand that Figure 8 The structure shown in the figure is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the smart device to which the solution of the present application is applied. A specific smart device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In one embodiment, a smart device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0156] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0157] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a smart device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the smart device performs the steps in the above-mentioned method embodiments.

[0158] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0160] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A monitoring method, characterized in that: The method comprises: In response to the visiting object being located in the triggering area, obtaining first image information taken by the first camera and second image information taken by the second camera, wherein the first camera and the second camera are cameras at different shooting angles; Based on the first image information taken by the first camera and the second image information taken by the second camera, a monitoring area corresponding to the second camera is monitored.

2. The monitoring method according to claim 1, characterized in that: The monitoring of the monitoring area corresponding to the second camera based on the first image information captured by the first camera and the second image information captured by the second camera includes: determining whether the first image information includes an object image of the visited object; If the first image information includes the object image of the visited object, determining the transferred object existing in the monitoring area corresponding to the second camera according to the second image information captured by the second camera; Monitoring information corresponding to the transfer object is generated according to the object image and the second image information.

3. The monitoring method according to claim 2, characterized in that: The step of generating monitoring information corresponding to the transfer object according to the object image and the second image information includes: determining a transfer type of the transfer object according to the second image information; If the transfer type is a placement type, generating a placement reminder corresponding to the transfer object; If the transfer type is a moving type, a moving reminder corresponding to the transfer object is generated based on the object image.

4. The monitoring method according to claim 2, characterized in that: The step of generating monitoring information corresponding to the transfer object according to the object image and the second image information includes: If there is a transfer object in the monitoring area, identifying the object identity of the visiting object according to the object image, and determining the object information of the transfer object according to the second image information; A target voice is generated according to the object identity and the object information.

5. The monitoring method according to claim 4, characterized in that: The generating a target speech according to the object identity and the object information includes: If the object identity is a preset identity, generating an object transfer voice of the visiting object according to the object information; If the object identity is not a preset identity, an alarm voice of the visiting object is generated according to the object information.

6. The monitoring method according to claim 2, characterized in that: The monitoring method further comprises: If it is determined according to the second image information that there is no transferred object in the monitoring area, and the number of visits of the visited object within the preset time period is greater than the preset number, the visited object is marked, and first risk warning information is generated based on the marked visited object; and / or, If it is determined according to the second image information that there is no transferred object in the monitoring area, and the stay time of the visited object is greater than the preset time, second risk warning information is generated based on the visited object.

7. The monitoring method according to claim 3, characterized in that: The step of generating monitoring information corresponding to the transfer object according to the object image and the second image information further includes: If the transfer type is a placement type, determining the placement time of the transfer object according to the second image information; If the placement time of the transferred object is greater than a preset value, an object retention reminder is generated.

8. The method according to any one of claims 1 to 7, characterized in that: Before monitoring the monitoring area corresponding to the second camera, the method includes: Displaying the second monitoring screen corresponding to the second camera; In response to the area generation operation for the second monitoring screen, the monitoring area is generated in the second monitoring screen.

9. A smart door lock, characterized in that: include: A door lock body, and a control module, a first camera and a second camera arranged on the door lock body, wherein the first camera and the second camera are cameras at different shooting angles, the first camera is used to shoot first image information, and the second camera is used to shoot second image information, and the control module is used to monitor the monitoring area corresponding to the second camera according to the first image information and the second image information.

10. A monitoring device, characterized in that: The device comprises: An acquisition module, configured to acquire, in response to the visitor being located in a triggering area, first image information captured by the first camera and second image information captured by the second camera, wherein the first camera and the second camera are cameras at different angles; The monitoring module is used to monitor the transfer object of the monitoring area corresponding to the second camera based on the first image information taken by the first camera and the second image information taken by the second camera.

11. An intelligent device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: A computer program is stored which can be loaded by a processor and execute the monitoring method according to any one of claims 1 to 8.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images