Monitoring video processing method of intelligent door lock and related equipment

Abstract

The invention provides a monitoring video processing method of an intelligent door lock and related equipment, which are applied to the technical field of video processing, and one specific embodiment of the method comprises the following steps: determining object monitoring information of a user based on monitoring object screening information of the user; obtaining feature information of each monitored object in the object monitoring information; and based on the feature information of each monitoring object, performing stream segmentation processing on a monitoring video stream of the monitoring device to obtain a video frame set of each monitoring object. According to the embodiment, video frame sets of different monitoring objects are provided for a user, the user can select and check the video frame set of the corresponding monitoring object according to actual requirements, related video content of the monitoring object needing to be watched can be quickly found, the user does not need to search in a dragging mode, and the user experience is improved. And the time required for finding the related video content of the to-be-monitored object is reduced.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a monitoring video processing method and related equipment for a smart door lock. Background Art

[0002] Smart door locks with monitoring functions are an important part of the smart home security system. Smart door locks combine traditional door lock functions with monitoring technology to provide users with a more convenient and safer entry and exit experience. Smart door locks with monitoring functions can monitor the situation outside the door in real time, and users can view the monitoring screen at any time.

[0003] The inventors have found that after using a smart door lock with a monitoring function, when users look for content of interest in the monitored video, they can currently only search for the video content of interest by dragging the video progress bar. Since the monitored video is very long, the traditional way of searching content requires a lot of time and effort to find the content that the user is interested in. Summary of the invention

[0004] In view of this, an embodiment of the present application provides a surveillance video processing method and related equipment for a smart door lock. The scheme provided in the embodiment of the present application is applied to process the surveillance video stream to obtain a collection of video frames of surveillance objects of concern to multiple users. This makes it convenient for users to find relevant video data of the surveillance objects they are concerned about in the surveillance video stream, thereby reducing the time it takes for users to find the video content of the surveillance objects in the surveillance video stream.

[0005] To achieve the above objectives, the present application provides the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a monitoring video processing method of a smart door lock, comprising:

[0007] Determine the object monitoring information of the user based on the monitoring object screening information of the user, wherein the object monitoring information includes each monitoring object that the user is concerned about; the monitoring object screening information is the object selection information input by the user in the monitoring object interface, or is the monitoring viewing record of the user;

[0008] Acquire characteristic information of each monitored object in the object monitoring information;

[0009] Based on the characteristic information of each of the monitored objects, the monitoring video stream of the monitoring device is segmented to obtain a set of video frames for each of the monitored objects.

[0010] In some embodiments, the above method, optionally, determining the user's object monitoring information based on the user's monitoring object screening information includes:

[0011] When the monitoring object screening information is object selection information input by the user in the monitoring object interface, generating object monitoring information based on each monitoring object in the object selection information;

[0012] When the monitoring object screening information is the monitoring viewing record of the user, obtaining the user's preferred video from historical monitoring videos based on the monitoring viewing record;

[0013] The respective video objects in the preferred video are identified, monitoring objects are screened in the respective video objects, and object monitoring information is generated based on the screened monitoring objects.

[0014] In some embodiments, in the above method, optionally, the screening of monitoring objects in each of the video objects includes:

[0015] For each of the video objects, based on the monitored viewing records and the preferred videos, determine the weight of the video object, and based on the weight of the video object, determine the user's estimated liking value for the video object;

[0016] A monitoring object is selected from each of the video objects based on the estimated likeness value of each of the video objects.

[0017] In some embodiments, the above method optionally includes performing stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each of the monitored objects to obtain a set of video frames of each of the monitored objects, including:

[0018] For each of the monitored objects, extracting each first video frame containing the monitored object from the monitored video stream based on the feature information of the monitored object;

[0019] Uploading the characteristic information of each monitored object and the monitoring video stream of the monitoring device to the cloud to obtain cloud video data of each monitored object fed back by the cloud, wherein the cloud video data of each monitored object includes each second video frame corresponding to the characteristic information of the monitored object extracted by the cloud from the monitoring video stream;

[0020] For each of the monitored objects, deduplication processing is performed on each of the first video frames and each of the second video frames of the monitored object to obtain each deduplicated video frame, and each video frame set of the monitored object is generated based on each deduplicated video frame.

[0021] In some embodiments, the above method, optionally, generating a set of video frames of the monitored object based on the deduplicated video frames includes:

[0022] Determine each video frame after deduplication as a target video frame;

[0023] Determining a timestamp of each of the target video frames;

[0024] Based on the timestamps of the target video frames, each of the target video frames is divided to obtain at least one video frame group, and the video frame groups are combined into a video frame set of the monitored object; the video frame group contains at least one target video frame, and the target video frames in the video frame group are sorted in the order of timestamps, and the time interval between adjacent target video frames is less than a preset duration.

[0025] In some embodiments, the above method may further include:

[0026] Displaying the video frame sets of each of the monitored objects to the user;

[0027] When receiving a viewing command fed back by the user, determining a target video frame set from each of the video frame sets based on the viewing command;

[0028] Determine a target video frame group in the target video frame set using the viewing command;

[0029] Each video frame in the target video frame group is output to the user.

[0030] In a second aspect, an embodiment of the present application provides a monitoring video processing device for a smart door lock, comprising:

[0031] A first acquisition unit is used to determine the object monitoring information of the user based on the monitoring object screening information of the user, wherein the object monitoring information includes each monitoring object that the user is concerned about; the monitoring object screening information is the object selection information input by the user in the monitoring object interface, or is the monitoring viewing record of the user;

[0032] A second acquisition unit, configured to acquire feature information of each monitored object in the object monitoring information;

[0033] The segmentation unit is used to perform stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each of the monitored objects to obtain a set of video frames for each of the monitored objects.

[0034] In some embodiments, in the above device, optionally, the first acquisition unit executes a process of acquiring the user's object monitoring information, including:

[0035] Obtaining the monitoring viewing record of the user;

[0036] Based on the monitoring viewing record, obtaining the user's preferred video from historical monitoring videos;

[0037] The respective video objects in the preferred video are identified, monitoring objects are screened in the respective video objects, and object monitoring information is generated based on the screened monitoring objects.

[0038] In some embodiments, in the above device, optionally, the first acquisition unit executes a process of acquiring the user's object monitoring information, including:

[0039] Displaying a monitoring object selection interface to the user;

[0040] Acquire object selection information input by the user on the monitoring object selection interface;

[0041] Object attention information is generated based on the object selection information.

[0042] In some embodiments, in the above device, optionally, the first acquisition unit performs a process of screening the monitoring object in each of the video objects, including:

[0043] For each of the video objects, based on the monitored viewing records and the preferred videos, determine the weight of the video object, and based on the weight of the video object, determine the user's estimated liking value for the video object;

[0044] A monitoring object is selected from each of the video objects based on the estimated likeness value of each of the video objects.

[0045] In some embodiments, the above-mentioned device, optionally, the segmentation unit performs stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each of the monitored objects to obtain a set of video frames for each of the monitored objects, including:

[0046] For each of the monitored objects, extracting each first video frame containing the monitored object from the monitored video stream based on the feature information of the monitored object;

[0047] Uploading the characteristic information of each monitored object and the monitoring video stream of the monitoring device to the cloud to obtain cloud video data of each monitored object fed back by the cloud, wherein the cloud video data of each monitored object includes each second video frame corresponding to the characteristic information of the monitored object extracted by the cloud from the monitoring video stream;

[0048] For each of the monitored objects, deduplication processing is performed on each of the first video frames and each of the second video frames of the monitored object to obtain each deduplicated video frame, and each video frame set of the monitored object is generated based on each deduplicated video frame.

[0049] In some embodiments, the above device, optionally, the segmentation unit performs a process of generating a set of video frames of the monitored object based on the deduplicated video frames, including:

[0050] Determine each video frame after deduplication as a target video frame;

[0051] Based on a preset video frame grouping rule, each of the target video frames is grouped to obtain a plurality of video frame groups, wherein the video frame group includes at least one target video frame.

[0052] In some embodiments, the above device may further include:

[0053] A display unit, used for displaying the video frame sets of each of the monitored objects to the user;

[0054] A first determining unit, configured to determine a target video frame set from each of the video frame sets based on the viewing command when receiving a viewing command fed back by the user;

[0055] a second determining unit, configured to determine a target video frame group in the target video frame set using the viewing command;

[0056] An output unit is used to output each video frame in the target video frame group to the user.

[0057] In a third aspect, an embodiment of the present application provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the monitoring video processing method as described above.

[0058] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a memory, and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to execute the monitoring video processing method as described above.

[0059] In a fifth aspect, an embodiment of the present application provides a door, including: a smart door lock;

[0060] The smart door lock includes a monitoring device, a communication device and a storage device;

[0061] The monitoring device is used to record the monitoring video stream;

[0062] The communication device is used to upload the monitoring video stream recorded by the monitoring device to the cloud and save it in the storage device, and is used to communicate with the smart terminal, and the smart terminal is used to execute the monitoring video processing method of the smart door lock as described above.

[0063] Compared with the prior art, this application has the following advantages:

[0064] The present application provides a monitoring video processing method and related equipment for a smart door lock, including: determining the user's object monitoring information based on the user's monitoring object screening information, the object monitoring information includes each monitoring object that the user is concerned about, and the monitoring object screening information is the object selection information entered by the user in the monitoring object interface, or the user's monitoring viewing record; obtaining the feature information of each monitoring object in the object monitoring information; based on the feature information of each monitoring object, performing stream segmentation processing on the monitoring video stream of the monitoring device to obtain a video frame set of each monitoring object. Thus, the user is provided with video frame sets of different monitoring objects, and the user can choose to view the corresponding video frame set of the monitoring object according to actual needs, and can quickly find the relevant video content of the monitoring object to be viewed, without the user having to search by dragging and dropping, thereby reducing the time required to find the relevant video content of the required monitoring object. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0066] Figure 1 A flowchart of a monitoring video processing method of a smart door lock provided in an embodiment of the present application;

[0067] Figure 2 A flow chart of a method for obtaining user object monitoring information provided in an embodiment of the present application;

[0068] Figure 3 A flowchart for obtaining user object monitoring information provided by an embodiment of the present application;

[0069] Figure 4 A flowchart of performing stream segmentation processing on a monitoring video stream of a monitoring device based on feature information of each monitoring object to obtain a set of video frames for each monitoring object provided in an embodiment of the present application;

[0070] Figure 5 An example diagram of a scene of surveillance video processing of a smart door lock provided in an embodiment of the present application;

[0071] Figure 6 A schematic diagram of the structure of a monitoring video processing device for a smart door lock provided in an embodiment of the present application;

[0072] Figure 7A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0074] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0075] The present application can be used in many general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, distributed computing environments including any of the above devices or equipment, etc.

[0076] Reference Figure 1 , which is a flow chart of a monitoring video processing method of a smart door lock provided in an embodiment of the present application, and is specifically described as follows:

[0077] S101. Determine the user's object monitoring information based on the user's monitoring object screening information.

[0078] The monitoring object screening information is the object selection information input by the user in the monitoring object interface, or the monitoring viewing record of the user. The object monitoring information includes each monitoring object of the user.

[0079] The embodiment of the present application provides a method, wherein the object monitoring information includes various monitoring objects that the user is concerned about. Preferably, each monitoring object can be various objects such as pets, people, and packages.

[0080] Preferably, there are two ways to obtain the user's object monitoring information. One is that the user inputs the relevant information of the object that needs to be paid attention to, and the other is that the device analyzes the user's viewing behavior and obtains the relevant information of the objects that the user frequently views; the objects described here are various items such as pets, people, and packages.

[0081] When the user's monitoring object screening information is the object selection information input by the user in the monitoring object interface, the object monitoring information is generated based on each monitoring object in the object selection information. The monitoring object interface displayed to the user includes multiple objects, and the user can confirm the monitoring object by checking. After the user completes the selection, the object selection information is generated, and the object selection information includes each monitoring object selected by the user.

[0082] When the user's monitoring object screening information is monitoring viewing history, refer to Figure 2 , is a flow chart of a method for determining a user's object monitoring information based on monitoring viewing records provided in an embodiment of the present application, and is specifically described as follows:

[0083] S201. Obtain user's preferred videos from historical surveillance videos based on surveillance viewing records.

[0084] It should be noted that the user's monitoring and viewing records can be obtained from the log library. When obtaining the user's monitoring and viewing records, the user's monitoring and viewing records within a preset time period can be obtained. For example, the preset time period can be within 1 month, within 3 months, or within a week, etc. The preset time period can be set according to the user's actual needs.

[0085] Furthermore, the monitoring viewing record may include a behavior record generated when the user drags and drops to view historical monitoring videos, a behavior record when viewing real-time monitoring videos, etc. The monitoring viewing record includes information such as the specific video content viewed by the user and the time of viewing the video.

[0086] Preferably, the historical surveillance videos are associated with a preset time period when the surveillance viewing records are obtained. Preferably, the historical surveillance videos are all surveillance videos within the preset time period.

[0087] Due to storage space limitations, the duration for which surveillance videos can be stored is limited. For example, surveillance videos can be stored for a maximum of 7 days, or a month. The duration for which surveillance videos can be stored is related to the storage space; this duration can also be regarded as the effective storage duration for surveillance videos.

[0088] When the effective storage period of surveillance video is 7 days and the preset time period is within 1 month, the historical surveillance video is the video within the past 7 days. When the preset time period is within 7 days, the historical surveillance video is the video within 7 days. In other words, the maximum duration of surveillance video is the effective storage period of surveillance video.

[0089] When obtaining the user's preferred video from the historical surveillance video, for each behavior record in the monitoring viewing record, determine whether there is a video corresponding to the behavior record in the historical surveillance video. If so, obtain the video segment corresponding to the behavior record; based on the obtained video segments, obtain the user's preferred video.

[0090] Preferably, the behavior record includes the time when the user watched the video, the recording time of the watched video, the number of times the video was watched, and other content.

[0091] According to the video recording time in the behavior record, determine whether there is a video corresponding to the behavior record in the historical surveillance video. When the video recording time in the behavior record exists in the recording time of the historical surveillance video, determine that there is a video corresponding to the behavior record in the historical surveillance video, and obtain the video clip according to the video recording time in the behavior record. When the video recording time in the behavior record does not exist in the recording time of the historical surveillance video, determine that there is no video corresponding to the behavior record in the historical surveillance video. At this time, the video clip corresponding to the behavior in the behavior record cannot be obtained.

[0092] For example, assuming that the recording time of the historical surveillance video is from October 24, 2024 to November 10, 2024, when the video recording time in the behavior record is from 9:00 to 10:00 on October 23, 2024, the video recording time of the behavior record is not within the recording time of the historical surveillance video, and therefore, the video clip corresponding to the behavior recorded in the behavior record cannot be obtained; when the video recording time in the behavior record is from 9:00 to 10:00 on October 25, 2024, the video clip in the historical surveillance video with the recording time from 9:00 to 10:00 on October 25, 2024 will be used as the video clip of the behavior record.

[0093] The obtained video clips are used to form the user's preferred videos. Preferably, the preferred videos can be understood as videos without the content that the user does not care about, that is, the content included in the preferred videos is mostly the content that the user cares about and is interested in.

[0094] S202: Identify each video object in the preferred video, filter monitoring objects in each video object, and generate object monitoring information based on the filtered monitoring objects.

[0095] The preferred video is processed using a preset object recognition model to identify each video object appearing in the preferred video. Exemplarily, the object recognition model analyzes each frame of the preferred video to obtain each object in each frame of the preferred video, and the identified objects are deduplicated to obtain each video object. Preferably, the video objects include but are not limited to pets, people, packages, plants and other objects.

[0096] Furthermore, the object recognition algorithms for constructing the object recognition model include but are not limited to convolutional neural networks, regional convolutional neural networks, YOLO algorithms, etc., which will not be illustrated here.

[0097] There are repeated objects among the objects identified from each frame of the preferred video, and the identified objects are deduplicated to obtain non-repeated video objects. Furthermore, different people and different pets can be used as different video objects.

[0098] Reference Figure 3 , which is a flowchart of obtaining the object monitoring information of a user provided in an embodiment of the present application, and is specifically described as follows:

[0099] S301 : For each video object, determine the weight of the video object based on monitoring viewing records and preferred videos, and determine the user's estimated preference value for the video object based on the weight of the video object.

[0100] For each video object, the monitoring viewing history and preferred videos are processed using a preset weight evaluation strategy to determine the weight of the video object; illustratively, the weight of the video object can be evaluated by determining the frequency of appearance of the video object in the preferred videos, the visual prominence of the video object in the video frame, the semantic importance, etc.

[0101] Furthermore, after obtaining the weight of each video object, the user's estimated preference value for the video object can be determined according to the weight of the video object. The estimated preference value can be understood as the user's estimated attention score for the video object. Preferably, the greater the weight, the higher the estimated preference value, and vice versa, the lower the estimated preference value.

[0102] S302: Select a monitoring object from each video object based on the estimated preference value of each video object.

[0103] Exemplarily, a video object having an estimated liking value greater than a preset value is determined as a monitoring object, thereby selecting a monitoring object from various video objects.

[0104] Exemplarily, the video objects are selected in order from large to small according to the estimated preference value, and the selection is stopped when the number of selected objects meets the preset number, and each selected video object is determined as a monitoring object.

[0105] In the method provided in the embodiment of the present application, the user's preferences are analyzed through the user's monitoring viewing records to obtain monitoring objects that the user pays high attention to or is interested in, so as to facilitate subsequent further processing of the monitoring video based on the obtained monitoring objects to obtain video information related to the monitoring objects, so that the user can view the video corresponding to the monitoring objects.

[0106] In the solution provided in the embodiment of the present application, in addition to the above-mentioned method, the method of obtaining the user's object monitoring information also includes:

[0107] Displaying a monitoring object selection interface to the user; obtaining object selection information input by the user in the monitoring object selection interface; and generating object attention information based on the object selection information.

[0108] In the method provided in the embodiment of the present application, the monitoring object selection interface can be understood as an interactive interface, in which each selectable object is displayed to the user. Preferably, each object displayed in the monitoring object selection interface can be added by the user independently or obtained by the device through monitoring video recognition.

[0109] Furthermore, the object selected by the user in the monitoring object selection interface is used as the monitoring object, and the object monitoring information is obtained.

[0110] The user can select the object of interest on the monitoring object selection interface, so that the device or system can provide the user with relevant video data of the selected object and the user can view the video content of the selected object.

[0111] This application supports users to select monitoring objects independently, which provides great convenience for users to find relevant videos of the objects they need to view, without having to drag the progress bar to find the required content. In addition, this application can also analyze the user's preferences for objects through the user's monitoring and viewing records, and then obtain relevant video information of objects that the user pays high attention to, so that subsequent users can find relevant video information of related objects.

[0112] S102: Acquire feature information of each monitored object in the object monitoring information.

[0113] The preset feature extraction model is used to extract the feature information of each monitored object in the object monitoring information.

[0114] The content of the feature information of different monitored objects is different. For example, when the monitored object is a person, the feature information includes the person's appearance features, such as facial features, body shape features, etc. When the monitored object is an animal, the feature information includes the animal's appearance, image and other features.

[0115] S103: Based on the feature information of each monitored object, the monitoring video stream of the monitoring device is segmented to obtain a video frame set of each monitored object.

[0116] The monitoring device can be a device used for monitoring, such as a camera, a cat's eye, a doorbell with monitoring function, etc.

[0117] The monitoring video stream of the monitoring device is the video stream saved during the monitoring process of the monitoring device.

[0118] Reference Figure 4 , which is a flowchart of performing stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each monitoring object provided in an embodiment of the present application to obtain a set of video frames of each monitoring object, and the specific description is as follows:

[0119] S401. For each monitored object, extract each first video frame containing the monitored object from a monitored video stream based on feature information of the monitored object.

[0120] Extract the first frame feature data of each video frame in the monitoring video stream. For each video frame, when the similarity between the first frame feature data of the video frame and the feature information of the monitored object is greater than or equal to a first preset value, determine the video frame as the first video frame containing the monitored object.

[0121] Preferably, the first preset value can be set according to actual needs, and the similarity between the first frame feature data of the video frame and the feature information of the monitored object can be determined using a distance algorithm, and the distance algorithm includes but is not limited to the Euclidean distance algorithm, the Manhattan distance algorithm, the cosine similarity algorithm, etc.

[0122] Furthermore, step S401 is executed by a local device or a terminal device that applies the method provided by the present application.

[0123] S402. Upload the characteristic information of each monitored object and the monitoring video stream of the monitoring device to the cloud to obtain cloud video data of each monitored object fed back by the cloud, wherein the cloud video data of each monitored object includes each second video frame corresponding to the characteristic information of the monitored object extracted from the monitoring video stream by the cloud.

[0124] After receiving the feature information and monitoring video stream of each monitored object, the cloud processes the monitoring video stream based on the feature information of each monitored object to obtain cloud video data of each monitored object.

[0125] Exemplarily, the cloud extracts second frame feature data of each video frame in the monitoring video stream, and for each monitored object, determines the similarity between the second frame feature data of each video frame in the monitoring video stream and the monitored object, and determines the video frame in the monitoring video stream whose similarity is greater than or equal to a second preset value as the second video frame of the monitored object, and then the obtained second video frames constitute the cloud video data of the monitored object.

[0126] The second preset value can be set according to actual needs. The similarity between the second frame feature data of each video frame in the monitoring video stream and the monitored object can be determined using a distance algorithm. The description of the distance algorithm can be found above and will not be repeated here.

[0127] Furthermore, the purpose of using cloud computing for processing here is to make up for the lack of computing power of local devices. The powerful computing power of the cloud can increase the complexity of data analysis, the analysis process is more comprehensive, and the data obtained is more comprehensive and accurate.

[0128] S403: For each monitored object, deduplicate the first video frames and the second video frames of the monitored object to obtain deduplicated video frames, and generate a set of video frames of the monitored object based on the deduplicated video frames.

[0129] In the method provided in the embodiment of the present application, the process of generating each set of video frames for monitoring each monitoring object is as follows:

[0130] Each of the deduplicated video frames is determined as a target video frame; based on a preset video frame grouping rule, each target video frame is grouped to obtain a plurality of video frame groups, each of which contains at least one target video frame.

[0131] The video frame grouping rules can be updated dynamically and can be set according to user needs, such as setting the grouping period, grouping by date, etc.

[0132] Furthermore, based on the preset video frame grouping rules, each target video frame is grouped to obtain a plurality of video frame groups, such as: determining the timestamp of each target video frame; dividing each target video frame based on the timestamp of each target video frame to obtain at least one video frame group, and composing each video frame group into a video frame set of the monitored object; the video frame group contains at least one target video frame, and each target video frame in the video frame group is sorted in the order of the timestamps, and the time interval between adjacent target video frames is less than or equal to the preset duration. Preferably, when a video frame group contains multiple target video frames, each target video frame in the video frame group can form a video, which is subsequently output in the form of a video.

[0133] The preset duration can be set according to actual needs, for example, set to 0.0417 seconds.

[0134] As a result, the time interval between the video frames in the obtained video frame group is less than the preset duration, which is conducive to the subsequent output of each video frame in the video frame group to the user in the form of video, effectively avoiding frame drops or discontinuous pictures.

[0135] Preferably, when dividing each video frame, in addition to dividing according to the time interval between video frames, it is also possible to divide according to date. Specifically, for each monitored object, each video frame of the monitored object after deduplication is determined as a target video frame; the timestamp of each target video is obtained, and the target video frames belonging to the same date are grouped together to obtain multiple first combinations; for each first combination, the target video frames in the first combination are divided according to the timestamps of the target video frames in the first combination to obtain multiple second combinations, and the target video frames in the second combination are sorted in order of timestamps, and the time interval between adjacent target video frames is less than or equal to a preset duration. The second combination can be regarded as a video frame group.

[0136] At this point, the various video frame groups of each monitored object on different monitoring dates can be obtained. At this time, the video frame groups of each monitored object in different time dimensions can be obtained, providing users with video frame groups of monitored objects in multiple time dimensions, which is convenient for subsequent users to query relevant video content of monitored objects on different dates.

[0137] In the method provided in the embodiment of the present application, based on the user's monitoring object screening information, the user's object monitoring information is determined, the object monitoring information includes various monitoring objects that the user is concerned about, and the monitoring object screening information is the object selection information entered by the user in the monitoring object interface, or the user's monitoring viewing record; the characteristic information of each monitoring object in the object monitoring information is obtained; based on the characteristic information of each monitoring object, the monitoring video stream of the monitoring device is stream segmented to obtain a video frame set of each monitoring object. Thus, the user is provided with video frame sets of different monitoring objects, and the user can choose to view the corresponding video frame set of the monitoring object according to actual needs, and can quickly find the relevant video content of the monitoring object to be watched, without the user having to search by dragging and dropping, thereby reducing the time required to find the relevant video content of the required monitoring object.

[0138] In another embodiment provided by the present application, after obtaining the video frame sets of each monitored object; the video frame sets of each monitored object are displayed to the user; when a viewing command fed back by the user is received, a target video frame set is determined in each video frame set based on the viewing command; the target video frame group is determined in the target video frame set using the viewing command; and each video frame in the target video frame group is output to the user.

[0139] Furthermore, when the video frame sets of each monitored object are displayed to the user, each video frame group in the video frame set of each monitored object may be displayed to the user in a list, so that the user can view according to his / her own needs.

[0140] Reference Figure 5, which is a scene example diagram of monitoring video processing of a smart door lock provided in an embodiment of the present application, as shown in the figure, includes a monitoring device, a terminal device, a cloud and a client. Preferably, the client can be set on the terminal device.

[0141] The monitoring device includes a camera for monitoring. The monitoring device sends a monitoring video stream to a terminal device and a cloud. The terminal device performs video analysis on the received monitoring video stream. The terminal device analyzes the local monitoring video stream to obtain each first video frame of each monitored object. The cloud analyzes the monitoring video on the cloud to obtain each second video frame of each monitored object. For each monitored object, processing is performed based on each first video frame and each second video frame of the monitored object to obtain each video frame set of each monitored object. Each video frame set is displayed to a user on a client so that the user can view the relevant video data of the corresponding monitored object on the client.

[0142] In the embodiment provided by the present application, by analyzing the videos or pictures uploaded by the monitoring device, some key features of the videos or pictures are found, such as all videos or pictures with family members, or videos or pictures with features such as strangers, packages, pets, etc. The videos or pictures are arranged in the order of their generation, and the videos or pictures are combined together to form different new videos. When the user views them, they can be played like normal videos, but the videos are all content that the user is concerned about, thereby filtering out normal videos or pictures that the user is not concerned about.

[0143] By applying the solution provided in this application, it is possible to analyze the surveillance video stream within a period of time, thereby obtaining relevant video data of the surveillance objects of concern to relatively scattered users, thereby making it easier for users to view the content they are concerned about, effectively filtering out video content that is not important to users, and making it easier for users to quickly find video content of the objects they are concerned about, thereby providing users with a better user experience.

[0144] In another embodiment provided in the present application, Figure 5 The monitoring device shown in is arranged in the smart door lock, and the smart door lock also includes a communication device and a storage device. The storage device is a local storage device of the smart door lock, such as a memory card, a U disk, etc. Further, the smart door lock is arranged on the door.

[0145] The communication device in the smart door lock is used to upload the surveillance video stream recorded by the monitoring device to the cloud and save the surveillance video stream in the storage device. It is also used to communicate with the terminal device. The terminal device communicates with the communication device and can view the relevant video data of the corresponding monitored object. Preferably, the terminal device can be a smart terminal, such as a mobile phone, computer, IPAD, etc.

[0146] Although the present application depicts operations in a particular order, this should not be understood as requiring that the operations be performed in the particular order shown or in a sequential order.Multitasking and parallel processing may be advantageous under certain circumstances.

[0147] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0148] In the embodiments provided in the present application, the present application also provides a monitoring video processing device, which is used to support Figure 1 For the specific implementation of the method shown, refer to Figure 6 , is a structural diagram of a monitoring video processing device of a smart door lock provided in an embodiment of the present application, and is specifically described as follows:

[0149] The first acquisition unit 501 is used to determine the object monitoring information of the user based on the monitoring object screening information of the user, where the object monitoring information includes each monitoring object that the user is concerned about; the monitoring object screening information is the object selection information input by the user in the monitoring object interface, or is the monitoring viewing record of the user;

[0150] A second acquisition unit 502 is used to acquire feature information of each monitored object in the object monitoring information;

[0151] The segmentation unit 503 is used to perform stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each of the monitored objects, so as to obtain a set of video frames for each of the monitored objects.

[0152] In the device provided in the embodiment of the present application, based on the user's monitoring object screening information, the user's object monitoring information is determined, the object monitoring information includes various monitoring objects that the user is concerned about, and the monitoring object screening information is the object selection information entered by the user in the monitoring object interface, or the user's monitoring viewing record; the characteristic information of each monitoring object in the object monitoring information is obtained; based on the characteristic information of each monitoring object, the monitoring video stream of the monitoring device is stream segmented to obtain a video frame set of each monitoring object. Thus, the user is provided with video frame sets of different monitoring objects, and the user can choose to view the corresponding video frame set of the monitoring object according to actual needs, and can quickly find the relevant video content of the monitoring object to be viewed, without the user having to search by dragging and dropping, thereby reducing the time required to find the relevant video content of the required monitoring object.

[0153] In another embodiment provided by the present application, the first acquisition unit of the device performs a process of determining the object monitoring information of the user based on the monitoring object screening information of the user, including:

[0154] When the monitoring object screening information is object selection information input by the user in the monitoring object interface, generating object monitoring information based on each monitoring object in the object selection information;

[0155] When the monitoring object screening information is the monitoring viewing record of the user, obtaining the user's preferred video from historical monitoring videos based on the monitoring viewing record;

[0156] The respective video objects in the preferred video are identified, monitoring objects are screened in the respective video objects, and object monitoring information is generated based on the screened monitoring objects.

[0157] In another embodiment provided by the present application, the first acquisition unit of the device performs a process of screening the monitoring object in each of the video objects, including:

[0158] For each of the video objects, based on the monitored viewing records and the preferred videos, determine the weight of the video object, and based on the weight of the video object, determine the user's estimated liking value for the video object;

[0159] A monitoring object is selected from each of the video objects based on the estimated likeness value of each of the video objects.

[0160] In another embodiment provided by the present application, the segmentation unit of the device performs stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each of the monitored objects to obtain a video frame set of each of the monitored objects, including:

[0161] For each of the monitored objects, extracting each first video frame containing the monitored object from the monitored video stream based on the feature information of the monitored object;

[0162] Uploading the characteristic information of each monitored object and the monitoring video stream of the monitoring device to the cloud to obtain cloud video data of each monitored object fed back by the cloud, wherein the cloud video data of each monitored object includes each second video frame corresponding to the characteristic information of the monitored object extracted by the cloud from the monitoring video stream;

[0163] For each of the monitored objects, deduplication processing is performed on each of the first video frames and each of the second video frames of the monitored object to obtain each deduplicated video frame, and each video frame set of the monitored object is generated based on each deduplicated video frame.

[0164] In another embodiment provided by the present application, the segmentation unit of the device performs a process of generating a set of video frames of the monitored object based on the deduplicated video frames, including:

[0165] Determine each video frame after deduplication as a target video frame;

[0166] Based on a preset video frame grouping rule, each of the target video frames is grouped to obtain a plurality of video frame groups, wherein the video frame group includes at least one target video frame.

[0167] In another embodiment provided by the present application, the device further includes:

[0168] A display unit, used for displaying the video frame sets of each of the monitored objects to the user;

[0169] A first determining unit, configured to determine a target video frame set from each of the video frame sets based on the viewing command when receiving a viewing command fed back by the user;

[0170] a second determining unit, configured to determine a target video frame group in the target video frame set using the viewing command;

[0171] An output unit is used to output each video frame in the target video frame group to the user.

[0172] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned monitoring video processing method.

[0173] The present application also provides an electronic device, whose structural diagram is as follows: Figure 7 As shown, it specifically includes a memory 601 and one or more instructions 602, wherein the one or more instructions 602 are stored in the memory 601 and are configured to be executed by one or more processors 603 to perform the following operations:

[0174] Determine the object monitoring information of the user based on the monitoring object screening information of the user, wherein the object monitoring information includes each monitoring object that the user is concerned about; the monitoring object screening information is the object selection information input by the user in the monitoring object interface, or is the monitoring viewing record of the user;

[0175] Acquire characteristic information of each monitored object in the object monitoring information;

[0176] Based on the characteristic information of each of the monitored objects, the monitoring video stream of the monitoring device is segmented to obtain a set of video frames for each of the monitored objects.

[0177] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions.

[0178] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of this application.

[0179] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0180] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0181] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A monitoring video processing method for a smart door lock, characterized in that: include: Determining object monitoring information of the user based on monitoring object screening information of the user, wherein the object monitoring information includes various monitoring objects that the user is concerned about; The monitoring object screening information is the object selection information input by the user in the monitoring object interface, or is the monitoring viewing record of the user; Acquire characteristic information of each monitored object in the object monitoring information; Based on the characteristic information of each of the monitored objects, the monitoring video stream of the monitoring device is segmented to obtain a set of video frames for each of the monitored objects.

2. The method according to claim 1, characterized in that The step of determining the object monitoring information of the user based on the monitoring object screening information of the user includes: When the monitoring object screening information is object selection information input by the user in the monitoring object interface, generating object monitoring information based on each monitoring object in the object selection information; When the monitoring object screening information is the monitoring viewing record of the user, obtaining the user's preferred video from historical monitoring videos based on the monitoring viewing record; The respective video objects in the preferred video are identified, monitoring objects are screened in the respective video objects, and object monitoring information is generated based on the screened monitoring objects.

3. The method according to claim 2, characterized in that The screening of monitoring objects in each of the video objects includes: For each of the video objects, based on the monitored viewing records and the preferred videos, determine the weight of the video object, and based on the weight of the video object, determine the user's estimated liking value for the video object; A monitoring object is selected from each of the video objects based on the estimated likeness value of each of the video objects.

4. The method according to claim 1, characterized in that: The step of performing stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each of the monitoring objects to obtain a set of video frames of each of the monitoring objects includes: For each of the monitored objects, extracting each first video frame containing the monitored object from the monitored video stream based on the feature information of the monitored object; Uploading the characteristic information of each monitored object and the monitoring video stream of the monitoring device to the cloud to obtain cloud video data of each monitored object fed back by the cloud, wherein the cloud video data of each monitored object includes each second video frame corresponding to the characteristic information of the monitored object extracted by the cloud from the monitoring video stream; For each of the monitored objects, deduplication processing is performed on each of the first video frames and each of the second video frames of the monitored object to obtain each deduplicated video frame, and each video frame set of the monitored object is generated based on each deduplicated video frame.

5. The method according to claim 4, characterized in that The generating of each set of video frames of the monitored object based on each deduplicated video frame comprises: Determine each video frame after deduplication as a target video frame; Based on a preset video frame grouping rule, each of the target video frames is grouped to obtain a plurality of video frame groups, wherein the video frame group includes at least one target video frame.

6. The method according to claim 5, characterized in that Also includes: Displaying the video frame sets of each of the monitored objects to the user; When receiving a viewing command fed back by the user, determining a target video frame set from each of the video frame sets based on the viewing command; Determine a target video frame group in the target video frame set using the viewing command; Each video frame in the target video frame group is output to the user.

7. A monitoring video processing device for a smart door lock, characterized in that: include: A first acquisition unit is used to determine the object monitoring information of the user based on the monitoring object screening information of the user, wherein the object monitoring information includes each monitoring object that the user is concerned about; the monitoring object screening information is the object selection information input by the user in the monitoring object interface, or is the monitoring viewing record of the user; A second acquisition unit, configured to acquire feature information of each monitored object in the object monitoring information; The segmentation unit is used to perform stream segmentation processing on the monitoring video stream of the monitoring device based on the feature information of each of the monitored objects to obtain a set of video frames for each of the monitored objects.

8. A storage medium, characterized in that: The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the monitoring video processing method of the smart door lock according to any one of claims 1 to 6.

9. An electronic device, characterized in that: It includes a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform the monitoring video processing method of the smart door lock as described in any one of claims 1-6.

10. A door, characterized in that: include: Smart door locks; The smart door lock includes a monitoring device, a communication device and a storage device; The monitoring device is used to record the monitoring video stream; The communication device is used to upload the monitoring video stream recorded by the monitoring device to the cloud and save it in the storage device, and is used to communicate with the smart terminal, and the smart terminal is used to execute the monitoring video processing method of the smart door lock described in any one of claims 1-6.

Images