Security system based on image recognition

Abstract

The application discloses an image recognition-based security system, and relates to the technical field of smart homes.The method comprises a target device and a server, wherein the target device is used for performing event detection on a video stream to obtain a video slice containing a preset event, and sends the video slice and environment data corresponding to the video slice to the server, wherein the preset event at least includes one of the following: an event of detecting a target subject and a preset abnormal event; the server is used for extracting a video frame sequence in the video slice; performing feature extraction on the video frame sequence and the environment data to determine descriptive text and an event category corresponding to the preset event, and identifying a risk level corresponding to the preset event according to the descriptive text and the event category, wherein the descriptive text is used for indicating subject attributes, behavior actions of the target subject and an environment state when the preset event occurs.

Description

Technical Field

[0001] This application relates to the field of smart home technology, and more specifically, to a security system based on image recognition. Background Technology

[0002] Existing home security systems typically consist of front-end smart cameras (device-side) and a back-end cloud platform (cloud-side). The device-side uploads complete video streams to the cloud for analysis. This solution requires uploading high-definition video streams around the clock, resulting in extremely high data throughput. For home networks with limited uplink bandwidth (e.g., below 20Mbps), continuous high-load uploads can easily cause network congestion, leading to stuttering, disconnections, or delays in other smart devices (such as smart speakers, robot vacuums, and video conferencing devices). Furthermore, uploading the complete video stream is time-consuming, especially during network fluctuations, where delays can reach several seconds or even tens of seconds. The cloud model's inference calculations on long video sequences are also extremely time-consuming. This cumulative latency results in a significant lag in security alarm responses.

[0003] Therefore, existing technologies suffer from problems such as excessive video transmission bandwidth consumption and large delays in security alarm response due to relying solely on cloud-based video analysis of videos detected by security systems.

[0004] There is currently no effective solution to the problems of excessive video transmission bandwidth consumption and large security alarm response delays caused by the existing technology of performing video analysis on videos detected by security systems only through the cloud. Summary of the Invention

[0005] This application provides an image recognition-based security system to at least solve the problems of excessive video transmission bandwidth consumption and large security alarm response delays caused by the prior art, which only performs video analysis on the video detected by the security system through the cloud.

[0006] According to one embodiment of this application, an image recognition-based security system is provided, comprising: a target device and a server, wherein: the target device is configured to perform event detection on a video stream to obtain video slices containing preset events, and send the video slices and corresponding environmental data to the server, wherein the preset events include at least one of the following: an event of detecting a target subject, or a preset abnormal event; the server is configured to extract video frame sequences from the video slices; perform feature extraction on the video frame sequences and the environmental data to determine descriptive text and event categories corresponding to the preset events, and identify the risk level corresponding to the preset events based on the descriptive text and the event categories, wherein the descriptive text is used to indicate the subject attributes, behaviors, and environmental states when the preset events occur.

[0007] In an exemplary embodiment, the target device is further configured to: detect each video frame in the video stream according to a target detection model to determine a first detection result corresponding to each video frame, wherein the first detection result includes at least one of the following: whether each video frame contains a target detection box, the category label of the target detection box, the first confidence level corresponding to the category label of the target detection box, and the target detection box being a detection box corresponding to the target subject; determine a first video frame set in the video stream according to the first detection result, wherein the first detection result of each video frame in the first video frame set indicates that the corresponding video frame contains the target detection box, and the category label of the target detection box is a target category label, and the first confidence level of the target category label is greater than or equal to a first confidence threshold; and filter video slices containing events that detect a target subject in the video stream according to the first video frame set.

[0008] In an exemplary embodiment, the target device is further configured to: determine the pixel change rate between each video frame in the video stream and the preceding video frame of each video frame; determine a second set of video frames in the video stream in which the pixel change rate is greater than or equal to a change rate threshold; and filter video slices containing preset abnormal events in the video stream according to the second set of video frames.

[0009] In an exemplary embodiment, the server is further configured to: acquire a plurality of first video frames in the video slice based on a preset step size, and calculate a first visual dissimilarity between each two adjacent first video frames; predict the probability distribution of the image category corresponding to each first video frame according to a classification model; calculate the entropy value and the second confidence level corresponding to the probability distribution of each first video frame; calculate a first semantic richness score for each first video frame based on the entropy value and the second confidence level; and extract the video frame sequence from the video slice based on the first visual dissimilarity and the first semantic richness score.

[0010] In an exemplary embodiment, the server is further configured to: determine a third set of video frames in the plurality of first video frames where the first visual difference is greater than or equal to a preset difference, and / or the first semantic richness score is greater than or equal to a first score threshold; determine whether the number of video frames contained in the third set of video frames is within a preset number range; and if the number of video frames contained in the third set of video frames is determined to be within the preset number range, construct the video frame sequence based on the video frames contained in the third set of video frames.

[0011] In an exemplary embodiment, the server is further configured to: when the number of video frames included in the third video frame set is less than the minimum value of the preset number range, acquire multiple second video frames in the video slice based on the preset step size, and calculate a second visual difference degree between each two adjacent second video frames and a second semantic richness score for each third video frame; determine at least one third video frame among the multiple second video frames whose second visual difference degree is greater than or equal to the preset difference degree, and add each third video frame to the third video frame set sequentially according to the second semantic richness of the at least one third video frame, until the number of video frames included in the third video frame set is within the preset number range; and construct the video frame sequence based on the third video frame set after adding the third video frames.

[0012] In an exemplary embodiment, the server is further configured to: encode the video frame sequence based on a preset encoder to determine a visual embedding vector corresponding to the video frame sequence, and to determine an environmental feature vector corresponding to the environmental data; based on a target attention mechanism, determine the visual embedding vector as a query vector, determine the environmental feature vector and the temporal context data corresponding to the video frame sequence as a key and a value, and perform feature fusion on the visual embedding vector and the environmental feature vector according to the query vector, the key and the value, to generate a fused feature; input the fused feature and the historical behavior data of the target subject into an inference model, so that the inference model generates the descriptive text and the event category through causal inference.

[0013] In an exemplary embodiment, the server is further configured to: extract at least one semantic keyword and a weight corresponding to each semantic keyword from the descriptive text; calculate a first risk score corresponding to the descriptive text based on the at least one semantic keyword and the weight corresponding to each semantic keyword, and calculate a second risk score corresponding to the event category; calculate a target score corresponding to the preset event based on the first risk score and the second risk score; and determine a risk level corresponding to the preset event based on the target score.

[0014] In an exemplary embodiment, the server is further configured to: acquire historical behavioral data of the target subject and health status tags of the target subject; correct the target score based on the historical behavioral data and the health status tags; and match the corrected target score with the score range corresponding to each risk level to determine the risk level corresponding to the preset event.

[0015] In an exemplary embodiment, the server is further configured to: cache the descriptive text and the event category when the risk level corresponding to the preset event is a first risk level; generate target notification information based on the descriptive text and the event category when the risk level is a second risk level, and send the target notification information to the target contact corresponding to the target subject; send a call request to the target contact when the risk level is a third risk level, and send the preset event to the target contact when the target contact responds to the call request, wherein the priority of the first risk level is lower than the priority of the second risk level, and the priority of the second risk level is lower than the priority of the third risk level.

[0016] The image recognition-based security system in this application embodiment includes a target device and a server. The target device performs event detection on a video stream to obtain video slices containing preset events. The preset events include at least one of the following: an event indicating the detection of a target subject, or a preset abnormal event. The target device sends the video slices and corresponding environmental data to the server. The server extracts video frame sequences from the video slices, performs feature extraction on the video frame sequences and environmental data to determine descriptive text and event categories corresponding to the preset events, indicating the subject attributes, behaviors, and environmental state when the preset events occur. It also identifies the risk level corresponding to the preset events based on the descriptive text and event categories. In other words, in this application embodiment, the end-user (e.g., the target device) and the cloud (including the server) interact. The target device does not directly send the video stream to the server; instead, it obtains video slices containing preset events from the video stream, sends the video slices and corresponding environmental data to the server, and then the server analyzes the video slices to determine the risk level of the preset events. The embodiments of this application can solve the problems of excessive video transmission bandwidth consumption and large security alarm response delay caused by video analysis of security system detected only through the cloud in the prior art. Therefore, the embodiments of this application can reduce video transmission bandwidth and reduce security alarm delay. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a structural block diagram of an image recognition-based security system according to an embodiment of this application;

[0020] Figure 2 This is a schematic diagram illustrating the steps performed by a smart home security system according to an optional embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] This embodiment provides a security system based on image recognition. Figure 1 This is a structural block diagram of an image recognition-based security system according to an embodiment of this application. The system includes: a target device 12 and a server 14, wherein:

[0024] The target device 12 is used to perform event detection on the video stream to obtain video slices containing preset events, and to send the video slices and the environmental data corresponding to the video slices to the server. The preset events include at least one of the following: an event that detects a target subject, or a preset abnormal event.

[0025] The target equipment mentioned above refers to security equipment. The target entities include, but are not limited to, people and animals.

[0026] Preset events include: detecting the presence of a person, detecting the presence of a pet, etc.

[0027] The server 14 is configured to extract video frame sequences from the video slice; perform feature extraction on the video frame sequences and the environmental data to determine the descriptive text and event category corresponding to the preset event; and identify the risk level corresponding to the preset event based on the descriptive text and the event category, wherein the descriptive text is used to indicate the subject attributes, behavior actions, and environmental state when the preset event occurs.

[0028] The aforementioned servers can be servers with edge computing capabilities, such as those in the cloud.

[0029] The aforementioned system includes a target device and a server. The target device performs event detection on the video stream to obtain video slices containing preset events. The preset events include at least one of the following: an event indicating the detection of a target subject, or a preset abnormal event. The target device sends the video slices and corresponding environmental data to the server. The server extracts video frame sequences from the video slices, performs feature extraction on the video frame sequences and environmental data to determine the corresponding descriptive text and event category indicating the subject's attributes, actions, and environmental state at the time of the preset event, and identifies the risk level corresponding to the preset event based on the descriptive text and event category. In other words, in this embodiment, the end-user (e.g., the target device) and the cloud (including the server) interact. The target device does not directly send the video stream to the server, but instead obtains video slices containing preset events from the video stream, sends the video slices and corresponding environmental data to the server, and then the server analyzes the video slices to determine the risk level of the preset events. The embodiments of this application can solve the problems of excessive video transmission bandwidth consumption and large security alarm response delay caused by video analysis of security system detected only through the cloud in the prior art. Therefore, the embodiments of this application can reduce video transmission bandwidth and reduce security alarm delay.

[0030] Optionally, the target device is further configured to: detect each video frame in the video stream according to a target detection model to determine a first detection result corresponding to each video frame, wherein the first detection result includes at least one of the following: whether each video frame contains a target detection box, the category label of the target detection box, the first confidence level corresponding to the category label of the target detection box, and the target detection box being a detection box corresponding to the target subject; determine a first video frame set in the video stream according to the first detection result, wherein the first detection result of each video frame in the first video frame set indicates that the corresponding video frame contains the target detection box, and the category label of the target detection box is a target category label, and the first confidence level of the target category label is greater than or equal to a first confidence level threshold; and filter video slices containing events that detect the target subject in the video stream according to the first video frame set.

[0031] Understandably, the target device has a built-in lightweight object detection model that performs real-time inference on each video frame in the video stream captured by the video acquisition device (e.g., a camera), and determines video slices in the first video stream. Specifically:

[0032] Locate the target subject in each video frame, such as a person, pet, or special object, and generate a bounding box (i.e., an object detection box); identify the category label within the bounding box, for example, if the target subject is a person, the category label is "elderly"; if the target subject is an animal, the category label is "cat," etc.; calculate the first confidence score corresponding to the category label.

[0033] For each video frame in the video stream, select the video frames from the first set of video frames based on the following conditions:

[0034] Condition 1: The video frame contains a bounding box for the target.

[0035] Condition 2: The category label of the object detection box is the target category label. For example, only focus on object detection boxes with the category label "elderly" or "pet".

[0036] Condition 3: The confidence level corresponding to the category label is greater than or equal to the first confidence threshold (e.g., 0.8).

[0037] Filter the video stream to find all video frames that meet conditions 1, 2, and 3. Then, select this video frame as the first set of video frames.

[0038] For example, in the 100th video frame, a target detection box is detected, and the category label of the target detection box is: cat. The confidence score corresponding to the category label is 0.95, which is greater than the first confidence score threshold of 0.8. Therefore, the 100th video frame is added to the first video frame set.

[0039] Furthermore, the video frames in the first video frame set are traced back in the video stream to determine the time window corresponding to the video frames in the first video frame set. Using the timestamps of the first frame and the last frame in the first video frame set as boundaries, the time length is extended forward and backward respectively, and then continuous video frames within this time period are extracted and determined as video slices.

[0040] Based on the above technical solution, assuming 24 hours of video per day, the actual video slices that generate preset events may be less than 10 minutes. Bandwidth usage is reduced by more than 99%. Even if 100 preset events occur per day, with each slice lasting only 3 seconds, the total amount of uploaded data is extremely small, avoiding home network congestion.

[0041] Optionally, the target device is further configured to: determine the pixel change rate between each video frame in the video stream and the previous video frame of each video frame; determine a second set of video frames in the video stream in which the pixel change rate is greater than or equal to a change rate threshold; and filter video slices containing preset abnormal events in the video stream according to the second set of video frames.

[0042] Understandably, the specific steps for filtering video clips containing preset abnormal events are as follows:

[0043] Each video frame in the video stream is processed in real time to calculate the pixel change rate between the current frame and the previous frame. Methods for calculating the pixel change rate include, but are not limited to, inter-frame differencing, optical flow, or background subtraction.

[0044] For example: convert the current frame and the previous frame to grayscale, calculate the absolute difference of pixel values ​​between the two frames, and count the number of pixels that have changed significantly; divide the number of pixels that have changed by the total number of pixels to obtain the pixel change rate.

[0045] Traverse the video stream, adding a frame to the second video frame set only if it meets all of the following conditions:

[0046] Condition 1: The pixel change rate is greater than or equal to the change rate threshold;

[0047] Condition 2: The pixel change rate continues for a certain period of time (target time period) or for multiple consecutive frames (N consecutive frames, where N is a positive integer) that are greater than or equal to the change rate threshold, in order to exclude instantaneous interference such as flash and snow noise.

[0048] Starting with the timestamp of the first frame in the second video frame set and ending with the timestamp of the last frame, the starting point and the ending point are extended forward and backward by a preset time period, respectively, to obtain video slices within this time period.

[0049] Optionally, the server is also used to extract video frame sequences from the video slices, specifically:

[0050] (1) Based on a preset step size, multiple first video frames are obtained in the video slice, and the first visual difference between each two adjacent first video frames in the multiple first video frames is calculated;

[0051] In this embodiment of the application, multiple first video frames are extracted from the video slice based on a preset step size, for example, one frame every 0.5 seconds.

[0052] For example, for a 3-second video slice with a preset step size of 0.5 seconds, the first video frames at 0s, 0.5s, 1s, 0.5s, 2s, 2.5s, and 3s are extracted.

[0053] Calculate the pixel difference or feature difference (i.e., the first visual difference) between every two consecutive frames in multiple first video frames to determine the drastic degree of change in the video frame image.

[0054] (2) Predict the probability distribution of the image category corresponding to each first video frame according to the classification model; calculate the entropy value and the second confidence value corresponding to the probability distribution of each first video frame; calculate the first semantic richness score of each first video frame according to the entropy value and the second confidence value;

[0055] Understandably, the method for calculating the first semantic richness score is as follows:

[0056] The classification model predicts the probability distribution of each first video frame belonging to various categories, including but not limited to: people, cats, sofas, etc. The information entropy of this probability distribution is calculated; this entropy value represents the complexity of the image content of the first video frame. The higher the entropy value, the more complex the image content of the first video frame, and the higher the uncertainty. Combining the entropy value and the second confidence level, a first semantic richness score is calculated for each first video frame. For example: Frame A (blurred human figure): high entropy value, low confidence level, therefore frame A has a low first semantic richness score; Frame B (clear image of an elderly person falling): relatively high entropy value, high confidence level, therefore frame B has a high first semantic richness score; Frame C (static wall): low entropy value, high confidence level, therefore frame C has a low first semantic richness score.

[0057] (3) Extract the video frame sequence from the video slice based on the first visual dissimilarity score and the first semantic richness score, specifically:

[0058] 1) Determine a set of third video frames in the plurality of first video frames where the first visual difference is greater than or equal to a preset difference, and / or the first semantic richness score is greater than or equal to a first score threshold;

[0059] A set of frames that meets any of the following conditions will be designated as the third video frame set: the first visual difference is greater than or equal to the preset difference; the first semantic richness score is greater than or equal to the first score threshold.

[0060] 2) Determine whether the number of video frames contained in the third video frame set is within a preset range; wherein:

[0061] If the number of video frames contained in the third video frame set is within the preset range, the video frame sequence is constructed based on the video frames contained in the third video frame set.

[0062] If the number of video frames in the third video frame set is less than the minimum of the preset number range, multiple second video frames are obtained from the video slice based on the preset step size, and a second visual difference degree between each two adjacent second video frames and a second semantic richness score for each third video frame are calculated; at least one third video frame is determined from the multiple second video frames whose second visual difference degree is greater than or equal to the preset difference degree, and each third video frame is added to the third video frame set sequentially according to the second semantic richness of the at least one third video frame until the number of video frames in the third video frame set is within the preset number range; the video frame sequence is constructed based on the third video frame set after adding the third video frames.

[0063] Understandably, if the number of frames in the third video frame set is within a preset range, the video frame sequence is directly constructed based on the third video frame set; if the number of frames in the third video frame set is less than the minimum value of the preset range, then video frames need to be added to the third video frame set to bring the number of frames in the third video frame set to the preset range, and the video frame set is constructed based on the added third video frame set.

[0064] Optionally, the server is further configured to: encode the video frame sequence based on a preset encoder to determine the visual embedding vector corresponding to the video frame sequence, and to determine the environmental feature vector corresponding to the environmental data; based on a target attention mechanism, determine the visual embedding vector as a query vector, determine the environmental feature vector and the temporal context data corresponding to the video frame sequence as keys and values, and perform feature fusion on the visual embedding vector and the environmental feature vector according to the query vector, the keys and values ​​to generate fused features; input the fused features and the historical behavior data of the target subject into an inference model, so that the inference model generates the descriptive text and the event category through causal inference.

[0065] Understandably, after extracting the video frame sequence, the server generates descriptive text and event categories for preset events based on the video frame sequence. Specifically:

[0066] The video frame sequence and environmental data are independently encoded and transformed into high-dimensional vector representations. Specifically, a pre-defined encoder, such as a visual encoder based on the Transformer architecture, is used to process the extracted video frame sequence. The pre-defined encoder captures the spatiotemporal relationships between frames, converting image pixel information into visual embedding vectors. Simultaneously, environmental data at the current moment (e.g., temperature and humidity, door lock status, heart rate bracelet data, smart speaker voice logs) is acquired and mapped into environmental feature vectors through a fully connected neural network or a specific environmental encoder.

[0067] Perform the following settings: 1) Query vector: Use a visual embedding vector as the Query. 2) Key and value: Use environmental feature vectors and time context data (such as the current time point and historical habit tags) as the Key and Value.

[0068] Calculate the similarity between the query and the key, aggregate the values ​​using weighted averages, and generate fused features.

[0069] By integrating features with historical behavioral data of the target subject (i.e., user profiles) into the inference model, the inference model can output descriptive text and event categories based on causal inference mechanisms.

[0070] Optionally, the server is further configured to determine the risk level corresponding to the preset event, specifically:

[0071] (1) Extract at least one semantic keyword and the weight corresponding to each semantic keyword from the descriptive text; calculate a first risk score corresponding to the descriptive text based on the at least one semantic keyword and the weight corresponding to each semantic keyword, and calculate a second risk score corresponding to the event category; calculate a target score corresponding to the preset event based on the first risk score and the second risk score.

[0072] Understandably, the target score corresponding to the preset event is calculated, specifically:

[0073] Natural language processing is performed on the descriptive text generated by the inference model (e.g., elderly person has not moved for a long time, abnormal heart rate, suspected fall). Semantic keywords (such as "fall," "abnormal heart rate," "long time") are extracted, and weights are assigned to each semantic keyword. A weighted sum or weighted average is performed based on the semantic keywords and their weights to obtain the first risk score.

[0074] The inferred event category is directly mapped to a second risk score. For example, if the event category is a high-risk fall event, the second risk score is set to 0.9; if the event category is an exercise-rest event, the second risk score is set to 0.1.

[0075] The target score is obtained by weighting and fusing the first risk score and the second risk score.

[0076] (2) Obtain the historical behavior data of the target subject and the health status tag of the target subject; correct the target score according to the historical behavior data and the health status tag; match the corrected target score with the score range corresponding to each risk level to determine the risk level corresponding to the preset event.

[0077] Understandably, the target score needs to be revised, and the risk level corresponding to the preset event needs to be determined based on the revised target score. Specifically:

[0078] Obtain historical behavioral data (such as daily routines and activity frequency) and health status labels (such as living alone, high blood pressure, and mobility impairment) of the target subject.

[0079] Adjust rating weights based on user tags.

[0080] The correction logic is as follows:

[0081] If the user is tagged as "elderly person living alone": increase sensitivity to events such as "stillness" and "falling" to improve the target score.

[0082] If the user is tagged as "healthy young adult": reduce sensitivity to similar events and lower the target score.

[0083] If the user is tagged as "person with limited mobility": increase the initial risk expectation of "prolonged inactivity" and improve the target score.

[0084] For example: Elderly people living alone, tag: high-risk group: target score: 0.792, correction factor: due to semantic keywords such as "living alone" and "advanced age", the correction value is determined to be +0.15, and the corrected target score is: 0.792 + 0.15 = 0.942.

[0085] Match risk levels, for example:

[0086] Level 5 (High Risk, i.e., the third risk level): The revised target score is ≥0.9;

[0087] Level 3 (Medium risk, i.e., the second risk level): 0.6 ≤ revised target score < 0.9;

[0088] Level 1 (Low risk, i.e., the first level of risk): The revised target score is <0.6.

[0089] Optionally, the server is further configured to: cache the descriptive text and the event category when the risk level corresponding to the preset event is a first risk level; generate target notification information based on the descriptive text and the event category when the risk level is a second risk level, and send the target notification information to the target contact corresponding to the target subject; send a call request to the target contact when the risk level is a third risk level, and send the preset event to the target contact when the target contact responds to the call request, wherein the priority of the first risk level is lower than the priority of the second risk level, and the priority of the second risk level is lower than the priority of the third risk level.

[0090] Understandably, different risk levels require different handling methods:

[0091] First risk level, for example: low risk, such as children running around in the living room or pets resting normally: execute background data caching operations, cache preset events, as well as the descriptive text and event category corresponding to the preset events;

[0092] Second risk level, for example: medium risk, such as pets chewing on sofas, elderly people not moving for a long time but with normal heart rate: generate a visual notification and send the visual notification to the target contact;

[0093] The third risk level, such as high risk, such as an elderly person falling or a person living alone experiencing cardiac arrest: dial the emergency contact number of the target contact. Once the target contact responds to the call, immediately transmit key information about the preset event (such as descriptive text or video clip links) to the target contact during the call.

[0094] To better understand the process of the above-described image recognition-based security system, the following description, in conjunction with optional embodiments, further illustrates the flow of the image recognition-based security system, but is not intended to limit the technical solutions of the embodiments of this application.

[0095] Existing home security systems are generally limited to a passive mode of event detection and alarm response, lacking a deep understanding of users' lifestyles and the ability to analyze personalized behaviors. These systems can only identify basic security events such as abnormal opening of doors and windows or movement of people; they cannot distinguish between scenarios involving elderly people and scenarios where an elderly person has remained still for an extended period and may be at risk of illness; they cannot detect behaviors such as pets chewing on sofas; and they cannot combine multi-dimensional contextual information such as time, habits, and health status to provide differentiated alerts and proactive intervention. This one-size-fits-all alarm mechanism leads to user fatigue, high false alarm rates, low trust, and a disconnect between the security system and daily life services.

[0096] Existing technologies include the following solutions: Pure cloud-based full video analysis: Uploading the entire video stream to the cloud for analysis presents the following problems: high bandwidth pressure, high risk of privacy leakage, high latency, and does not conform to the trend of edge computing; Rule-based multi-sensor linkage systems: For example, a preset "alarm will be triggered if the door lock is closed + no one moves + temperature is below 18℃". This presents the following problems: inability to adapt to individual differences, lack of semantic understanding, and easy to miss cases (e.g., an elderly person lying in bed with a normal temperature); Single voice analysis systems: Only analyzing voice content collected by smart speakers (e.g., cries for help). This presents the following problems: inability to detect silent anomalies (e.g., a fall without cries for help), large blind spots; Using general-purpose large models (e.g., GPT-4 visual model) to directly analyze video presents the following problems: general-purpose large models are not fine-tuned for home security scenarios, resulting in low inference efficiency, high energy consumption, slow response, and lack of support for local deployment, as well as poor privacy compliance.

[0097] To address the aforementioned issues, this application proposes an optional embodiment of a smart home security system based on a Vision-Language Model (VLM). Specifically, it includes three parts: an edge-side intelligent video analysis module, a cloud-based multimodal large model engine, and a personalized interaction and intervention center. Among these, as shown... Figure 2 As shown:

[0098] I. Edge-side Intelligent Video Analysis Module: Equipped with algorithms for human detection, pet detection, and anomaly detection. It can initially detect three types of events: "Someone is present," "A pet is present," and "Anomaly is present." Events are reported as video segments (i.e., video cuts) of fixed duration T. Specifically:

[0099] The edge-side intelligent video analytics module performs the following steps:

[0100] (1) Continuous real-time video acquisition, performing three types of basic video detection: 1) Human target detection: normal / abnormal human body; 2) Pet target detection: normal / abnormal pet; 3) Abnormal event detection;

[0101] (2) Determine whether a valid event (i.e. a preset event) has been detected. If a valid event is detected, capture an event video stream (i.e. a video slice) of duration T. If no valid event is detected, continue to monitor silently and poll in a loop.

[0102] In summary, smart cameras with edge computing capabilities are deployed in the home to continuously collect video streams. The security device (i.e., the target device) does not upload the complete video stream. Instead, it automatically extracts T-second video slices based on triggered events (such as motion detection). After target detection and motion trajectory coarse screening by an edge-side algorithm model (e.g., MobileNetV3), only the video of events containing people / pets / abnormalities (i.e., video slices) is uploaded, significantly reducing bandwidth usage. These video slices can be encrypted before uploading to reduce the risk of privacy leaks.

[0103] II. Cloud-based Multimodal Large Model Engine: This engine comprises two parts: 1) Extracting keyframe sequences from video slices; 2) Extracting and analyzing features from the keyframe sequences output in the first part, producing descriptive text for the subject, behavior, and environment, as well as subdivided event categories, and providing specific protection reminders or suggestions for each event. Specifically:

[0104] (1) Extract the keyframe sequence from the video;

[0105] (2) Determine if the keyframe sequence is abnormal; if it is determined that the keyframe sequence is abnormal, mark the abnormality and perform a downgraded simplified analysis; if it is determined that the keyframe sequence is not abnormal, continue to execute the subsequent steps;

[0106] (3) Perform feature extraction and deep semantic analysis of video frame sequences;

[0107] (4) Perform multi-dimensional recognition to obtain subject attribute description, behavior action description, and environmental scene description (wherein, descriptive text includes but is not limited to: subject attribute description, behavior action description, and environmental scene description), and determine the subdivided event category of the preset event;

[0108] (5) Event risk level determination:

[0109] If the event is determined to be a low-risk / accidental event, i.e. a low-priority event, it should be recorded and archived.

[0110] If the event is determined to be a medium-risk event, i.e. a routine event, then the routine event intervention channel will be activated (enter the personalized interaction and intervention center).

[0111] If an event is determined to be a high-risk emergency event, i.e. a high-priority event, the high-priority event intervention channel will be activated.

[0112] In summary, the cloud (i.e., the server) inputs video slices along with synchronously reported environmental data (temperature and humidity, door lock status, wearable device heart rate, smart speaker voice logs) into a multimodal large model based on the Transformer architecture (such as LLaVA-Video or Customized Video-LLaMA). The multimodal large model possesses the following capabilities: Video semantic understanding: recognizing actions (standing, falling, rolling), objects (elderly, children, pets), and interactions (biting the sofa, petting the pet); Context fusion: combining time (3 AM, 2 PM, etc.), historical behavior (the elderly person naps daily at 3 PM for the past 30 days), and device status (air conditioner off, curtains closed) for comprehensive judgment; Risk classification modeling: based on a pre-trained knowledge base and user profiles, risk scores are assigned to events (levels 1–5): Level 5 (high risk, i.e., third risk level): The elderly person has not moved for 2 hours, the room temperature is abnormal, and the heart rate sensor is silent, triggering an emergency call to notify family members (i.e., the target contact person); Level 3 (medium risk, i.e., second risk level): The pet continues to bite the sofa, and the sofa material sensor alarms, sending a notification to a regular application (APP), for example: suggesting the purchase of an anti-scratch mat; Level 1 (low risk, i.e., first risk level): A child runs around in the living room and the lights are dimmed, without alerting, only recorded as an "active period".

[0113] The optimal intervention method is automatically selected based on the risk level.

[0114] Level 5: Automatically dials emergency contacts and simultaneously pushes video clips and health data;

[0115] Level 3: Push notifications to the app and a voice reminder from the smart speaker: "Pay attention to pet behavior";

[0116] Level 1: Background records, used to optimize user behavior models.

[0117] Third, the Personalized Interaction and Intervention Center is used to generate more comprehensive content and more personalized, humanistic copywriting styles based on information output from the cloud, so that users receive more user-friendly and comprehensive message notifications.

[0118] If the event is determined to be a medium-risk event, i.e. a routine event, a message notification is sent to the user, and it is determined whether the user has responded and handled it in a timely manner; if it is determined that the user has responded and handled it in a timely manner, the event is closed in a closed loop, and a complete event log is maintained; if it is determined that the user has not responded and handled it in a timely manner, a tiered secondary reminder is issued, and the alert is upgraded to a fallback warning.

[0119] Meanwhile, optional embodiments of this application can generate health reports based on preset event cycles, such as weekly, and analyze sleep patterns, activity frequency, abnormal event trends, etc.

[0120] In summary, the embodiments of this application significantly reduce the false alarm rate: through contextual understanding (such as "elderly person taking a nap"), it avoids traditional systems misjudging normal behavior as abnormal, improving alarm accuracy by over 60% (actual test data); it enhances user trust and reliance: personalized and warm reminders encourage users to shift from "turning off alarms" to "actively viewing reports," increasing system stickiness; it enables proactive health protection: for high-risk groups such as elderly people living alone and patients with chronic diseases, intervention can be triggered before a crisis occurs, reducing the risk of accidental injury or death; it is resource-efficient and privacy-secure: edge processing reduces bandwidth and cloud computing power consumption; uploaded data does not contain original video, complying with privacy compliance trends.

[0121] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0122] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0123] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0124] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A security system based on image recognition, characterized in that, include: Target devices and servers, including: The target device is used to perform event detection on the video stream to obtain video slices containing preset events, and to send the video slices and the corresponding environmental data to the server. The preset events include at least one of the following: an event that detects a target subject, or a preset abnormal event. The server is configured to extract video frame sequences from the video slices; perform feature extraction on the video frame sequences and the environmental data to determine the descriptive text and event category corresponding to the preset event; and identify the risk level corresponding to the preset event based on the descriptive text and the event category, wherein the descriptive text is used to indicate the subject attributes, behaviors, and environmental state when the preset event occurs.

2. The security system based on image recognition according to claim 1, characterized in that, The target device is also used for: Each video frame in the video stream is detected according to the target detection model to determine a first detection result corresponding to each video frame. The first detection result includes at least one of the following: whether each video frame contains a target detection box, the category label of the target detection box, the first confidence level corresponding to the category label of the target detection box, and the target detection box is the detection box corresponding to the target subject. Based on the first detection result, a first video frame set is determined in the video stream, wherein the first detection result of each video frame in the first video frame set indicates that the corresponding video frame contains the target detection box, and the category label of the target detection box is the target category label, and the first confidence level of the target category label is greater than or equal to the first confidence threshold of the first video frame set. Based on the first set of video frames, video slices containing events that detected the target subject are filtered from the video stream.

3. The security system based on image recognition according to claim 1, characterized in that, The target device is also used for: Determine the pixel change rate between each video frame in the video stream and the previous video frame for each video frame; In the video stream, determine a second set of video frames in which the pixel change rate is greater than or equal to the change rate threshold; Based on the second set of video frames, video slices containing preset abnormal events are selected from the video stream.

4. The security system based on image recognition according to claim 1, characterized in that, The server is also used for: Based on a preset step size, multiple first video frames are obtained in the video slice, and the first visual difference between each two adjacent first video frames in the multiple first video frames is calculated. The probability distribution of the image category corresponding to each first video frame is predicted based on the classification model. Calculate the entropy value and second confidence level corresponding to the probability distribution of each first video frame; Calculate the first semantic richness score for each first video frame based on the entropy value and the second confidence level; The video frame sequence is extracted from the video slice based on the first visual dissimilarity score and the first semantic richness score.

5. The security system based on image recognition according to claim 4, characterized in that, The server is also used for: In the plurality of first video frames, a third set of video frames is determined in which the first visual difference is greater than or equal to a preset difference, and / or the first semantic richness score is greater than or equal to a first score threshold. Determine whether the number of video frames contained in the third video frame set is within a preset range; If the number of video frames contained in the third video frame set is within the preset range, the video frame sequence is constructed based on the video frames contained in the third video frame set.

6. The security system based on image recognition according to claim 5, characterized in that, The server is also used for: If the number of video frames in the third video frame set is less than the minimum value of the preset number range, multiple second video frames are obtained in the video slice based on the preset step size, and the second visual difference degree between each two adjacent second video frames and the second semantic richness score of each third video frame are calculated. In the plurality of second video frames, at least one third video frame is determined to have a second visual difference degree greater than or equal to the preset difference degree, and each third video frame is added to the third video frame set in sequence according to the second semantic richness of the at least one third video frame, until the number of video frames contained in the third video frame set is within the preset number range; The video frame sequence is constructed based on the set of third video frames after the addition of the third video frame.

7. The security system based on image recognition according to claim 1, characterized in that, The server is also used for: The video frame sequence is encoded based on a preset encoder to determine the visual embedding vector corresponding to the video frame sequence, and to determine the environmental feature vector corresponding to the environmental data. Based on the target attention mechanism, the visual embedding vector is determined as the query vector, the environmental feature vector and the temporal context data corresponding to the video frame sequence are determined as the key and value, and the visual embedding vector and the environmental feature vector are fused according to the query vector, the key and value to generate fused features; The fusion features and the historical behavior data of the target subject are input into the inference model, so that the inference model generates the descriptive text and the event category through causal inference.

8. The security system based on image recognition according to claim 1, characterized in that, The server is also used for: Extract at least one semantic keyword and the weight corresponding to each semantic keyword from the descriptive text; A first risk score is calculated for the descriptive text based on the at least one semantic keyword and the weight corresponding to each semantic keyword, and a second risk score is calculated for the event category. Calculate the target score corresponding to the preset event based on the first risk score and the second risk score; The risk level corresponding to the preset event is determined based on the target score.

9. The security system based on image recognition according to claim 8, characterized in that, The server is also used for: Obtain the target subject's historical behavior data and the target subject's health status tag; The target score is corrected based on the historical behavioral data and the health status tags; The corrected target score is matched with the score range corresponding to each risk level to determine the risk level corresponding to the preset event.

10. The security system based on image recognition according to claim 1, characterized in that, The server is also used for: If the risk level corresponding to the preset event is the first risk level, cache the descriptive text and the event category; When the risk level is the second risk level, target notification information is generated based on the descriptive text and the event category, and the target notification information is sent to the target contact person corresponding to the target subject; When the risk level is the third risk level, a call request is sent to the target contact. If the target contact responds to the call request, the preset event is sent to the target contact. The priority of the first risk level is lower than that of the second risk level, and the priority of the second risk level is lower than that of the third risk level.

Images