A home monitoring method and system based on multi-agent large model

Abstract

This invention discloses a home monitoring method and system based on a multi-agent large model. The multi-agent large model includes a central large model and agents. The central large model is used to schedule at least one agent to perform monitoring, analysis, and alarm tasks. It reads video stream data from home security cameras in real time, preprocesses the source data to obtain a unified video format, and inputs the processed video data into the multi-agent large model. The central large model then dispatches tasks to at least one agent, which performs corresponding monitoring, analysis, and alarm operations according to the task. By processing the real-time camera video streams in a unified format, the system ensures data accuracy. Through a multi-agent collaborative decision-making mechanism, combined with the intelligent analysis and reasoning capabilities of a large language model, it achieves monitoring, identification, and intelligent alarming of abnormal situations in the home environment, significantly improving the reliability, intelligence, and user experience of the home security monitoring system.

Description

Technical Field

[0001] This invention belongs to the field of home monitoring technology, and in particular relates to a home monitoring method and system based on a multi-agent large model. Background Technology

[0002] As living standards continue to improve, people are paying increasing attention to home security, and home monitoring systems are becoming increasingly popular. Traditional home monitoring systems mainly rely on single sensors or cameras for passive monitoring, which suffers from many problems such as slow response times, high false alarm rates, and insufficient intelligence. For example, ordinary cameras can only provide video recording and lack intelligent analysis capabilities; infrared sensors are easily affected by environmental interference, leading to false alarms; and standalone smoke detectors cannot be linked with other devices, making it difficult to achieve comprehensive early warning. Furthermore, traditional home monitoring systems cannot provide real-time notifications to family members and take early warning measures for abnormal events in the home environment, such as pets escaping or elderly people falling, failing to meet the modern family's needs for security, convenience, and intelligence. Summary of the Invention

[0003] The purpose of this invention is to provide a home monitoring method and system based on a multi-agent large model, so as to solve the technical problems of traditional monitoring systems, such as lag, high false alarm rate and insufficient intelligence.

[0004] To solve the above-mentioned technical problems, the specific technical solution of the present invention is as follows:

[0005] In some embodiments of this application, a home monitoring method based on a multi-agent large model is provided, including the following steps:

[0006] S1. Construct a multi-agent large model, which includes a central large model and at least one agent. The central large model is used to schedule the at least one agent to perform monitoring, analysis and alarm tasks.

[0007] S2. Read video stream data from home security cameras in real time, preprocess the source data, and obtain unified video format data;

[0008] S3. Input the processed video data into the multi-agent big model, and the central big model dispatches tasks to at least one agent. The at least one agent performs corresponding monitoring, analysis and alarm operations according to the tasks.

[0009] In some embodiments of this application, step S1 includes: selecting a large language model as the central large model, constructing at least one agent including a video analysis agent, an anomaly handling agent, and a RAG dialogue agent, and connecting the at least one agent to the central large model.

[0010] In some embodiments of this application, step S2 includes: creating a coroutine to read video stream data from a home camera and reading frame images, then processing the frame images to convert them into uint8 format, or if the image is a grayscale image, converting it into BGR format, and storing the processed frame images in a frame image buffer.

[0011] In some embodiments of this application, in step S3, the central large model dispatches analysis tasks to the video analysis agent, and the video analysis agent performs the following operations:

[0012] Create a video writer to convert the formatted frame data into Base64 encoded video;

[0013] Configure video analytics prompt and anomaly analysis prompt;

[0014] The video segment and the video analysis prompt are encapsulated into a request body, and the visual big model is called through the API to obtain the text description of the current video segment;

[0015] Read the text descriptions of recent historical video clips and generate a historical summary;

[0016] Based on the aforementioned anomaly analysis prompt, combined with the current video description and historical summaries, a comprehensive judgment is made as to whether any abnormal phenomena have occurred inside the room.

[0017] The analysis results are returned to the central large model.

[0018] In some embodiments of this application, if the video analysis agent determines that an anomaly has occurred, the central big model triggers an anomaly handling agent, which performs the following operations:

[0019] Call the system backend's SMS and email sending interfaces;

[0020] Create a T wilio client to notify family members of unusual information during the current time period via a virtual number;

[0021] Establish an SMTP connection and send an email containing abnormal information. The SMS and email contain the current time period, the type of abnormality, and a text description of the video.

[0022] In some embodiments of this application, the video analysis task employs a sliding window mechanism, whereby the central large model reads video frames at time interval x+1 from the frame image buffer and then passes the set of frame data into the video analysis agent.

[0023] Some embodiments of this application also include the following steps:

[0024] S4. Construct a vector knowledge base to store video analysis information and early warning information;

[0025] The RAG dialogue agent converts user input into vectors, performs similarity matching with vectors in the vector database, and returns the k most similar vectors. The RAG dialogue agent combines the acquired vector information to realize dialogue and query based on historical video information.

[0026] In some embodiments of this application, the vector knowledge base uses the Chroma vector database, and step S4 includes:

[0027] Create a word vector embedding model to convert the video segment text descriptions, anomaly analysis results, and video history summaries obtained by the video analysis agent into word vectors for storage, and persist the database in the system storage folder.

[0028] A home monitoring system based on a multi-agent large model, employing the aforementioned monitoring method, includes:

[0029] The video processing module is used to read data from home security cameras in real time, perform video frame preprocessing, obtain a unified data format, and store it in the system frame buffer.

[0030] The central large model module is used to connect multiple agents and also serves as the system control center, executing global task dispatch, receiving results returned by agents and making decisions.

[0031] The multi-agent module includes a video analytics agent, an anomaly handling agent, and a RAG dialogue agent;

[0032] The backend interface module is used to provide services such as sending system emails, sending SMS messages, adding, deleting, modifying, and querying vector knowledge bases, and calling large models;

[0033] The central big model module and the multi-agent module constitute the multi-agent big model of the system, and the central big model module performs task scheduling.

[0034] In some embodiments of this application, the agent in the multi-agent module is a ReAct agent. The ReAct agent generates an analysis path by setting inference steps, calls tools or interfaces to solve tasks by setting action steps, sets a loop path, and performs inference and action steps in a loop until the task is completed.

[0035] The central big model module uses the LangGraph framework to build a coordinator multi-agent framework. The central big model acts as the central coordinator, controlling all communication flows and task delegations. It decides to call agents based on the current context and task requirements. The system uses global state information for communication between agents and the central big model.

[0036] The backend interface module uses the FastAPI framework as the system backend framework to implement interfaces such as email sending, SMS sending, vector knowledge base CRUD operations, and large model calls. Different tasks are grouped using different paths, and a unified corresponding format is constructed for the interfaces. A global exception handler is built to handle interface call exceptions.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] By leveraging large models and multi-agent collaboration, the system achieves deep understanding and intelligent analysis of complex home scenarios. Combined with historical information and multimodal analysis, it effectively reduces false alarm rates and improves the accuracy of anomaly identification. Furthermore, the system proactively and promptly notifies family members when anomalies are detected, transforming passive monitoring into proactive early warning and avoiding delays. Through RAG dialogue agents, users can easily query historical monitoring information without manually reviewing videos, saving time and effort. The multi-agent architecture allows for easy addition of new agents or adjustment of existing agents' prompts to adapt to more home scenarios (such as water leakage detection and gas leak detection). The clear division of labor among the multiple agents and their collaborative work are more efficient than a single model handling all tasks. Moreover, by processing real-time camera video streams in a unified format, the system ensures data accuracy. Through a multi-agent collaborative decision-making mechanism, combined with the intelligent analysis and reasoning capabilities of large language models, it achieves monitoring, identification, and intelligent alarms for anomalies in the home environment. This significantly improves the reliability, intelligence, and user experience of the home security monitoring system, realizing intelligent, precise, and proactive home security monitoring. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0040] Figure 1 A schematic diagram of the main process of the home monitoring method based on a multi-agent large model provided in an embodiment of the present invention;

[0041] Figure 2 This is a schematic diagram of the framework of a home monitoring system based on a multi-agent large model provided in an embodiment of the present invention. Detailed Implementation

[0042] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0043] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0044] See appendix Figure 1-2 As shown, this embodiment of the invention proposes a home monitoring method based on a multi-agent large model, such as... Figure 1 As shown, the home monitoring method based on a multi-agent large model includes:

[0045] S1. Using DeepSeek-V3 as the base model, a multi-agent large model is constructed as the control center for home monitoring tasks.

[0046] S2. Read video stream data from home security cameras in real time, preprocess the source data, and obtain unified video format data.

[0047] S3. Input the processed video data into the multi-agent big model. The central big model dispatches tasks to the video analysis agent. The agent generates the current video description result and combines it with historical information to determine whether an anomaly has occurred at home. If so, the anomaly handling agent is triggered to execute the early warning push task and notify the family members. Otherwise, the monitoring task continues.

[0048] S4. Construct a vector knowledge base to store video analysis information and early warning information, and realize the dialogue and query functions of RAG dialogue agent based on historical video information.

[0049] The working principle of the above technical solution is as follows: For step S1:

[0050] DeepSeek-V3 was chosen as the system's foundation model and central large model. ReAct agents were constructed, including three major agents: a video analysis agent, a RAG dialogue agent, and an anomaly handling agent. These agents were used to monitor, analyze, and alert on the home environment. The three major agents were connected to the central large model, which received the data returned by the agents, made aggregated decisions, and assigned tasks, thus constructing a multi-agent large model.

[0051] For step S2:

[0052] Real-time data collection from home security cameras, such as doorway and living room cameras, is performed, followed by standardized format processing. First, a video reader is built to read frame images from the video stream. Then, the frame images are formatted by converting them to uint8 format; if the image is grayscale, it is converted to BGR format. After processing, the data is stored in a frame image buffer. This format processing method ensures the data conforms to the OpenCV standard format, better supporting subsequent video recognition.

[0053] For step S3, it also includes:

[0054] S3.1 The system runs continuously in the background. When the analysis interval is reached, the central big model dispatches the analysis task to the video analysis agent. The agent creates a video writer to convert the formatted frame data into Base64 encoded video. By constructing a video analysis prompt and anomaly analysis prompt, the agent is guided to perform standardized output. Then, the video segment and the video analysis prompt are encapsulated into a request body. The visual big model is called through the API to obtain the text description of the current video segment and read the text descriptions of recent historical video segments to generate a historical summary. Finally, the agent judges whether an abnormal phenomenon has occurred in the room based on the anomaly analysis prompt, the current video description, and the historical summary. The output format of the analysis result has been standardized in the prompt. If no abnormality has occurred, only "no abnormality" is output. If an abnormality such as an elderly person falling occurs, the abnormality type (in this example, the abnormality type is "person falls"), time period, and abnormal video segment description result will be returned to the central big model, which will then perform subsequent task scheduling.

[0055] These abnormal phenomena include, but are not limited to, elderly people falling at home, pets escaping, and fires breaking out.

[0056] In addition, to prevent a continuous action in the monitoring from being split into two parts for analysis (such as when an elderly person falls and the action happens to occur in the time interval between the two parts of the analysis), which could lead to misjudgment, a sliding window is set to ensure the continuity of each video analysis. By setting the analysis interval for each system analysis, for any independent camera in the home, when the analysis time interval x is reached, the central big model will trigger the allocation of video analysis tasks. The central big model will obtain the video frames of the time interval (x+1) from the frame image buffer of that camera, and then pass the frame data set into the video analysis agent.

[0057] S3.2 When the central big model receives anomaly information (such as a pet running away or a person falling) from the video analysis agent, it immediately triggers the anomaly handling agent to execute the anomaly handling task. First, the agent creates a Twilio client, reads the phone numbers of family members configured in the system, and sends a text message to family members via a virtual number to inform them of the anomaly information for the current time period. In addition, the agent also creates an SMTP connection, reads the email information of family members configured in the system, uses the anomaly type as the email subject, the current time period as the subtitle, and the video text description as the email body, and finally sends the constructed email to the family members.

[0058] For step S4:

[0059] By constructing a Chroma vector database, the text descriptions of video segments and video history summaries obtained by the video analysis agent are converted into word vectors and stored after each video analysis task is completed.

[0060] Specifically, a word vector embedding model is created to convert the text descriptions of video segments and summaries of video history into vectors, which are then stored in the system's vector database. The stored data enables the RAG dialogue agent to perform dialogue and query functions based on historical video information.

[0061] Specifically, when a user asks the central big data model whether any unusual problems have occurred in their home during a certain period of time, the central big data model will call the RAG dialogue agent. The agent will convert the key information in the user's input text into vectors, and perform similarity matching by calculating the cosine distance between the vector and the vectors in the system's vector database. The agent will then obtain the few vectors with the highest similarity. The agent will summarize the user's question and the returned vectors before returning an answer, thereby realizing the dialogue and query functions based on historical video information.

[0062] The beneficial effects of this invention are as follows: By preprocessing camera video stream data to obtain a unified video format, the processed video can be more easily recognized for content and is also compatible with the format requirements of the OpenCV library. By constructing a multi-agent large model, dividing multiple agents into different tasks, and using a central large model for task scheduling, the entire process of monitoring, analysis, and alarm in home surveillance is autonomously executed without external human assistance. Compared to monitoring systems based on a single large model, this offers higher operational efficiency and scalability. By storing the analysis results in a vector database, dialogue and query functions based on historical video information are implemented, allowing users to more easily obtain historical camera information and reducing the time cost of manual video retrieval. This invention improves the intelligence, accuracy, and proactivity of home security monitoring. Furthermore, as needs expand, new agents can be flexibly added or new large model prompts can be designed to adapt to more home scenarios and requirements.

[0063] This invention proposes a home monitoring system based on a multi-agent large model, such as... Figure 2 As shown, the home monitoring system based on a multi-agent large model includes: a video processing module, a central large model module, a multi-agent module, and a back-end interface module.

[0064] The video processing module is responsible for reading data from home security cameras and performing video frame preprocessing. First, it converts the frame image into uint8 format. If the image is grayscale, it converts it into BGR format. This method obtains a unified data format, which is then stored in the system frame buffer.

[0065] The central large model module is responsible for reading the video frame buffer, connecting multiple agents, and serving as the system control center. It executes global task dispatch and summarizes the returns from various agents to make overall decisions. It also serves as the user's dialogue window.

[0066] The multi-agent module includes a video analytics agent, an anomaly handling agent, and a RAG dialogue agent.

[0067] The backend interface module is responsible for providing services such as sending system emails and SMS messages, adding, deleting, modifying, and querying vector knowledge bases, and calling large models. It uses the FastAPI framework as the system's backend framework to build interfaces, groups different tasks using different paths, and builds interfaces with a unified corresponding format, including three fields: code, message, and data. It also builds a global exception handler to handle exceptions during interface calls.

[0068] Among them, the central big model module and the multi-agent module constitute the system's multi-agent big model. The system's multi-agent big model adopts a coordinator multi-agent framework, with the central big model acting as the central coordinator to control all communication flows and task delegations. It decides to call agents based on the current context and task requirements. The system uses global state information for communication between agents.

[0069] In addition, the multi-agent module includes:

[0070] The video analytics agent is responsible for analyzing the content monitored by home security cameras. When the video analytics task cycle is reached, it receives a set of video frames from the central big model, calls the visual big model API in the agent to generate a text description of the current video segment, reads the text descriptions of recent historical video segments to generate a summary of historical video segments, and finally combines the summary of historical time periods to make a comprehensive judgment on whether any abnormal phenomena have occurred in the home (such as whether an elderly person has fallen, a pet has escaped, or a fire has occurred), and returns the results to the central big model.

[0071] The anomaly handling agent is responsible for early warning tasks after an anomaly occurs. When the agent is triggered by the central big model, it reads the family member information configured by the system, calls the SMS and email sending backend interfaces, and sends the current time period, anomaly type, and video text description to family members via SMS and email, ensuring that family members can be informed of the anomaly at home in a timely and accurate manner.

[0072] The RAG dialogue agent is responsible for dialogue and query tasks based on historical video information. When a user asks the central big data model whether any abnormal problems have occurred in their home during a certain period of time, the central big data model triggers the agent. The agent converts the user's key input information into a vector, then calculates the cosine distance between the vector and the vector in the system's vector database, performs similarity matching, and then returns the several vectors with the highest similarity. Combining the obtained vectors with the user's question, a complete response is constructed and then returned to the central big data model.

[0073] The system's agent type is ReAct agent. The agent construction is divided into two steps. The first step is to set the reasoning steps and generate the analysis path by using the language understanding capability of the large model. The second step is to set the action steps and solve the task by calling tools or interfaces. A loop path is set to repeatedly perform reasoning and action steps until the agent completes the task and returns the task result to the central large model.

[0074] This embodiment is applicable to home monitoring scenarios, including but not limited to elderly falls, pet escapes, fires, and suspicious intrusions. Traditional home monitoring systems mainly rely on single sensors or cameras for passive monitoring, which suffers from problems such as slow response, high false alarm rates, and insufficient intelligence. Furthermore, compared to monitoring systems based on a single large model, this embodiment offers higher operational efficiency and scalability. This embodiment preprocesses the video sources from home cameras and constructs a multi-agent large model to divide and schedule monitoring, analysis, and early warning tasks. This not only frees up manpower in traditional monitoring systems but also provides a more intelligent and reliable monitoring and early warning solution for the home environment.

[0075] In the description of this application, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0076] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0077] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0079] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A home monitoring method based on a multi-agent large model, characterized in that, Includes the following steps: S1. Construct a multi-agent large model, which includes a central large model and at least one agent. The central large model is used to schedule the at least one agent to perform monitoring, analysis and alarm tasks. Specifically, a large language model is selected as the central large model, and at least one agent is constructed, including a video analysis agent, an anomaly handling agent, and a RAG dialogue agent. The at least one agent is then connected to the central large model. S2. Read video stream data from home security cameras in real time, preprocess the source data, and obtain unified video format data; S3. Input the processed video data into the multi-agent big model, and the central big model dispatches tasks to at least one agent. The at least one agent performs corresponding monitoring, analysis and alarm operations according to the task. The central big model dispatches analysis tasks to the video analysis agent, which then performs the following operations: Create a video writer to convert the formatted frame data into Base64 encoded video; Configure video analytics prompt and anomaly analysis prompt; The video segment and the video analysis prompt are encapsulated into a request body, and the visual big model is called through the API to obtain the text description of the current video segment; Read the text descriptions of recent historical video clips and generate a historical summary; Based on the aforementioned anomaly analysis prompt, combined with the current video description and historical summaries, a comprehensive judgment is made as to whether any abnormal phenomena have occurred inside the room. The analysis results are returned to the central macro-model. Step S2 includes: Create a coroutine to read video stream data from a home security camera and read frame images. Then process the frame images by converting them to uint8 format. If the image is grayscale, convert it to BGR format. After processing the frame images, store them in the frame image buffer. If the video analysis agent determines that an anomaly has occurred, the central big model triggers an anomaly handling agent, which performs the following operations: Call the system backend's SMS and email sending interfaces; Create a Twilio client to notify family members of unusual information during the current time period via a virtual number; Establish an SMTP connection and send an email containing abnormal information. The SMS and email contain the current time period, the type of abnormality, and a text description of the video. The video analysis task uses a sliding window mechanism. The central big model reads video frames at time interval x+1 from the frame image buffer, and then passes the frame data set into the video analysis agent. It also includes the following steps: S4. Construct a vector knowledge base to store video analysis information and early warning information; The RAG dialogue agent converts user input into vectors, performs similarity matching with vectors in the vector database, and returns the k most similar vectors. The RAG dialogue agent combines the acquired vector information to realize dialogue and query based on historical video information. The vector knowledge base uses the Chroma vector database, and step S4 includes: Create a word vector embedding model to convert the video segment text descriptions, anomaly analysis results, and video history summaries obtained by the video analysis agent into word vectors for storage, and persist the database in the system storage folder.

2. A home monitoring system based on a multi-agent large model, employing the monitoring method of claim 1, characterized in that, include: The video processing module is used to read data from home security cameras in real time, perform video frame preprocessing, obtain a unified data format, and store it in the system frame buffer. The central large model module is used to connect multiple agents and also serves as the system control center, executing global task dispatch, receiving results returned by agents and making decisions. The multi-agent module includes a video analytics agent, an anomaly handling agent, and a RAG dialogue agent; The backend interface module is used to provide services such as sending system emails, sending SMS messages, adding, deleting, modifying, and querying vector knowledge bases, and calling large models; The central big model module and the multi-agent module constitute the multi-agent big model of the system, and the central big model module performs task scheduling.

3. The system according to claim 2, characterized in that, The agent in the multi-agent module is a ReAct agent. The ReAct agent generates an analysis path by setting inference steps, calls tools or interfaces to solve the task by setting action steps, sets a loop path, and performs inference and action steps in a loop until the task is completed. The central big model module uses the LangGraph framework to build a coordinator multi-agent framework. The central big model acts as the central coordinator, controlling all communication flows and task delegations. It decides to call agents based on the current context and task requirements. The system uses global state information for communication between agents and the central big model. The backend interface module uses the FastAPI framework as the system backend framework to implement interfaces such as email sending, SMS sending, vector knowledge base CRUD operations, and large model calls. Different tasks are grouped using different paths, and a unified corresponding format is constructed for the interfaces. A global exception handler is built to handle interface call exceptions.

Images