Home monitoring method and system based on multi-agent large model

Abstract

The invention discloses a home monitoring method and system based on a multi-agent large model, the multi-agent large model comprises a central large model and agents, and the central large model is used for scheduling at least one agent to execute monitoring, analysis and alarm tasks; the method comprises the following steps: reading video stream data of a household camera in real time, and preprocessing source data to obtain unified video format data; and inputting the processed video data into the multi-agent large model, distributing a task to at least one agent by the central large model, and executing corresponding monitoring, analysis and alarm operations by the at least one agent according to the task. By performing uniform format processing on real-time read camera video streams, the accuracy of system data is ensured, and monitoring identification and intelligent alarm of abnormal conditions in a home environment are realized through a multi-agent collaborative decision-making mechanism in combination with intelligent analysis and reasoning capabilities of a large language model. And the reliability, the intelligence, the user experience and the like of the home safety monitoring system are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of home monitoring technology, and in particular relates to a home monitoring method and system based on a multi-agent large model. Background Art

[0002] With rising living standards, people are placing increasing emphasis on home security, and home surveillance systems are becoming increasingly popular. Traditional home surveillance systems primarily rely on passive monitoring with single sensors or cameras, which can suffer from numerous issues such as delayed response, high false alarm rates, and insufficient intelligence. For example, ordinary cameras only provide video recording and lack intelligent analysis capabilities; infrared sensors are susceptible to environmental interference, leading to false alarms; and standalone smoke alarms lack the ability to interact with other devices, making comprehensive early warning difficult. Furthermore, traditional home surveillance systems are unable to provide real-time notification and early warning measures for unusual events such as pet escapes and elderly people falling, making them unable to meet the demands of modern families for security, convenience, and intelligence. Summary of the Invention

[0003] The present invention aims to provide a home monitoring method and system based on a multi-agent large model to solve the technical problems of traditional monitoring systems such as hysteresis, high false alarm rate and insufficient intelligence.

[0004] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows:

[0005] In some embodiments of the present application, a home monitoring method based on a multi-agent large model is provided, comprising the following steps:

[0006] S1. Build a multi-agent large model, the multi-agent large model including a central large model and at least one agent, the central large model being used to schedule the at least one agent to perform monitoring, analysis, and alarm tasks;

[0007] S2, read the video stream data of the home camera in real time, pre-process the source data, and obtain the data in a unified video format;

[0008] S3. Input the processed video data into the multi-agent big model, and the central big model assigns tasks to at least one agent, and the at least one agent performs corresponding monitoring, analysis and alarm operations according to the tasks.

[0009] In some embodiments of the present application, step S1 includes: selecting a large language model as the central large model, constructing at least one agent including at least a video analysis agent, an exception handling agent and a RAG dialogue agent, and connecting the at least one agent to the central large model.

[0010] In some embodiments of the present application, step S2 includes: creating a coroutine to read the video stream data of the home camera, and reading the frame image, then formatting the frame image, converting the image into uint8 format, and if the image is a grayscale image, converting it into BGR format, and storing it in the frame image buffer after the frame image processing is completed.

[0011] In some embodiments of the present application, in step S3, the central large model dispatches the analysis task to the video analysis agent, and the video analysis agent performs the following operations:

[0012] Create a video writer to convert the formatted frame data into a Base64-encoded video;

[0013] Configure the video analysis prompt and anomaly analysis prompt;

[0014] Encapsulate the video segment and the video analysis prompt into a request body, call the visual model through the API, and obtain the text description of the current video segment;

[0015] Read the text description of recent historical video segments and generate a historical summary;

[0016] Based on the abnormality analysis prompt, combine the current video description and historical summary to comprehensively determine whether abnormal phenomena have occurred in the house;

[0017] The analysis results are returned to the central macro model.

[0018] In some embodiments of the present application, if the video analysis agent determines that an abnormality has occurred, the central large model triggers the abnormality handling agent, and the abnormality handling agent performs the following operations:

[0019] Call the system backend SMS sending and email sending interfaces;

[0020] Create a Twilio client and notify family members of abnormal information in the current time period through a virtual number;

[0021] Create an SMTP connection and send an email containing abnormal information. The text message and email include the current time period, abnormality type, and video text description.

[0022] In some embodiments of the present application, the video analysis task uses a sliding window mechanism, and the central large model reads video frames at a time interval x+1 from the frame image buffer, and then passes the frame data set into the video analysis agent.

[0023] In some embodiments of the present application, the steps are further included:

[0024] S4. Build a vector knowledge base to store video analysis information and warning information;

[0025] The RAG dialogue agent converts user input into vectors, performs similarity matching with vectors in the vector database, and returns the k vectors with the highest similarity. The RAG dialogue agent combines the obtained vector information to implement dialogue and query based on historical video information.

[0026] In some embodiments of the present application, the vector knowledge base adopts the Chroma vector database, and step S4 includes:

[0027] Create a word vector embedding model, convert the text description of the video segment, anomaly analysis results, and video history summary obtained by the video analysis agent into word vectors for storage, and save the database persistently in the system storage folder.

[0028] A home monitoring system based on a multi-agent large model, using the above monitoring method, includes:

[0029] The video processing module is used to read the home camera data in real time, perform video frame preprocessing, obtain a unified data format, and store it in the system frame buffer;

[0030] The central large model module is used to connect multiple intelligent agents and serve as the system control center, executing global task dispatch, receiving the results returned by the intelligent agents and making decisions;

[0031] Multi-agent module, including video analysis agent, exception handling agent and RAG dialogue agent;

[0032] The back-end interface module is used to provide services such as system email sending, SMS sending, vector knowledge base addition, deletion, modification and query, and large model calling;

[0033] Among them, the central large model module and the multi-agent module constitute the multi-agent large model of the system, and the central large model module performs task scheduling.

[0034] In some embodiments of the present application, the agent in the multi-agent module is a ReAct agent, which generates an analysis path by setting reasoning steps, calls a tool or calls an interface to solve a task by setting action steps, and sets a loop path to loop through reasoning and action steps until the task is completed;

[0035] The central large model module uses the LangGraph framework to build a coordinator multi-agent framework. The central large model acts as a central coordinator, controlling all communication flows and task delegations, and deciding to call agents based on the current context and task requirements. The system uses global state information for communication between agents and the central large model.

[0036] The back-end interface module uses the FastAPI framework as the system back-end framework to implement interfaces such as email sending, SMS sending, vector knowledge base addition, deletion, modification and query, and large model calling. It adopts different path groupings for different tasks, builds a unified corresponding format for the interface, and builds a global exception handler to implement interface call exception handling.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] By leveraging large models and multi-agent collaboration, we achieve a deep understanding and intelligent analysis of complex home scenarios, combined with historical information and multimodal analysis, effectively reducing the false alarm rate and improving the accuracy of anomaly identification. Furthermore, the system proactively and promptly notifies family members when an anomaly is detected, transforming passive monitoring into active warnings and avoiding lags. Through the RAG dialogue agent, users can easily query historical monitoring information without having to manually flip through videos, saving time and energy. The multi-agent architecture makes it easy to add new agents or adjust the prompts of existing agents based on new needs, adapting to more home scenarios (such as flood detection, gas leaks, etc.). The multi-agents have clear division of labor and work collaboratively, making them more efficient than a single model in handling all tasks. Furthermore, by uniformly formatting the real-time camera video stream, the accuracy of the system data is ensured. Through the multi-agent collaborative decision-making mechanism, combined with the intelligent analysis and reasoning capabilities of the large language model, monitoring, identification, and intelligent warning of abnormal situations in the home environment are achieved, significantly improving the reliability, intelligence, and user experience of the home security monitoring system, and achieving intelligent, precise, and proactive home security monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0040] Figure 1 A schematic diagram of the main process of a home monitoring method based on a multi-agent large model provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of the framework of a home monitoring system based on a multi-agent large model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0043] In order to better understand the purpose, structure and function of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0044] See attached Figure 1-2 As shown, the embodiment of the present invention proposes a home monitoring method based on a multi-agent large model, such as Figure 1 As shown, the home monitoring method based on the multi-agent large model includes:

[0045] S1. Using DeepSeek-V3 as the base model, we build a large multi-agent model as the control center for home monitoring tasks.

[0046] S2. Read the video stream data of the home camera in real time, pre-process the source data, and obtain the unified video format data.

[0047] S3. The processed video data is input into the multi-agent large model, and the central large model assigns tasks to the video analysis agent. The agent generates the current video description result and combines historical information to determine whether an abnormality has occurred in the home. If so, the abnormality handling agent is triggered to execute the early warning push task and inform family members. Otherwise, the monitoring task continues.

[0048] S4. Build a vector knowledge base to store video analysis information and warning information, and realize the dialogue and query functions of the RAG dialogue agent based on historical video information.

[0049] The working principle of the above technical solution is as follows: for step S1:

[0050] DeepSeek-V3 is selected as the system base model and the central model to build the ReAct agent, including three major agents: video analysis agent, RAG dialogue agent, and exception handling agent, to realize tasks such as monitoring, analysis, and alarm of the home environment. The three major agents are connected to the central model, which receives the data returned by the agents and makes summary decisions and task allocation to construct a multi-agent model.

[0051] For step S2:

[0052] Real-time data from home cameras, such as those monitoring entrances and indoor living rooms, is then processed into a unified format. First, a video reader is built to read frames from the video stream. These frames are then formatted, converting them to uint8 format. Grayscale images are converted to BGR format. After processing, the frames are stored in the frame buffer. This formatting ensures that the data conforms to the OpenCV standard, better supporting subsequent video recognition.

[0053] Step S3 specifically includes:

[0054] S3.1. The system runs continuously in the background. When the analysis time interval is reached, the central model dispatches the analysis task to the video analysis agent. The agent creates a video writer and converts the formatted frame data into a Base64-encoded video. It guides the agent to perform standardized output by constructing a video analysis prompt and an exception analysis prompt. Then, the video segment and the video analysis prompt are encapsulated as a request body. The visual model is called through the API to obtain the text description of the current video segment, and the text description of the recent historical video segments is read to generate a historical summary. Finally, the agent judges whether an abnormal phenomenon occurs in the house based on the exception analysis prompt, the current video description and the historical summary. The output format of the analysis result has been standardized in the prompt. If no abnormality occurs, only "no abnormality" will be output. If an abnormal situation such as an elderly person falling occurs, the abnormality type (in this case, the output abnormality type is a person falling), time period and abnormal video segment description will be returned to the central model, and the central model will perform subsequent task scheduling.

[0055] Among them, abnormal phenomena include but are not limited to elderly people falling at home, pets running away, fires, etc.

[0056] In addition, in order to prevent a coherent action in monitoring from being split into two-end analysis (such as when the elderly person falls and the action happens to occur in the time interval of the analysis at both ends), which leads to misjudgment, a sliding window is set to ensure the consistency of the content of each video analysis. By setting the analysis interval of each system, for any independent camera in the home, when the analysis time interval x is reached, the central large model will trigger the allocation of video analysis tasks. The central large model will obtain the video frames of the time interval (x+1) from the frame image buffer of the camera, and then pass the frame data set into the video analysis agent.

[0057] S3.2. When the central model receives abnormal information (such as a pet escaping or a person falling) from the video analysis agent, it will immediately trigger the exception handling agent to perform the exception handling task. First, the agent will create a Twilio client, read the family member's phone number configured in the system, and send a text message to the family member through the virtual number to inform them of the abnormal information in the current time period. In addition, the agent will also create an SMTP connection, read the family member's email information configured in the system, and use the abnormality type as the email title, the current time period as the subtitle, and the video text description as the email body. Finally, the constructed email will be sent to the family member.

[0058] For step S4:

[0059] By building a Chroma vector database, the text description of the video segment and the video history summary obtained by the video analysis agent are converted into word vectors for storage after each video analysis task is completed.

[0060] By creating a word embedding model, the text descriptions of video segments and video history summaries are converted into vectors and then stored in the system's vector database. This stored data supports the RAG conversational agent in conducting conversations and queries based on historical video information.

[0061] Specifically, when a user asks the central model whether any abnormal problems have occurred in the home during a certain period of time, the central model will call the RAG dialogue agent. The agent converts the key information in the text input by the user into a vector, and calculates the cosine distance between the vector and the vector in the system vector database for similarity matching to obtain several vectors with the highest similarity. The agent summarizes the user's questions and the returned vectors and then returns the answer, thereby realizing dialogue and query functions based on historical video information.

[0062] The beneficial effects of the present invention are: by pre-processing the camera video stream data to obtain a unified video format, the processed video can be more convenient for content recognition and can also be compatible with the format requirements of the OpenCV library. By constructing a multi-agent large model, building a multi-agent to divide different tasks, and using the central large model for task scheduling, the entire process of monitoring, analysis, and alarming in home monitoring can be autonomously executed without external human assistance. Compared with the monitoring system based on a single large model, it has higher operating efficiency and scalability. By storing the analysis results in a vector database, a dialogue and query function based on historical video information is realized, allowing users to more conveniently obtain camera historical information and reduce the time cost of manual video query. The invention improves the intelligence, precision and initiative of home security monitoring. At the same time, as the demand expands, new agents can be flexibly added or new large model prompts can be designed to adapt to more home scenarios and needs.

[0063] The embodiment of the present invention proposes a home monitoring system based on a multi-agent large model, such as Figure 2 As shown, the home monitoring system based on the multi-agent large model includes: a video processing module, a central large model module, a multi-agent module and a back-end interface module.

[0064] The video processing module is responsible for reading the data from the home camera and performing video frame preprocessing. First, the frame image is converted into uint8 format. If the image is a grayscale image, it is converted into BGR format. This method obtains a unified data format and then stores it in the system frame buffer.

[0065] The central large model module is responsible for reading the video frame buffer, connecting multiple intelligent agents, and serving as the system control center. It executes global task dispatch and aggregates the responses from each intelligent agent to make overall decisions. It is also the user's dialogue window.

[0066] Multi-agent module, including video analysis agent, exception handling agent and RAG dialogue agent.

[0067] The back-end interface module is responsible for providing services such as system email sending, SMS sending, vector knowledge base addition, deletion, modification and query, and large model calling. By using the FastAPI framework as the system back-end framework, the interface is built, different paths are grouped for different tasks, and a unified corresponding format is constructed for the interface, including three fields: code, message, and data. A global exception handler is built to implement interface call exception handling.

[0068] Among them, the central large model module and the multi-agent module constitute the system's multi-agent large model. The system's multi-agent large model adopts a coordinator multi-agent framework. The central large model serves as the central coordinator, controlling all communication flows and task delegations. It decides to call agents based on the current context and task requirements. The system uses global state information for communication between agents.

[0069] In addition, the multi-agent module includes:

[0070] The video analysis agent is responsible for analyzing the current home camera monitoring content. When the video analysis task cycle is reached, it receives a set of video frames input by the central large model, calls the visual large model API in the agent, generates a text description of the current video segment, and reads the text description of recent historical video segments to generate a summary of historical video segments. Finally, it combines the summary of historical time periods to comprehensively judge whether there are any abnormal phenomena in the home (such as whether the elderly have fallen, pets have escaped, fires, etc.), and returns the results to the central large model.

[0071] The exception handling agent is responsible for the early warning task after an exception occurs. When triggered by the central large model, the agent reads the family member information configured by the system, calls the SMS and email sending back-end interfaces, and sends the current time period, exception type and video text description content to family members via SMS and email, ensuring that family members can be informed of abnormal phenomena at home in a timely and accurate manner.

[0072] The RAG dialogue agent is responsible for dialogue and query tasks based on historical video information. When the user asks the central model whether any abnormal problems have occurred at home in a certain period of time, the central model triggers the agent. The agent converts the user's key input information into a vector, and then calculates the cosine distance between the vector and the vector in the system vector database, performs similarity matching, and then returns several vectors with the highest similarity. Combining the obtained vectors with the user's questions, it constructs a complete reply and returns it to the central model.

[0073] Among them, the system's intelligent agent type is ReAct intelligent agent. The intelligent agent construction is divided into two steps. The first step is to generate an analysis path by setting reasoning steps and using the language understanding ability of the large model. The second step is to set action steps and solve tasks by calling tools or calling interfaces. Set a loop path and loop the reasoning and action steps until the intelligent agent completes the task and returns the task results to the central large model.

[0074] This embodiment is suitable for home monitoring application scenarios, including but not limited to elderly people falling, pets escaping, fires, and suspicious intruders. Traditional home monitoring systems mainly rely on a single sensor or camera for passive monitoring, which has many problems such as delayed response, high false alarm rate, and insufficient intelligence. In addition, compared with monitoring systems based on a single large model, it has higher operating efficiency and scalability. This embodiment pre-processes the video source of home cameras and constructs a multi-agent large model to divide and dispatch monitoring, analysis, and early warning tasks. It not only liberates the human resources of traditional monitoring systems, but also provides a smarter and more reliable monitoring and early warning solution for home environments.

[0075] In the description of this application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application.

[0076] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature specified as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout this application, unless otherwise specified, "plurality" means two or more.

[0077] In the description of this application, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal connections between two components. Those skilled in the art will understand the specific meanings of the above terms in this application based on the specific circumstances.

[0078] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0079] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A home monitoring method based on a multi-agent large model, characterized in that: The following steps are involved: S1. Build a multi-agent large model, the multi-agent large model including a central large model and at least one agent, the central large model being used to schedule the at least one agent to perform monitoring, analysis, and alarm tasks; S2, read the video stream data of the home camera in real time, pre-process the source data, and obtain the data in a unified video format; S3. Input the processed video data into the multi-agent big model, and the central big model assigns tasks to at least one agent, and the at least one agent performs corresponding monitoring, analysis and alarm operations according to the tasks.

2. A home monitoring method based on a multi-agent large model according to claim 1, characterized in that: The step S1 includes: selecting a large language model as the central large model, constructing at least one agent including at least a video analysis agent, an exception handling agent and a RAG dialogue agent, and connecting the at least one agent to the central large model.

3. The method for home monitoring based on a multi-agent large model according to claim 1, characterized in that: The step S2 includes: creating a coroutine to read the video stream data of the home camera, and reading the frame image, then formatting the frame image, converting the image into uint8 format, and if the image is a grayscale image, converting it into BGR format, and storing the frame image in the frame image buffer after the frame image processing is completed.

4. The method for home monitoring based on a multi-agent large model according to claim 1, characterized in that: In step S3, the central large model dispatches the analysis task to the video analysis agent, and the video analysis agent performs the following operations: Create a video writer to convert the formatted frame data into a Base64-encoded video; Configure the video analysis prompt and anomaly analysis prompt; Encapsulate the video segment and the video analysis prompt into a request body, call the visual model through the API, and obtain the text description of the current video segment; Read the text description of recent historical video segments and generate a historical summary; Based on the abnormality analysis prompt, combine the current video description and historical summary to comprehensively determine whether abnormal phenomena have occurred in the house; The analysis results are returned to the central macro model.

5. The method for home monitoring based on a multi-agent large model according to claim 1, characterized in that: If the video analysis agent determines that an abnormality has occurred, the central model triggers the abnormality handling agent, which performs the following operations: Call the system backend SMS sending and email sending interfaces; Create a Twilio client and notify family members of abnormal information in the current time period through a virtual number; Create an SMTP connection and send an email containing abnormal information. The text message and email include the current time period, abnormality type, and video text description.

6. A home monitoring method based on a multi-agent large model according to claim 4, characterized in that: The video analysis task uses a sliding window mechanism. The central model reads video frames at a time interval of x+1 from the frame image buffer, and then passes the frame data set into the video analysis agent.

7. The method for home monitoring based on a multi-agent large model according to claim 1, characterized in that: Also includes the steps: S4. Build a vector knowledge base to store video analysis information and warning information; The RAG dialogue agent converts user input into vectors, performs similarity matching with vectors in the vector database, and returns the k vectors with the highest similarity. The RAG dialogue agent combines the obtained vector information to implement dialogue and query based on historical video information.

8. The method for home monitoring based on a multi-agent large model according to claim 7, characterized in that: The vector knowledge base adopts the Chroma vector database, and step S4 includes: Create a word vector embedding model, convert the text description of the video segment, anomaly analysis results, and video history summary obtained by the video analysis agent into word vectors for storage, and save the database persistently in the system storage folder.

9. A home monitoring system based on a multi-agent large model, using any monitoring method in claims 1-8, characterized in that: include: The video processing module is used to read the home camera data in real time, perform video frame preprocessing, obtain a unified data format, and store it in the system frame buffer; The central large model module is used to connect multiple intelligent agents and serve as the system control center, executing global task dispatch, receiving the results returned by the intelligent agents and making decisions; Multi-agent module, including video analysis agent, exception handling agent and RAG dialogue agent; The back-end interface module is used to provide services such as system email sending, SMS sending, vector knowledge base addition, deletion, modification and query, and large model calling; Among them, the central large model module and the multi-agent module constitute the multi-agent large model of the system, and the central large model module performs task scheduling.

10. The system according to claim 9, characterized in that The agent in the multi-agent module is a ReAct agent, which generates an analysis path by setting reasoning steps, calls a tool or calls an interface to solve a task by setting action steps, sets a loop path, and loops through reasoning and action steps until the task is completed; The central large model module uses the LangGraph framework to build a coordinator multi-agent framework. The central large model acts as a central coordinator, controlling all communication flows and task delegations, and deciding to call agents based on the current context and task requirements. The system uses global state information for communication between agents and the central large model. The back-end interface module uses the FastAPI framework as the system back-end framework to implement interfaces such as email sending, SMS sending, vector knowledge base addition, deletion, modification and query, and large model calling. It adopts different path groupings for different tasks, builds a unified corresponding format for the interface, and builds a global exception handler to implement interface call exception handling.

Images