Video surveillance system

Abstract

A video surveillance system includes: a user terminal for displaying video surveillance footage and receiving natural language commands input by the user; an AI intelligent agent core device for receiving the natural language commands, parsing the temporal semantics implied in the natural language commands, determining whether they belong to a past query, real-time analysis, or future warning, and generating corresponding monitoring tasks and model call commands based on the parsing results; and a cloud service and data layer device, including: a large language model (LLM), a visual language model (VLM), and a vector database. The AI ​​intelligent agent core device, based on temporal semantics, collaboratively calls the LLM, VLM, and vector database to perform semantic retrieval and temporal analysis of historical video events, semantic understanding and response to real-time video content, and continuous monitoring and warning triggering based on conditional rules. The video surveillance footage includes a smart assistant icon. When triggered, the user terminal automatically extracts the time node or visual frame of the currently displayed image as context and transmits it along with the natural language command to the AI ​​intelligent agent core device.

Description

Video surveillance system VIDEO SURVEILLANCE SYSTEM Technical Field

[0001] This work relates to a video surveillance system, particularly an artificial intelligence agent video surveillance system with temporal semantics at its core. Prior Technology

[0002] In the fields of intelligent surveillance and human-computer interaction, general-purpose artificial intelligence assistants (e.g., dialogue systems based on large language models) and existing surveillance camera applications have been widely used. However, existing technologies still have limitations in integrating contextual understanding of surveillance video data, supporting query needs across different time scales, and providing natural language interactive interfaces.

[0003] For example, while general-purpose AI assistants can parse natural language commands input by users and perform semantic reasoning, they usually cannot directly access or instantly integrate video data or historical records generated by specific monitoring equipment. In contrast, while traditional surveillance camera applications can access real-time or historical video data, they mostly only provide predefined operation interfaces and are difficult to parse natural language commands containing complex semantics or time conditions, thus limiting their application flexibility in human-computer interaction and intelligent analysis. Summary of the Invention

[0004] To overcome the aforementioned technical problems, this invention provides a video surveillance system, comprising: a user terminal for displaying video surveillance footage and receiving natural language commands input by the user; an AI intelligent agent core device communicatively connected to the user terminal, the AI ​​intelligent agent core device being configured to: receive the natural language commands; analyze the temporal semantics implied in the natural language commands and determine whether they belong to one of past queries, real-time analysis, or future warnings; and generate corresponding monitoring tasks and model call commands based on the analysis results; and a cloud service and data layer device, connected to the AI... The intelligent agent core device is connected via communication and includes: a Large Language Model (LLM) and a Visual Language Model (VLM), wherein: the VLM is configured to receive image data and generate corresponding semantic descriptions or event summaries; the LLM is configured to receive the semantic descriptions, time information, and natural language instructions, and perform semantic reasoning, temporal correlation judgment, and logical decision-making accordingly; the time information is generated based on the parsed temporal semantics and / or based on the timestamp corresponding to the image data; and a vector database, configured to store vectors converted from the semantic descriptions generated by the VLM and associate them with the corresponding timestamps. The AI... The core device of the intelligent agent, based on the temporal semantics, coordinates the invocation of the LLM, the VLM, and the vector database to perform semantic retrieval and temporal analysis of historical image events, semantic understanding and response of real-time image content, and continuous monitoring and early warning triggering based on conditional rules. The image monitoring screen of the user terminal includes an intelligent assistant icon. When the intelligent assistant icon is triggered, the user terminal automatically extracts the time node or visual frame of the currently displayed image as a context and transmits it along with the natural language command to the AI ​​intelligent agent core device.

[0005] According to some specific embodiments of this invention, the image event record of the vector database is established through the following steps: the VLM performs in-depth analysis on the original image or event frame and produces a multimodal semantic summary text; and the multimodal semantic summary text is converted into the vector using an embedding model and stored in the vector database along with the timestamp of the image data.

[0006] According to certain specific embodiments of this invention, when the temporal semantics are determined to be a past query, the AI ​​intelligent agent core device performs the following steps: extracting a target time range and a semantic query feature based on the natural language instruction; using a retrieval augmented generation (RAG) architecture, filtering records that match the target time range in the vector database and performing vector similarity retrieval to obtain relevant image event summaries; and using the LLM to perform temporal correlation reasoning on the multiple relevant image event summaries obtained from the retrieval to generate responses regarding the duration of behavior or the causality of events.

[0007] According to some specific embodiments of this invention, when the time semantics is determined to be real-time analysis, the AI ​​intelligent agent core device actively extracts the current key frame from the real-time video stream of the video monitoring screen through an agent collaboration interface with tool calling capability, and hands it over to the VLM and LLM for analysis to produce a real-time summary; wherein the real-time summary is then converted into a vector and stored in the vector database, so that the real-time event is converted into a searchable historical record.

[0008] According to certain specific embodiments of this invention, the proxy collaboration interface with tool invocation capability is a Model Context Agreement (MCP) tool.

[0009] According to certain specific embodiments of this invention, when the time semantics are determined to be a future warning, the system executes a continuous monitoring process, which includes: the AI ​​intelligent agent core device establishing a future task schedule based on the natural language instruction, the task schedule including a logical triggering rule generated by the LLM; performing preliminary filtering of the real-time video stream through a primary event detection module; and when the primary event detection module is triggered, submitting the current image, time information and the logical triggering rule to the VLM and the LLM for high-level semantic reasoning, and sending a warning notification when the reasoning result matches the logical triggering rule.

[0010] According to some specific embodiments of this invention, the primary event detection module is at least one of dynamic detection, human body contour recognition, or sound detection; the advanced semantic reasoning is used to determine specific object associations, complex behavioral patterns, or user-defined specific logical situations.

[0011] According to some specific embodiments of the present creation, the video surveillance system further includes a user feedback mechanism. The AI intelligent agent core device adjusts the prompting word strategy for semantic reasoning of the LLM or optimizes the retrieval weight of the vector database according to the evaluation result of the user on the reply or the warning, so as to achieve continuous learning. Brief Description of the Drawings

[0012] FIG. 1 is a structural diagram of the video surveillance system of the present creation. Embodiment

[0013] The following will illustrate the present novelty through multiple embodiments. The attached drawings are only used to assist in illustrating the following embodiments, rather than to limit the protection scope of the present novelty. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those of ordinary skill in the technical field to which the present novelty belongs.

[0014] As used herein, the term "a", unless otherwise specified, means at least one (one or more).

[0015] Please refer to FIG. 1. The present creation provides an AI intelligent agent video surveillance system (10) with Time Semantics as the core. The system mainly consists of three major architectures: a user terminal (110), an AI intelligent agent core device (120), and a cloud service and data layer device (130). The AI intelligent agent core device (120) is communicatively connected to the user terminal (110), and the cloud service and data layer device (130) is communicatively connected to the AI intelligent agent core device (120).

[0016] This invention also provides an image monitoring method applicable to an image monitoring system (10) as described herein, which includes a user terminal (110), an AI intelligent agent core device (120), and a cloud service and data layer device (130). The method includes: the user terminal (110) displaying an image monitoring screen and receiving a natural language command input by a user; wherein, when a smart assistant icon in the image monitoring screen is triggered, the user terminal automatically captures the time node or visual frame of the currently displayed image as a context, and transmits it along with the natural language command to the AI ​​intelligent agent core device (120); the AI ​​intelligent agent core device (120) receives the natural language command, analyzes the temporal semantics implied in the natural language command, and determines whether it belongs to a past query, real-time analysis, or future warning; the AI ​​intelligent agent core device (120) generates corresponding monitoring tasks and model call commands based on the analysis results; and utilizes the cloud service and data layer device (130). The Visual Language Model (VLM) receives image data and generates corresponding semantic descriptions or event summaries. A Large Language Model (LLM) receives the semantic descriptions, time information, and natural language instructions, and performs semantic reasoning, temporal correlation judgment, and logical decision-making accordingly. The time information is generated based on the parsed temporal semantics and / or the timestamp corresponding to the image data. A vector database in the cloud service and data layer device stores the vectors converted from the semantic descriptions generated by the VLM and establishes a correlation with the corresponding timestamps. The AI ​​intelligent agent core device (120) coordinates the invocation of the LLM, the VLM, and the vector database based on the temporal semantics to perform semantic retrieval and temporal analysis of historical image events, semantic understanding and response of real-time image content, and continuous monitoring and early warning triggering based on conditional rules.

[0017] User terminal (110): In one embodiment, this refers to a mobile device or computer with a monitoring application (App) installed, but is not limited thereto. For example, the interface of the user terminal (110) may also be implemented through a web interface. The terminal provides an operating interface and includes a smart assistant icon. When the user triggers this icon while watching live view or playback, the system automatically retrieves the current visual context, including but not limited to the current frame, timestamp, and device ID.

[0018] The AI ​​intelligent agent core device (120) serves as the system's scheduling hub. It includes an AI agent engine responsible for parsing natural language commands. Its core function lies in temporal intent recognition, classifying commands into three tenses: "Past," "Present," or "Future," and assigning different agent tasks accordingly. Its specific implementation includes the following four stages.

[0019] 1. Temporal Semantic Analysis and Intent Classification

[0020] When the AI ​​intelligent agent core device receives a natural language instruction, it first extracts the following features through the built-in semantic parsing module: (1) Temporal Entity Extraction: Identifies time words in the instruction (such as "yesterday", "just now", "next week", "whenever..."). (2) Tense Discrimination Logic: Based on the grammatical structure and verb tense, the intent is mapped to one of "Past-oriented", "Present-oriented" or "Future-oriented". (3) Visual Context Alignment: If the instruction is accompanied by a visual context from the user terminal (for example, the user is clicking on a playback screen), the agent core will automatically use the timestamp and device ID of the image as implicit query conditions.

[0021] To address the common issue of ambiguous referential meaning in natural language (such as "Who is this person?"), this system implements a context-aware mechanism deeply integrated with the user interface (UI). Context Extraction: When a user triggers the intelligent assistant function on the user interface (such as an event list, live video, or replay screen), the user terminal automatically extracts the metadata of the currently displayed content as "implicit input." This metadata includes, but is not limited to, time nodes, keyframes, video file paths, or event thumbnails. Prompt Synthesis: The system maintains a System Prompt Library, dynamically injecting the extracted visual background knowledge into the input prompts of the Large Language Model (LLM). This allows the model to obtain a "visual context" consistent with the user's perspective, accurately interpreting the true intent behind omitted subjects or ambiguous referential meanings in instructions, eliminating the need for users to manually input lengthy time and feature descriptions.

[0022] 2. Task Planning and Chain Reasoning

[0023] The agent core does not directly output the answer, but generates a series of execution steps (Plan) based on the intent: (1) Dynamic tool scheduling: The core maintains a set of tool lists, including "vector retrieval tools", "real-time image capture tools", "time series calculation tools", etc. (2) Inference chain construction: For example, for the complex problem "who took the package", the core will plan: call vector retrieval to find the time T1 when "the package appeared". Call vector retrieval to find the time T2 when "the package disappeared". Call VLM to perform key frame comparison analysis on the events in the interval from T1 to T2.

[0024] The core device of this AI-powered intelligent agent can be equipped with a high-level router, characterized by its ability to perform temporal deconstruction of natural language commands. When the system receives a command containing multiple tenses or complex logic (e.g., "Has anyone been here in the last few days? Is anyone at the door now?"), the core does not execute a single linear query, but instead initiates a chain-like reasoning process. Subtask decomposition: The core parses and decomposes the original command into multiple independent but logically related subtasks. Parallel processing: The system assigns subtasks involving historical queries to the "past agent," while subtasks involving real-time status checks are assigned to the "present agent." These two agents operate in parallel, respectively calling a vector database and a real-time video stream, and finally the core aggregates the results from both to generate a single response.

[0025] 3. Integration and implementation of agent collaboration interfaces with tool call capabilities.

[0026] To address the technical challenge of LLMs' inability to directly control hardware, this AI-powered intelligent agent core device integrates an agent collaboration interface layer (e.g., a Model Context Protocol, MCP) with tool-calling capabilities. When an intent is determined to be "present" or "future," the agent core generates standardized API call commands (such as `get_live_frame` or `set_event_trigger`). This collaboration interface layer translates these abstract commands into signals executable by the specific monitoring hardware, thereby acquiring real-time image content or setting underlying dynamic detection parameters. When the identified intent relates to the hardware's real-time state (i.e., "present" or "future" intent), the collaboration interface layer translates abstract semantic commands (such as "check the door") into standard API requests recognizable by the hardware (such as capturing real-time frames or setting sensor threshold values).

[0027] This work discloses a proxy collaboration interface with tool call capabilities, one specific embodiment of which is MCP. This protocol layer sits between the AI ​​core and the underlying hardware, responsible for signal translation and encapsulation. Semantic translation: MCP encapsulates the RESTful APIs that monitoring devices could originally only access via HTTP or RTSP into "tool functions" that the AI ​​model can recognize and that have semantic descriptions (e.g., encapsulating the get_snapshot API into the get_live_frame tool). Tool list maintenance: The AI ​​core maintains a dynamic tool list (including vector retrieval, real-time capture, parameter setting, etc.). When the LLM parses the user's intent, it autonomously decides and selects the appropriate tool function from the list to call, rather than executing pre-written code.

[0028] Heterogeneous Device Integration: MCP adopts a standardized JSON data structure and communication interface, enabling cameras from different brands and with different communication protocols to access the AI ​​system in a unified format. Concurrency: For complex commands involving multiple monitoring endpoints (such as "check all floor entrances"), the MCP module has concurrency control capabilities, allowing it to simultaneously send tool requests to multiple hardware endpoints, rather than executing them sequentially. This mechanism significantly reduces system response latency in multi-device environments, ensuring the timeliness of real-time monitoring.

[0029] 4. Three-state memory management and sharing

[0030] To achieve temporal continuity, the core device of the AI ​​intelligent agent implements a shared memory mechanism: Short-term working memory: Stores the context of the current dialogue, ensuring that subsequent questions from the user (such as "And then what?") can inherit the temporal semantics of the preceding text. Long-term semantic memory (linked with a vector database): After the "present agent" analyzes a real-time video clip, the core automatically writes the semantic summary and timestamp of that clip into the "long-term memory (vector database)". This mechanism ensures that the system's data flow continuously moves from the "present" to the "past", allowing the "past agent" to retrieve real-time events that occurred just seconds ago, eliminating the gap between historical and real-time data.

[0031] The cloud service and data layer device (130) includes the following features: (1) LLM / VLM intelligent service: using a large language model (LLM) to process logical reasoning and instruction parsing, and using a visual language model (VLM) to transform unstructured image data into structured text descriptions. (2) Vector Database: storing semantic vectors transformed from images through a retrieval-enhanced generation (RAG) architecture. Each record strictly corresponds to a high-precision timestamp, forming the system's "long-term memory".

[0032] The cloud service and data layer device (130) is the core of this system's computation and memory center, responsible for translating raw image signals into semantic information that can be understood and retrieved. Its specific implementation details are as follows:

[0033] 1. Implementation of LLM / VLM Intelligent Service Components

[0034] This intelligent service component is composed of a large language model (LLM) and a visual language model (VLM) working together, and its specific responsibilities are as follows:

[0035] (1) Visual semantic translation

[0036] The VLM is configured to perform multimodal understanding tasks. When it receives image data (such as dynamically triggered frames or real-time captured images), the VLM identifies objects (people, vehicles, packages, etc.), behavioral attributes (movement direction, stationary behavior), and environmental conditions (lights on / off, door and window status) in the scene, and produces a multimodal semantic summary with temporal description (e.g., "At 3:15 pm, a courier in a green uniform placed a brown package at the door").

[0037] (2) Higher-order reasoning and decision making

[0038] The LLM receives natural language instructions from the AI ​​intelligent agent core device (120) and combines semantic information from the VLM or vector database to perform reasoning. The LLM has the ability to determine temporal correlation, calculate the time difference of multiple event summaries, determine logical causality (e.g., determine whether "the package disappeared" is due to "the person taking it"), and generate a response to the user or trigger a warning instruction based on the reasoning result.

[0039] 2. Implementation of Vector Databases and RAG Indexing Mechanism

[0040] To achieve efficient historical event retrieval, this system adopts a Retrieval-Augmented Generation (RAG) architecture, whose core component is a vector database.

[0041] (1) Semantic vectorization (Embedding)

[0042] All semantic summaries generated by VLM are transformed into numerical vectors in a high-dimensional space through an embedding model. These vectors represent the semantic features of the image events.

[0043] (2) Hybrid retrieval index: Each record in this vector database is stored in the form of "vector + metadata". The metadata includes a high-precision timestamp, device ID and event type.

[0044] (3) Retrieval Logic

[0045] When performing a past query task, the database performs a hybrid search: first, it filters metadata for a specific time period based on temporal semantics, and then performs vector similarity comparison within that period (e.g., cosine similarity calculation), thereby accurately locating the key image fragments that best match the query intent from thousands of hours of records.

[0046] The following explains how to improve the efficiency and accuracy of historical queries for large-scale surveillance video databases through specific data structures and retrieval strategies.

[0047] (1) The composite index structure is synchronized with the entire system.

[0048] Data Structure: The system's vector database adopts a composite storage format of "vector + metadata". The primary key of each record consists of the device ID and a high-precision timestamp accurate to 0.1 seconds.

[0049] Time synchronization: All devices connected to the system are calibrated via the Network Time Protocol (NTP). When writing data, the timestamp generated by the device is strictly used to ensure absolute consistency between the "image occurrence time" and the "database indexing time". This unified time base also serves as the unique identification key for aligning image streams from different perspectives when tracking across cameras.

[0050] (2) Hybrid retrieval strategy of first time sequence and then vector

[0051] Given the strong temporal characteristics of surveillance images, this invention can implement an optimized retrieval algorithm:

[0052] Pre-filtering: The system prioritizes parsing the time semantics in natural language instructions (such as "yesterday afternoon") and uses metadata indexes to filter out candidate datasets.

[0053] Vector Similarity Search: Vector similarity search is performed only on the time-filtered candidate dataset mentioned above.

[0054] Technical benefits: Compared to full-database vector search, this strategy significantly reduces the scope of high-dimensional vector operations and substantially improves the response speed for semantic retrieval in thousands of hours of video data.

[0055] 3. Continuous Indexing Process for Image Events

[0056] In this embodiment, the cloud service and data layer device (130) executes a continuous data pipeline to maintain the continuity of monitoring: (1) Triggered capture: when the underlying device detects dynamics or when a user initiates a request, key images are captured; (2) Real-time summary: text descriptions are immediately generated by VLM; (3) Vector storage: the descriptions are converted into vectors and timestamped and stored in the database; and (4) Dynamic update: this process ensures that real-time images every second can be immediately converted into searchable historical semantic records.

[0057] 4. Implementation of Early Warning Logic Based on Conditional Rules

[0058] For the "future early warning" task, the cloud service and data layer device (130) implemented a logic comparison engine: (1) LLM interprets the user’s natural language instructions (e.g., “Notify me if a stranger loiters for more than 2 minutes”) into logical judgment conditions. (2) When the VLM continuously generates real-time image summaries, the LLM will compare the summary content at each time point with the preset conditions. (3) If the difference between the “person characteristics” and the “stay time stamp” reported by VLM meets the preset conditions, the system will determine that the logic is valid and trigger an early warning signal.

[0059] The key to this work lies in breaking down the monitoring task into three temporal dimensions and coordinating them through a unified semantic framework.

[0060] 1. Past Agent: Historical Semantic Retrieval

[0061] When a user asks a question about a historical event (e.g., "Who took the package yesterday afternoon?"), the system executes the following process:

[0062] (1) Data indexing: The system continuously performs VLM summaries on events detected by the camera (e.g., "the courier placed a brown cardboard box") and converts them into vectors to store in the vector database.

[0063] (2) Retrieval and reasoning: After the AI ​​intelligent agent core device parses the instructions, it simultaneously performs "time range filtering (yesterday)" and "semantic similarity retrieval (package, take away)" from the vector database.

[0064] (3) Logical interpretation: LLM aggregates multiple search results, determines the order of events (e.g., the time difference between the appearance and disappearance of a package), and finally generates a natural language response.

[0065] 2. Present Agent: Immediate Contextual Understanding

[0066] When a user asks about their current status (e.g., "How is things at home right now?"):

[0067] (1) Abstraction of protocol layer scheduling: The agent core actively retrieves the latest frames from the real-time stream through an agent collaboration interface (e.g., Model Context Protocol, MCP) with tool calling capabilities.

[0068] (2) Real-time summary: VLM analyzes the image and generates a summary (e.g., "No one is in the living room right now, and the pet is resting on the sofa").

[0069] (3) Memory conversion: The summary is automatically stored in the vector database while being sent back to the user. This step ensures the continuity of the timeline, so that what is happening now can be immediately included in the search scope of the past.

[0070] 3. Future Agent: Conditional Logic Warning

[0071] When a user sets future conditions (e.g., "Notify me if someone reappears at the door within half an hour"):

[0072] (1) Task scheduling: The AI ​​intelligent agent core device parses the instruction into a set of monitoring tasks with logical conditions (if-then).

[0073] (2) Two-stage triggering: The system first continuously monitors with low-power initial event detection (such as motion sensing). Once triggered, the image and context are packaged and sent to VLM / LLM for high-level semantic reasoning to determine whether it meets complex conditions (such as identity recognition of "repeated occurrence").

[0074] The following explains how "Future Agent" achieves high-level semantic warnings by using layered filtering and state control while minimizing cloud computing costs.

[0075] (1) Two-Stage Filtering

[0076] To address the high computational and bandwidth consumption caused by continuous AI analysis, this system implements a layered triggering mechanism from the edge to the cloud. Phase 1 (Edge): Low-power initial event detection (such as motion detection, pixel change detection, or basic human contour recognition) is performed locally by the camera. During this phase, the system does not transmit images to the cloud, nor does it consume cloud VLM computing power. Phase 2 (Cloud): Only when a potential triggering event is detected in Phase 1 does the system capture key frames and time information for uploading, activating the cloud's VLM and LLM for higher-level semantic reasoning (such as specific behavior analysis and object association determination).

[0077] (2) Task control of Finite State Machine (FSM)

[0078] To ensure the predictability of AI agent behavior and the accuracy of resource release during long-term monitoring tasks, this invention incorporates state machine technology to manage tracking logic. State transitions: Taking "wandering alert" as an example, the AI ​​agent's operation sequentially goes through states such as "Idle," "Edge Triggered," "Semantic Verification," "Logic Tracking," and "Alert / Reset." Resource release: The system utilizes a well-defined FSM to define clear termination conditions. For example, if semantic verification fails (non-target object) or the tracking timer expires, the state machine immediately resets and releases cloud inference resources, preventing the AI ​​agent from indefinitely consuming computing resources.

[0079] 4. Continuous learning and feedback mechanism

[0080] This system features a user feedback loop. If a user corrects the AI ​​agent's judgment (e.g., changing "stranger" to "family member"), the system will update the corresponding label in its vector database or adjust the LLM's prompt strategy, thereby optimizing the accuracy of subsequent event prediction and understanding.

[0081] The following explains how the system achieves self-optimization and personalized adaptation of the model through human-computer interaction feedback.

[0082] (1) Intentional inheritance and short-range memory

[0083] The system can be configured with a short-term working memory module to save the context of the conversation. When the user asks a follow-up question that omits time or subject (such as "And then what?"), the system can inherit the time anchor and focus of the previous conversation and automatically extend the search scope to maintain the continuity of the conversation.

[0084] (2) Closed-loop implementation of Few-shot Learning

[0085] This system has closed-loop learning capabilities, enabling it to convert user corrections into long-term memory.

[0086] Correct input: When the user makes a correction to the AI's judgment (e.g., "This is not a stranger, it is the father").

[0087] Example storage: The system stores the image features of the misclassified image and the correct label provided by the user as a "Few-shot Example" in the database.

[0088] Strategy Update: During subsequent inference, the system dynamically retrieves relevant few-sample examples and injects them into the LLM's prompt word strategy (In-context Learning), or adjusts the weight parameters of vector retrieval. This allows the system to optimize weights for specific scene or person features, avoiding repeated misjudgments.

[0089] Example 1: Historical Event Query of Past Agents

[0090] This embodiment aims to illustrate how the authoring system handles complex natural language queries from users regarding historical video events. Please refer to Figure 2, which details the complete process of processing past event queries. This process can be divided into two main stages: "data index building" and "past time query retrieval".

[0091] 1. Indexing Phase

[0092] The goal of this stage is to transform unstructured raw image data into structured vector data that can be retrieved using both temporal and semantic methods, laying the foundation for subsequent rapid queries.

[0093] (1) Image Input and Annotation: When the system captures an original image or event, it immediately attaches a high-precision timestamp to it as its absolute coordinate on the timeline. The high-precision timestamp is accurate to at least the second, and preferably to 0.1 seconds.

[0094] (2) Image understanding and summarization: The image is sent to VLM for in-depth analysis. VLM will output a sentence or paragraph of multimodal summary text describing the core content, behavior or semantics of the image, such as "A deliveryman puts a brown cardboard box at the door".

[0095] (3) Textual Vector (Embedding): The summary text generated by VLM is converted into a high-dimensional mathematical vector through the Embedding model. This vector mathematically represents the semantics of the text.

[0096] (4) Establish a unified database: Finally, the high-precision timestamp of the event and its corresponding text vector are stored together in the vector database to form a record that can be retrieved.

[0097] 2. Past Time Query and Retrieval Phase

[0098] This phase demonstrates how the system responds to users' natural language queries about historical events. The following example illustrates how a user queries "Was there a package at my door for more than 20 minutes yesterday?":

[0099] (1) Natural language input: Users input the above query commands through the terminal App.

[0100] (2) Semantic analysis: After receiving the instruction, the AI ​​intelligent agent core decomposes it into two core elements: time range (yesterday) and search intent (the package stayed for more than 20 minutes).

[0101] (3) Vector Retrieval and Time Filtering (RAG): Based on the parsed intent, the agent core initiates an integrated query to the vector database. Under the Retrieval Enhancement Generation (RAG) framework, this query simultaneously applies a timestamp filter (filtering out all records with timestamps in the range of "yesterday") and vector similarity retrieval (finding the vectors that are semantically most relevant to "package stay"), thereby accurately and efficiently finding all potentially relevant historical events.

[0102] (4) LLM Comprehensive Interpretation: Submit the summary text and timestamps of multiple related events retrieved to the LLM assistant. The LLM will perform correlation and logical judgment on these time series data. For example, it will find the two closest events of the package "appearance" and "disappearance" and calculate the difference between their timestamps to determine whether the stay time exceeds 20 minutes.

[0103] (5) Output of results: The system outputs the final interpretation result of the LLM (e.g., "Yes, the package delivered at 2:15 pm yesterday was picked up after being left at the door for about 35 minutes") to the user.

[0104] Example 2: Real-time monitoring of the present agent

[0105] This embodiment aims to illustrate how the system handles natural language queries from users regarding the current real-time image status. Please refer to Figure 3, which illustrates the operational flow of the Present Agent in handling such queries. The following example illustrates the user's query, "Who is at the door now?":

[0106] 1. Intent Recognition: After receiving the instruction, the AI ​​intelligent agent first recognizes from the word "now" that the user is querying the current status, and the intent is "Present".

[0107] 2. Calling the MCP tool: Since answering this question requires real-time visual information, the agent's core decision is to call the Model Context Protocol (MCP) tool. This tool is a specific function or API that enables the AI ​​agent to proactively extract visual information from real-time video streams.

[0108] 3. Capture live footage: The MCP tool instantly captures one or more of the most representative key frames (e.g., in JPG format) from the camera's live stream and adds a high-precision timestamp to them.

[0109] 4. VLM / LLM Summary: The captured real-time images and timestamps are sent to the VLM / LLM assistant for analysis. Based on the user's original query intent, VLM / LLM generates a multimodal summary of the current visual content, such as: "A deliveryman in a red coat is putting down a package at the door."

[0110] 5. Processing Decision: Based on the summarized results, the system will execute one or both of the following two paths:

[0111] (1) Directly reply to the user: Output the summary text generated above directly to the user and provide an immediate answer.

[0112] (2) Storage for RAG retrieval: Simultaneously, the summary text is vectorized and stored in the vector database along with its timestamp. This step is crucial, as it ensures that events occurring "now" can immediately become searchable "history," thus achieving seamless continuity of time.

[0113] Example 3: Conditional Early Warning for Future Agents

[0114] This embodiment aims to illustrate how the system handles user-defined future monitoring tasks that include complex conditions. This process is manifested as a continuous monitoring and judgment loop. Please refer to Figure 4, which illustrates the operation flow of the Future Agent.

[0115] 1. Setting Future Commands and Creating Tasks: The user inputs a command containing future conditions, such as "Notify me if someone approaches again in the next hour" or "Notify me when someone is walking their dog." The AI-powered intelligent agent core receives the command and parses the intent to trigger the "future condition." The agent core then creates a future task, such as `monitor(next 60min, rule="stranger near door")`, and schedules the corresponding real-time alert logic.

[0116] 2. Entering Continuous Monitoring Loop: The system enters continuous monitoring mode. In this loop, the system continuously captures images from the video stream and passes them to the initial event detection module (e.g., based solely on motion, human silhouette, or sound detection) for low-power preliminary filtering. If no trigger occurs, the system continues this monitoring loop. Once a potential event matching the basic criteria is detected (e.g., detecting "human"), the next step is triggered.

[0117] 3. Collect context: Once the initial event is triggered, the system will immediately package three key context information: (1) the JPG image at the time of the trigger; (2) the high-precision timestamp of the event; and (3) the complex questions or conditions preset by the user (e.g., "determine whether this person is a stranger and is approaching the door").

[0118] 4. VLM / LLM Intelligent Reasoning: The aforementioned packaged context is sent to the VLM / LLM intelligent service. VLM is responsible for deeply understanding the visual content of the image, while LLM is responsible for logically comparing and judging the visual analysis results with the complex rules preset by the user. For example, determining whether the person in the picture is a known family member and whether their direction of movement is towards the door.

[0119] 5. Trigger Alarm: If the LLM's judgment result meets the complex conditions set by the user, the Future Agent will ultimately trigger a smart event and output an alert and notification through the user terminal, completing this future task. If the conditions are not met, the system will return to the continuous monitoring loop in step two.

[0120] Example 4: AI Core: Chain Reasoning and Dynamic Context Injection

[0121] This embodiment aims to illustrate how the AI ​​Agent Core handles ambiguous instructions with mixed temporalities and how it leverages UI context to enhance the accuracy of Prompt Engineering.

[0122] 1. Chain of Thought with Mixed Temporal Instructions

[0123] When the system receives complex instructions containing different temporalities (e.g., "Has anyone been here in the last few days? Is anyone at the door now?"), the router (Router Agent) does not answer directly, but instead initiates chained reasoning and task planning:

[0124] (1) Instruction decomposition: The core decomposes instructions into subtasks that are processed in parallel.

[0125] (2) Task assignment: Subtask A "Has anyone been here in the last few days?" is assigned to "Past Agent" to perform historical vector retrieval; Subtask B "Is anyone at the door now?" is assigned to "Current Agent" to perform real-time screen analysis.

[0126] (3) Result aggregation: The core aggregates the execution results of the two into a single natural language response output.

[0127] 2. Dynamic Context Injection Mechanism

[0128] The system maintains a "System Prompt Library" and dynamically adjusts the context input to the LLM based on the user's actions on the terminal interface (UI).

[0129] (1) Triggering scenario: When a user clicks on the smart assistant icon in the "Event List" or "Replay Screen" and asks a referential question (such as: "Who are these people?").

[0130] (2) Metadata capture: The terminal automatically captures the "time node", "visual frame (key frame)" or "metadata (such as precise occurrence time, recording path)" of the currently displayed image.

[0131] (3) Prompt word synthesis: The system dynamically injects the visual background knowledge and time information extracted above into the prompt words. This mechanism enables the LLM to understand that "these people" specifically refers to "the people appearing at the current time point in the screen", without requiring the user to manually input lengthy time and feature descriptions.

[0132] Example 5: Model Context Contract (MCP) and Hardware Tooling

[0133] This embodiment illustrates how the system connects abstract AI inference to closed monitoring hardware through MCP, achieving deep integration of software and hardware.

[0134] 1. API encapsulation and utility functionization

[0135] The MCP layer encapsulates and transforms the HTTP-based RESTful API of traditional monitoring devices:

[0136] (1) Conversion logic: Encapsulate the underlying APIs (such as RTSP stream requests and PIR sensor settings) into standardized "tool functions" that can be directly recognized and called by the AI ​​model.

[0137] (2) Tool list maintenance: The core maintains a set of tool lists, including query_event (vector retrieval), get_live_frame (real-time retrieval), etc.

[0138] 2. Semantic translation and concurrent execution

[0139] (1) Example implementation: When the AI ​​interprets the user’s intent as “find the person who ate food in the classroom today”, it will automatically generate an MCP call command, such as query_event("someone eats food in the classroom", time="today").

[0140] (2) Concurrency: For complex requirements involving multiple cameras (such as "checking all floor entrances"), MCP adopts a standardized JSON communication interface, which can send tool requests to multiple hardware endpoints concurrently and aggregate the returned data, significantly improving processing efficiency in multi-device environments.

[0141] Example 6: Two-Stage Filtering and Finite State Machine (FSM) for Future Agents

[0142] This embodiment details how the "Future Agent" balances the high-end AI computing costs with the need for real-time monitoring through a resource management mechanism.

[0143] 1. Two-Stage Filtering Logic

[0144] To reduce the continuous computational costs of cloud-based VLM / LLM, the system implements tiered processing:

[0145] (1) First stage (Edge end): Low-power initial event detection (such as motion detection and human contour recognition) is performed by the camera end. This stage does not consume cloud computing power.

[0146] (2) Second stage (Cloud): The system will only capture key frames (JPG) and time information and send them to the cloud when a potential event (such as the appearance of a shadow) is detected in the first stage, and wake up the VLM to perform high-level semantic reasoning.

[0147] 2. Finite State Machine (FSM) Task Management

[0148] The system introduces state machine technology to restrict the behavior of the AI ​​agent to a predefined logical path, taking "suspicious loitering alert at night" as an example:

[0149] (1) Idle: The task scheduler starts at a set time (e.g., 22:00) and only enables edge detection.

[0150] (2) Edge Triggered: When the camera detects a "human" entering the frame, a state transition is triggered.

[0151] (3) Semantic Verification: The AI ​​agent calls the VLM to determine whether the person is "in front of the garage" and "has the intention to stay". If true, it enters the tracking state; if false (such as a passerby passing by quickly), it resets to the standby state.

[0152] (4) Logical Tracking: Start a timer. If the dwell time reaches a threshold (e.g., 5 minutes), the system will switch to alarm mode.

[0153] (5) Alert & Release: After sending a notification, cloud inference resources are automatically released to complete the task cycle.

[0154] Example 7: Hybrid Retrieval Algorithm Based on High-Precision Timestamps

[0155] This embodiment illustrates how to improve query efficiency for large-scale surveillance video databases through optimized indexing structures and retrieval strategies.

[0156] 1. Consistency between index structure and data

[0157] (1) Data structure: Each record in the vector database is stored in the form of "vector + metadata".

[0158] (2) Core primary key: The device ID and a high-precision timestamp accurate to 0.1 seconds are used as the primary key. All connected devices are synchronized through NTP time synchronization to ensure that the "image occurrence time" and "database indexing time" are absolutely consistent.

[0159] (3) Cross-camera alignment: This precise timestamp serves as a unique identification key for the entire system, enabling deep linking of video archives, event annotations, and vector features to achieve temporal alignment across camera perspectives.

[0160] 2. Hybrid Search Optimization Algorithm

[0161] The system executes a strategy of "filtering by time first, then retrieving by vector":

[0162] Step 1: Based on the time semantics parsed from the natural language instructions (such as "yesterday afternoon"), prioritize using the metadata index to filter out candidate datasets within the target time range.

[0163] Step 2: Perform vector similarity comparison only for this candidate dataset.

[0164] This method significantly reduces the amount of data that requires high-dimensional vector operations, thereby greatly improving the retrieval speed and accuracy in thousands of hours of video recordings.

[0165] Example 8: Closed-loop learning mechanism based on user feedback

[0166] This embodiment illustrates how the system achieves self-optimization and personalized learning through user feedback and corrections.

[0167] 1. Intention inheritance and short-range memory

[0168] The system has short-term working memory and can understand contextual inheritance relationships. For example, when the user asks "And then what happened?", the system can inherit the time and objects of the previous conversation and automatically retrieve relevant events.

[0169] 2. Closed-loop learning process

[0170] Taking "family member identification optimization" as an example:

[0171] (1) False alarm occurs: The system mistakenly identifies a family member (such as the father) who is near the door as a stranger and sends an alarm.

[0172] (2) User feedback: The user clicked on the assistant icon and replied: "This is my dad. I don't need to call the police if I see him in the future."

[0173] (3) Few-shot learning: After receiving the correction instruction, the AI ​​core stores the image features that were misjudged as "positive examples" in the database as few-shot learning data.

[0174] (4) Strategy Update: The system synchronously adjusts the prompt strategy or retrieval weight during semantic reasoning. When the father reappears, the system combines this new memory to make inferences, correctly identify his identity, suppress alarms, and complete the learning loop.

[0175] 10: Video surveillance system 110: User Terminal 120: AI Smart Agent Core Device 130: Cloud services and data layer devices

Claims

1. A video surveillance system, comprising: a user terminal for displaying video surveillance footage and receiving natural language commands input by the user; an AI intelligent agent core device communicatively connected to the user terminal, the AI ​​intelligent agent core device being configured to: receive the natural language commands; parse the temporal semantics implied in the natural language commands and determine whether they belong to one of past queries, real-time analysis, or future warnings; and generate corresponding monitoring tasks and model call commands based on the parsing results; and a cloud service and data layer device communicatively connected to the AI ​​intelligent agent core device, and comprising: a Large Language Model (LLM) and a Visual Language Model (VLM), wherein: The VLM is configured to receive image data and generate corresponding semantic descriptions or event summaries. The LLM is configured to receive the semantic descriptions, time information, and natural language instructions, and perform semantic reasoning, temporal correlation judgment, and logical decision-making accordingly. The time information is generated based on the parsed temporal semantics and / or based on the timestamps corresponding to the image data. A vector database is configured to store vectors converted from the semantic descriptions generated by the VLM and associate them with the corresponding timestamps. The AI... The core device of the intelligent agent, based on the temporal semantics, coordinates the invocation of the LLM, the VLM, and the vector database to perform semantic retrieval and temporal analysis of historical image events, semantic understanding and response of real-time image content, and continuous monitoring and early warning triggering based on conditional rules. The image monitoring screen of the user terminal includes an intelligent assistant icon. When the intelligent assistant icon is triggered, the user terminal automatically extracts the time node or visual frame of the currently displayed image as a context and transmits it along with the natural language command to the AI ​​intelligent agent core device.

2. The video surveillance system as described in claim 1, wherein the video event records of the vector database are established through the following steps: the VLM performs deep analysis on the original video or event frames and produces a multimodal semantic summary text; and uses an embedding model to convert the multimodal semantic summary text into the vector, and stores it in the vector database along with the timestamp of the video data.

3. The video surveillance system as described in claim 1, wherein when the temporal semantics are determined to be a past query, the AI ​​intelligent agent core device performs the following steps: extracting a target time range and a semantic query feature based on the natural language instruction; using a retrieval augmented generation (RAG) architecture to filter records that match the target time range in the vector database and perform vector similarity retrieval to obtain relevant video event summaries; and using the LLM to perform temporal correlation reasoning on the retrieved plurality of relevant video event summaries to generate a response regarding the duration of behavior or the causality of events.

4. The video surveillance system as described in claim 1, wherein when the time semantics are determined to be real-time analysis, the AI ​​intelligent agent core device actively extracts the current key frame from the real-time video stream of the video surveillance screen through an agent collaboration interface with tool calling capability, and hands it over to the VLM and the LLM for analysis to produce a real-time summary; wherein the real-time summary is then converted into a vector and stored in the vector database, so that the real-time event is converted into a searchable historical record.

5. The video surveillance system as described in claim 1, wherein when the time semantics are determined to be a future warning, the system executes a continuous monitoring process, comprising: the AI ​​intelligent agent core device establishing a future task schedule based on the natural language instruction, the task schedule including a logical triggering rule generated by the LLM; performing preliminary filtering of the real-time video stream through a primary event detection module; and when the primary event detection module is triggered, submitting the current video, time information and the logical triggering rule to the VLM and the LLM for high-level semantic reasoning, and sending a warning notification when the reasoning result matches the logical triggering rule.

Images