Human abnormal behavior monitoring method based on large-model multi-agent

Abstract

The invention discloses a human abnormal behavior monitoring method based on a large-model multi-agent, which is executed by a modular multi-agent system deployed on a back-end server, obtains information through a monitoring camera, and comprises the following steps: obtaining a video stream from the monitoring camera by a sensing agent and extracting human body posture features; analyzing the key frame by a scene understanding agent by using a visual large model, and constructing a time sequence dynamic scene graph; the core reasoning agent evaluates the scene semantic conformity based on the pre-trained large model and performs abnormal preliminary judgment; performing fine-grained classification, interpretation generation and risk assessment on the abnormal behaviors; and the report and action agent generates an alarm and records event data. According to the invention, through multi-agent cooperative work and a large model technology, efficient and accurate monitoring of human abnormal behaviors is realized, and the intelligent level of the monitoring system and the abnormal behavior identification accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of monitoring technology, and in particular to a method for monitoring abnormal human behavior based on a large model and multiple intelligent agents. Background Art

[0002] In recent years, the rapid development of artificial intelligence (AI), particularly breakthroughs in large-scale models and multimodal technologies, has brought new technical approaches and solutions to abnormal behavior monitoring systems. While these technologies have made significant progress in video surveillance and abnormal behavior detection, existing technologies still suffer from the following shortcomings: Most existing abnormal behavior detection systems rely on predefined behavioral patterns or simple visual features, lacking deep semantic understanding of human behavior in complex and dynamic scenarios, and struggle to accurately identify abnormal behaviors that require contextual information. Existing technologies also have limitations in multimodal information fusion, making it difficult to effectively integrate and reason with multiple sources of information, including visual scenes, target interactions, human posture, and domain rules. This results in low accuracy in complex scenarios. Most existing systems can only provide simple abnormal event classification results, lacking fine-grained classification of abnormal behaviors and interpretable natural language descriptions, limiting the intelligence and practical value of monitoring systems. Existing technologies also have limited capabilities in adapting to new scenarios, identifying new types of abnormal behaviors, and dynamically updating domain knowledge. This leads to high system maintenance and upgrade costs, making it difficult to meet evolving security monitoring needs. Traditional methods perform poorly when dealing with complex behaviors that require common sense reasoning or professional knowledge judgment, especially when dealing with ambiguous, uncertain abnormal behaviors, it is often difficult to make accurate judgments. Summary of the Invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a method for monitoring abnormal human behavior based on a large model and multiple intelligent agents, so as to solve the technical problems existing in the existing video surveillance system in detecting abnormal human behavior, such as insufficient semantic understanding ability, lack of multimodal information fusion mechanism, inadequate description of abnormal behavior, weak ability to adapt to new scenarios, and limited ability to handle complex behaviors.

[0004] Technical Solution: The present invention discloses a method for monitoring abnormal human behavior based on a large-scale multi-agent model, which is executed by a modular multi-agent system deployed on a backend server and includes the following steps: (1) The perception agent obtains the video stream from the surveillance camera, performs key frame smoothing sampling, extracts human posture information and processes it into structured posture features; (2) The scene understanding agent uses the visual large model (VLLM) to analyze key frames, extract scene entities, relationships, and text descriptions, construct a temporal dynamic scene graph, and use the posture features as character node attributes to enhance the scene graph, and output a structured multimodal scene representation data package; (3) The core reasoning agent uses a fine-tuned text / large multimodal model (LLM / VLM) and external knowledge dynamically retrieved by the retrieval enhancement generation RAG module to perform semantic understanding of the data packet, evaluate the paradigm conformity and situational rationality of the behavior, and output an abnormal tendency score and preliminary judgment; (4) If the abnormal propensity score exceeds the threshold, the large model is driven to perform fine-grained classification of abnormal behavior, generate a natural language event summary, and assess the risk level; (5) The reporting and action agent generates customized alerts, stores structured data across the entire chain, and proposes action recommendations based on risk levels.

[0005] Furthermore, in step (1), the posture feature processing is as follows: OpenPose or MediaPipe is used to extract the coordinates of 17 key skeleton points; the skeleton point sequence is processed into two types of structured data: the original coordinate sequence and the posture classification label; the posture classification label is selected from predefined discrete categories, including static posture, dynamic posture, interactive posture, abnormal posture and hand posture.

[0006] Furthermore, in step (2), the construction of the temporal dynamic scene graph is as follows: the visual large model outputs scene entity attributes, entity relationship triples and scene text description; the scene graph within 3-10 seconds is integrated into a temporal dynamic structure to capture the time evolution of entity relationships; the action state attributes of the character nodes are enhanced through posture classification labels.

[0007] Furthermore, in step (3), the anomaly assessment mechanism is as follows: Prompt templates are dynamically selected based on the scenario type, and scenario graph data and RAG search results are inserted; the RAG module retrieves security rules, behavioral patterns, and abnormal case characteristics from the knowledge base and returns the top 3-5 relevant entries; the anomaly propensity score is a continuous value from 0 to 1, and a multi-level threshold triggers a response; among them, ATS < 0.3 is normal; 0.3 ≤ ATS < 0.7 is suspicious behavior; ATS ≥ 0.7 triggers in-depth analysis.

[0008] Furthermore, in step (4), the fine-grained classification includes: the first-level classification is property-related, personal-related, or rule violation; the second-level classification is based on the refinement of the first-level category, including theft, destruction, attack, harassment, breaking into restricted areas, or illegal operation; the event summary contains five elements: the subject involved, time, location, behavior description, and abnormal basis.

[0009] The present invention provides a large-model multi-agent-based human abnormal behavior monitoring system, comprising: Perception agent module: used to integrate the video stream processing unit, key frame sampling module, skeleton point extraction and posture classification model; Scene understanding agent module: used for visual large model interface, temporal dynamic scene graph construction engine and posture feature integration module; Core reasoning agent module: used for built-in LLM / VLM model, Prompt management module, RAG module and anomaly assessment module; Reporting and Action Agent Module: used to realize alarm generation, multi-channel notification, event data recording and action recommendation generation functions; The knowledge base is used to store security rules, behavioral patterns, and abnormal case characteristics, supporting semantic retrieval of the RAG module; The agent communication and orchestration infrastructure is used to achieve multi-agent collaboration through message queues, API gateways, and workflow engines.

[0010] Furthermore, the posture classification model of the perceptual agent is as follows: a temporal convolutional network (TCN) or a long short-term memory (LSTM) network is used to process the skeleton point sequence; and composite posture pattern labels are output, including pushing, crowd gathering, or crowd evacuation.

[0011] Furthermore, the core reasoning agent was fine-tuned as follows: LLM / VLM was trained using annotated multimodal scene representation data packages, with annotations including anomaly propensity scores, fine-grained classifications, and event summaries; fine-tuning focused on scene graph parsing capabilities, covering human-human, human-object, and human-environment interaction modes; and prompt design guided the model to analyze time evolution logic and behavior-scene context matching.

[0012] Furthermore, the knowledge base is constructed specifically as follows: knowledge entries are organized by scenario type, behavior category, and risk level; vector indexes are generated using BERT or Sentence-BERT to support semantic similarity retrieval; and a regular update mechanism is set up to dynamically expand new abnormal patterns and security rules.

[0013] Furthermore, the reporting and action agents are as follows: generating specific instructions for medium and high-risk events, including dispatching security personnel, locking down areas, or activating emergency plans; continuously tracking the development of events and dynamically updating risk ratings Beneficial effects: Compared with the prior art, the present application has the following remarkable advantages: scene analysis by a visual large model and core reasoning by a large language model / multimodal large model, realizing deep and fine-grained semantic understanding of human behavior and the scene in which it occurs; by constructing a dynamic scene graph to fuse multi-modal information and dynamically introducing external knowledge using the Retrieval-Augmented Generation (RAG) mechanism, the comprehensive judgment and reasoning ability of the system for complex situations is significantly enhanced; not only can it accurately judge whether the behavior is abnormal, but also can classify the abnormal behavior in multiple dimensions, generate detailed natural language descriptions and risk assessments, and improve the usability and explainability of the monitoring results; through modular agent design, an updatable RAG knowledge base, and fine-tuning and Prompt engineering of large models, the system can more conveniently adapt to new scenarios, new rules and new types of abnormal behavior; by comprehensively evaluating the "paradigm conformity" and "context rationality" of the behavior, it can more accurately distinguish between complex normal behavior and real abnormal behavior, effectively reducing false positives and false negatives; it represents the direction of the new generation of intelligent monitoring technology, and upgrades the monitoring system from simple event detection to the analysis and decision assistance level with preliminary cognitive intelligence. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0015] The technical solutions of the present application will be further described below in conjunction with the drawings.

[0016] As shown in Figure 1 , the present application is a method for monitoring abnormal human behavior based on a large model multi-agent, which is executed by a modular multi-agent system deployed on a backend server. Each agent interacts through a pre-defined interface and data format, and the multi-agent system obtains information through a monitoring camera. The method includes the following steps executed by a specific agent in sequence or in parallel, constituting a clear data processing and decision link: S100: Data perception and feature preparation step (executed by a perception agent) S110: The perception agent obtains video streams from one or more monitoring cameras in real time and performs smooth sampling of key frames. Specifically, the perception agent is connected to a network of high-definition monitoring cameras deployed in public places such as shopping malls, subway stations and schools, and receives 1080p or 4K resolution video stream data. Smooth sampling uses an adaptive sampling algorithm, which samples at a lower frequency (e.g. 1-2 frames per second) when the scene is static, and increases the sampling frequency (e.g. 5-10 frames per second) when a scene change or personnel activity is detected, to balance the computational resource consumption and the integrity of information capture.

[0017] S120: The perception agent extracts or receives human pose information in the video frames using mature visual processing techniques and processes it into structured pose features. In this embodiment, the perception agent integrates open-source human pose estimation libraries such as OpenPose or MediaPipe to extract 17 key skeleton point coordinates from video frames, including two-dimensional or three-dimensional spatial coordinates of head, shoulder, elbow, wrist, hip, knee, and ankle joints. These raw skeleton point coordinates are then processed into two types of structured features: one is the raw skeleton point coordinate sequence, which retains complete pose details; the other is the pose classification description based on skeleton point analysis, such as "standing", "walking", "running", "falling", "raising hands", "squatting", "fighting posture", etc. discrete category labels. Pose classification is achieved through a pre-trained pose classification model, which is trained based on skeleton point time series data and can recognize 20-30 basic human motion types (see the following supplementary explanation for specific categories).

[0018] S130: The perception agent passes the extracted pose features and smoothly sampled key frames to the scene understanding agent through internal message queues or API calls. Specifically, the perception agent packages the processed data into standardized JSON format messages, including the following fields: video source ID, timestamp, key frame image (Base64 encoded or file path reference), skeleton point coordinate sequence (multi-dimensional array in the format [frame_id, person_id, joint_type, x, y, z, confidence]), and posture classification result (array in the format [person_id, posture_type, confidence]). These data are transmitted to the scene understanding agent through the system's internal RabbitMQ message queue or RESTful API interface, ensuring the reliability and real-time performance of data transmission.

[0019] S200: Dynamic scene understanding and structured representation based on visual large models (performed by the scene understanding agent) S210: The scene understanding agent receives multi-modal data from the perception agent (S130). The scene understanding agent maintains an input data buffer that can receive and temporarily store continuous data streams from the perception agent, forming a processing window with a time span of 3-10 seconds to capture sufficient temporal context information.

[0020] S220: The scene understanding agent core utilizes a pre-trained visual large model to analyze the received keyframe sequence. In this embodiment, the scene understanding agent integrates visual large models such as GPT-4V or CLIP-ViT, which have been pre-trained on large-scale image-text pair data and fine-tuned for scene understanding and relationship extraction tasks. The visual large model processes the input keyframe sequence, extracting three types of key information: 1) key target entities and their attributes in the scene, such as "man in red clothes", "black backpack", "exit door", etc., each entity is accompanied by its location information and confidence in the image; 2) interaction relationships between entities, represented as triples, such as "man-holding-backpack", "woman-standing-in-front-of-counter", "child-running-toward-exit", etc.; 3) natural language text description of the overall keyframe scene, such as "a man in red clothes stands in front of the bank counter, holding a black backpack, and is talking with the clerk". These information are extracted from visual input through the model's multi-head attention mechanism and cross-modal understanding ability, and output in structured and natural language form.

[0021] S230: Based on the output of the visual large model (S220), combined with time series information, the scene understanding agent constructs a time-series dynamic scene graph. The scene graph uses a graph data structure to represent, where nodes represent entities (such as people, objects, scene elements), and edges represent relationships (such as spatial relationships, interaction behaviors). Each node and edge contains an attribute field that stores entity features and relationship types. The scene understanding agent enhances the human pose features obtained in step S120 as the attributes of the corresponding "person" nodes. Specifically, for each person entity node identified in the scene, the system adds its corresponding pose classification description (such as "standing", "running", "falling", etc.) as the pose attribute field of the node to the scene graph. This enhancement makes the scene graph not only contain semantic information extracted by the visual large model, but also integrate more accurate human motion state information, providing more comprehensive input for subsequent abnormal behavior analysis.

[0022] S240: The scene understanding agent passes the generated structured multi-modal scene representation data package to the core reasoning agent. The data package contains: a time-series scene graph (serialized in JSON or a dedicated graph data format), a scene text description generated by the visual large model (S220), and human pose features integrated in the scene graph (S120 / S230). The data package also contains metadata fields such as processing timestamp, scene ID, data source, etc., for subsequent processing and tracing. Data transmission is also completed through the system's internal message queue or API call, ensuring data integrity and consistency.

[0023] S300: Scene semantic conformity assessment and initial anomaly judgment based on the large model (performed by the core reasoning agent) S310: The core reasoning agent integrates a pre-trained large text model or multimodal model through software packaging, which has been fine-tuned for the task of understanding and judging abnormal behavior patterns in specific monitoring scenarios. In this embodiment, the core reasoning agent uses a large language model based on architectures such as GPT-4o or Qwen2.5VL, which has been fine-tuned specifically for the task of understanding and judging abnormal behavior in security monitoring scenarios. The fine-tuning process uses an annotated dataset containing both normal and abnormal behavior scenarios, enabling the model to understand behavioral norms and abnormal patterns in different environments.

[0024] S320: The core reasoning agent receives the structured multimodal scene representation data package (S240) and dynamically generates complex prompts based on a pre-defined prompt template library to guide the large model in performing deep semantic analysis. The prompt template library contains templates customized for different scene types (such as shopping malls, banks, schools, subway stations, etc.) and analysis tasks. Each template is designed to guide the model to focus on key behavioral characteristics and norms in a specific environment. For example, a prompt template for a bank scenario will guide the model to pay special attention to queuing behavior, interactions with tellers, and the presence of suspicious items; while a prompt for a school scenario will focus more on student group behavior and campus safety zone restrictions. Based on the current scene type and data characteristics, the core reasoning agent selects the most appropriate template from the template library and inserts the scene representation data (scene graph, text description, gesture features) into the corresponding position in the template to generate a complete analysis prompt.

[0025] S330: Before or during reasoning on the large model, the core reasoning agent triggers the RAG module to initiate a query to the external knowledge base based on the current scene representation content and the Prompt instruction (S320). The RAG module first extracts keywords and semantic features from the scene representation to construct a search query. It then sends a query request to the knowledge base to retrieve context-assisted knowledge such as safety rules, normal behavior paradigm definitions, and known abnormal event characteristics that are most relevant to the current scenario. The knowledge base stores structured and unstructured knowledge such as safety regulations, standard operating procedures, historical abnormal event cases, and their characteristic descriptions for various locations. The search results are sorted by relevance, and the top 3-5 most relevant knowledge items are selected, formatted, and added to the Prompt as contextual reference information.

[0026] S340: The core judgment task of the large model is to perform a comprehensive semantic understanding of the input structured multimodal scene representation (S240) and enhanced prompts (S320+S330), assessing the behavioral paradigm conformity and contextual rationality of the described events, interactions, and individual behaviors within a specific context. The large model analyzes the relationship patterns between entities in the scene graph, temporal changes, the alignment of human posture characteristics with the scene context, and consistency with retrieved knowledge to form a comprehensive judgment on the rationality of the behavior in the current scenario. Model outputs include: 1) an abnormality propensity score, a floating-point number between 0 and 1 indicating the degree to which the behavior deviates from the norm, with 0 representing completely normal and 1 representing extremely abnormal; 2) a preliminary judgment of the abnormality type, such as "normal," "minor deviation," "suspicious behavior," or "significant abnormality"; and 3) a summary of the judgment basis, briefly describing the key observations and reasoning behind the judgment.

[0027] S350: The core reasoning agent passes the preliminary judgment result (S340) and related confidence information to the next processing step (S400). Specifically, it packages the anomaly propensity score, anomaly type judgment, judgment basis summary, and related confidence indicators (such as the model's judgment certainty) into a standard result object and transmits it to subsequent processing modules through the system's internal communication mechanism. If the anomaly propensity score exceeds a preset threshold (e.g., 0.7), the system automatically triggers a more in-depth analysis process (S400).

[0028] S400: Fine-grained classification of abnormal behavior, explanation generation, and risk assessment steps (performed by the core reasoning agent or an independent analysis agent) S410: If the initial judgment in step S350 indicates that the behavior is abnormal or highly suspicious (for example, if the abnormality propensity score exceeds 0.7 or the abnormality type is determined to be "suspicious behavior" or "significant abnormality"), the in-depth analysis process is triggered. This process can be executed by a dedicated module within the core reasoning agent or transferred to a separate analysis agent, which is responsible for fine-grained analysis and risk assessment of abnormal behavior.

[0029] S420: The system further drives the large-scale model to perform fine-grained semantic classification of abnormal behavior based on the preliminary judgment (S340 / S350) and the original scene representation (S240), generate an interpretable natural language event summary, and assess the potential risk level. Fine-grained semantic classification categorizes abnormal behavior into a predefined multi-level classification system. For example, the first-level classification includes broad categories such as "property-related," "personal-related," and "rule violation." Second-level classifications are more specific, such as "property-related - theft," "property-related - vandalism," "personal-related - assault," "personal-related - harassment," "rule violation - trespassing," and "rule violation - illegal operation." The event summary is a 50-200-word natural language description that clearly explains the five W elements of the abnormal event: who, when, where, what, and why. Risk assessment includes an evaluation of the potential degree of harm caused by the incident (low, medium, and high) and the scope of possible impact (individual, small group, and large area).

[0030] S430: The system outputs a structured, in-depth analysis report. This report uses a standardized JSON format and contains the following key fields: abnormal event ID, occurrence time, location, fine-grained classification of abnormal behavior (primary and secondary categories), involved entities (people and objects), event summary text, risk level assessment, confidence level indicator, and a list of key observation points supporting the analysis. The report also includes metadata fields such as analysis timestamp, model version used, and analysis agent ID to facilitate subsequent traceability and auditing.

[0031] S500: Intelligent alerting, recording, and action recommendation steps (performed by the reporting and action agent) S510: Generate Intelligent Alerts: Based on the in-depth analysis report (S430), customized alert messages are generated for different recipients (e.g., security personnel, management, emergency response teams). Alert messages include an event summary, location information (possibly with a map or floor plan marker), risk level, and recommended response measures. Alerts can be delivered via various channels, such as mobile app push, SMS, email, or dedicated security terminals.

[0032] S520: Record comprehensive event data: Store the complete information of the event into the system database, including: the storage path or reference of the original video clip (S110), the keyframe sequence (S110), the extracted skeletal sequence data (S120), the JSON representation of the scene graph (S230), the scene description text generated by the large model (S220), the anomaly judgment result (S340), the fine-grained classification result (S420), the risk assessment (S420), and the key inference path of the large model (such as attention weight heat map or decision basis explanation). These data are stored in a structured format for subsequent query, analysis and audit.

[0033] S530: Generate action recommendations: For abnormal events of medium and high risk level (S420), the system also generates specific action recommendations, such as "immediately dispatch the nearest security personnel to the scene", "lock the relevant area entrance", "start the specific emergency plan", etc. These recommendations are generated based on event type, risk level and pre-set response protocol, aiming to guide the personnel receiving the alarm to take appropriate measures.

[0034] S540: Event tracking and updating: The system continuously monitors the relevant area, tracks the development of the abnormal event, and updates the alarm and recommendations when the situation changes. For example, if an event initially judged as "suspicious behavior" escalates to an explicit threat, the system will automatically update the risk rating and action recommendations.

[0035] In step S120 of the embodiment, the pose features extracted by the perception agent are discrete pose categories obtained based on skeletal point analysis, which are used to enrich the semantic information of the human nodes in the scene graph (S230). Specifically, the perception agent integrates a special pose classification model that receives the original skeletal point coordinate sequence as input and outputs predefined pose category labels. The predefined pose categories include: Basic static poses: standing, sitting, squatting, lying down; Basic dynamic poses: walking, running, jumping, crawling; Interactive poses: shaking hands, hugging, pushing, fighting; Abnormal poses: falling, convulsions, unnatural twisting; Gesture poses: waving, pointing, holding objects; Each posture category is assigned a confidence score (a floating-point number between 0 and 1) indicating the degree of certainty in the classification. These discrete posture categories serve as attribute fields for character nodes in the scene graph (S230), significantly enriching the scene graph's semantic expressiveness. For example, a character node in a scene graph not only contains a visual description such as "male in red" but also action state information such as "running" or "falling." This enables subsequent abnormal behavior analysis (S300 and S400) to more accurately understand the character's behavior.

[0036] In addition, the posture classification model can also identify complex posture patterns of multi-person interactions, such as "A pushes B", "multi-person gathering", "crowd evacuation", etc. This information is also encoded into the scene graph (S230) as a supplementary feature of the relationship between entities.

[0037] In this embodiment, the “training” or adaptation of the large model in the core reasoning agent includes three key aspects: 1. Model fine-tuning: Using sample pairs containing (structured multimodal scene representation data packages, RAG simulation retrieval context, corresponding behavior paradigm conformance labels / preliminary anomaly judgment labels, fine-grained anomaly category labels, and event summaries), supervised fine-tuning or instruction fine-tuning is performed on the pre-trained base LLM / VLM. This allows it to learn to assess behavior rationality from scene representations and perform qualitative and quantitative anomaly analysis. Specifically, the fine-tuning process uses the following steps: Data Preparation: We collected surveillance video datasets covering various scenarios (shopping malls, banks, schools, subway stations, etc.), and had professional security personnel annotate anomalous behaviors. For each video clip, we generated complete processing chain data, including extracted scene graphs, text descriptions, gesture features, simulated RAG retrieval results, and corresponding annotation labels (anomaly propensity score, anomaly type, fine-grained classification, and event summary).

[0038] Model selection: Choose a suitable pre-trained large model as the base model, such as GPT-4, Qwen2.5VL-2, or other open source large models. The base model should have strong language understanding capabilities and multimodal processing potential.

[0039] Fine-tuning methods: supervised fine-tuning or instruction fine-tuning methods are used. Supervised fine-tuning uses a standard cross-entropy loss function to train the model to predict correct anomaly assessment results; instruction fine-tuning uses carefully designed command-response pairs to guide the model to learn to analyze abnormal behavior according to specific formats and standards.

[0040] Training Process: Using the prepared dataset, the model is trained on appropriate computing resources (such as multi-GPU servers or cloud computing platforms). Techniques such as gradient accumulation and mixed-precision training are used to optimize memory usage and training speed. Training is performed using mini-batch gradient descent, with a learning rate starting at 1e-5 and a cosine annealing schedule.

[0041] Evaluation and selection: Use an independent validation set to evaluate model performance, focusing on metrics such as anomaly detection accuracy, recall, and F1 score. Select the model checkpoint that performs best on the validation set as the final model for deployment.

[0042] 2. Prompt Engineering: Design and optimize prompt templates that guide the model to perform specific evaluation tasks, utilize RAG context, and output structured information as needed. Prompt engineering includes the following steps: Template design: Design specialized prompt templates for different scenario types and analysis tasks. Each template includes task instructions, input data format instructions, expected output format instructions, and RAG context integration guidance.

[0043] Variable placeholders: Set variable placeholders in the template to insert actual scene representation data, RAG search results and other dynamic content at runtime.

[0044] Instruction optimization: Through experimentation and iteration, optimize the clarity, specificity, and guidance of instructions to ensure that the model can accurately understand the task requirements and generate expected outputs.

[0045] Output structure: Design a structured output format (such as a JSON template) and explicitly instruct the model in the prompt to output results in this format to facilitate subsequent processing and parsing.

[0046] Examples include: Include a few-shot examples in the prompt to show the correspondence between input data and expected output, helping the model better understand the task.

[0047] 3. RAG knowledge base construction and optimization: Continuously collect and update knowledge related to monitoring scenarios, such as safety rules, standard operating procedures, normal behavior patterns, and typical abnormal case characteristics, and optimize their organization and retrieval strategies. RAG knowledge base construction includes: Knowledge Collection: Collect relevant knowledge from sources such as safety manuals, industry standards, regulations and policies, and historical case studies. This knowledge includes safety regulations for different locations, descriptions of normal behavior, characteristics of abnormal behavior, and countermeasures.

[0048] Knowledge Organization: The collected knowledge is classified and labeled according to dimensions such as scenario type, behavior category, risk level, etc., forming structured knowledge entries. Each knowledge entry contains fields such as title, content, applicable scenario, keyword tags, etc.

[0049] Vectorized Indexing: Using advanced text embedding models (such as BERT, Sentence-BERT, etc.) to convert knowledge entries into high-dimensional vector representations and establish vector indexes, supporting efficient retrieval based on semantic similarity.

[0050] Retrieval Strategy Optimization: Design and optimize retrieval query construction strategies, including keyword extraction, query expansion, result reordering, etc., to improve the relevance and accuracy of retrieval results.

[0051] Knowledge Update Mechanism: Establish regular update and dynamic expansion mechanisms to ensure that the knowledge base content remains up-to-date and can be adjusted according to new emerging abnormal patterns and security requirements.

[0052] In this embodiment, the "indicator measuring the degree of deviation from the normal range" output by the large model in the core reasoning agent is used to trigger the subsequent analysis process. Specifically, this indicator is a floating-point number between 0 and 1, called "Anomaly Tendency Score" (ATS), where 0 represents completely normal behavior and 1 represents extremely abnormal behavior. The system sets multiple threshold levels to trigger different levels of response: When ATS < 0.3, the behavior is judged as "normal", the system continues regular monitoring and does not trigger special processing. When 0.3 ≤ ATS < 0.7, the behavior is judged as "slightly deviated" or "suspicious behavior", the system increases the attention to the relevant area and increases the sampling frequency, but does not immediately trigger deep analysis or alarm. When ATS ≥ 0.7, the behavior is judged as "significant abnormality", the system automatically triggers the fine-grained analysis process, starts the deep analysis module of the deep analysis agent or the core reasoning agent, and performs fine-grained classification, explanation generation and risk assessment of abnormal behavior. In addition, the system also considers the time rate of change of ATS. If the ATS rapidly rises (such as increasing by 0.3 or more) in a short period of time (such as 5-10 seconds), even if the absolute value has not reached 0.7, the system will trigger the deep analysis process to deal with the possible rapid deterioration. The system also allows dynamic adjustment of these thresholds according to different scenario types and security requirements. For example, in high-security level places (such as bank vaults, important facilities, etc.), the threshold for triggering deep analysis may be lowered to 0.5 or lower to increase the sensitivity of security protection.

[0053] The present application also provides a system for monitoring human abnormal behavior based on a large model multi-agent, which is deployed on a backend server and connected to a monitoring camera, comprising: S610: Perception Agent: This is a software module or microservices that includes a built-in video stream processing unit, keyframe sampling logic, and a sophisticated visual processing module for extracting or receiving human posture information and converting it into structured posture features. The Perception Agent is responsible for executing the perception and feature preparation steps defined in claim 1(a). Specifically, the Perception Agent consists of the following components: Video Stream Receiver: Responsible for establishing a connection with the surveillance camera network and receiving real-time video stream data. It supports multiple video formats and protocols, such as RTSP, RTMP, and HLS, and can process multiple video inputs simultaneously.

[0054] Keyframe Sampler: This implements an adaptive sampling algorithm that dynamically adjusts the sampling frequency based on scene changes. The sampler includes a scene change detection module that determines scene activity by calculating difference metrics (such as pixel difference and feature difference) between consecutive frames.

[0055] Human detection and tracking module: Uses target detection algorithms (such as YOLO, SSD, or Faster R-CNN) to detect human bodies in video frames, and uses multi-target tracking algorithms (such as DeepSORT, ByteTrack) to establish correspondence between human IDs between consecutive frames.

[0056] Pose Estimation Module: Integrates with OpenPose, MediaPipe, or other pose estimation libraries to extract skeletal coordinates from detected human regions. Supports 2D and 3D pose estimation and can handle multi-person scenes and partial occlusion.

[0057] Pose classifier: Based on the extracted skeleton point sequence, a temporal convolutional network (TCN) or a long short-term memory network (LSTM) is used to classify human actions and output predefined pose categories and confidence levels.

[0058] Data packaging and transmission module: Packages the processing results (key frames, skeleton point coordinates, posture classification results) into a standard format and transmits them to the scene understanding agent through the system's internal communication mechanism.

[0059] S620: Scene Understanding Agent: This agent, as one or more independent software modules or microservices, has as its core components one or more instances of a large visual model or its call interface, along with input data preprocessing logic and algorithms for constructing and enhancing a temporal dynamic scene graph based on VLLM outputs (triples, scene text descriptions) and pose features. The agent is responsible for executing the steps defined in claim 1(b). Specifically, the scene understanding agent consists of the following components: Data reception and buffering module: Receives multimodal data from the perception agent and maintains a time window buffer to capture sufficient temporal context.

[0060] Vision Large Model Interface: encapsulates the invocation of pre-trained vision large models (such as GPT-4V, CLIP-ViT, etc.). Includes input preprocessing (such as image scaling, format conversion), API call management, result parsing, and other functions.

[0061] Scene Graph Construction Engine: based on the entity, relationship, and description information output by the vision large model, constructs an initial scene graph. Uses a graph data structure to represent the scene, with nodes storing entity information and edges storing relationship information.

[0062] Temporal Scene Graph Enhancer: integrates the scene graphs of consecutive time windows into a temporal dynamic scene graph, capturing the temporal evolution of entities and relationships. Implements entity cross-frame matching, relationship consistency maintenance, and time tagging.

[0063] Pose Feature Integration Module: associates the human pose features (such as pose classification descriptions) provided by the perception agent with the person nodes in the scene graph, as node attributes for enhancement.

[0064] Data Packet Generator: packages the generated temporal scene graph, scene text description, and integrated pose features into a structured multi-modal scene representation data packet, and passes it to the core reasoning agent.

[0065] S630: Core Reasoning Agent: as one or more independent software modules or microservices, its core components are one or more fine-tuned text large models or multi-modal large model instances, and it is equipped with a Prompt management module, a RAG module (connected to an external knowledge base), and input and output data processing logic; responsible for executing the defined large model-based scene semantic compliance evaluation and preliminary abnormality judgment steps, and deep analysis steps. Specifically, the core reasoning agent is composed of the following components: Data Reception and Parsing Module: receives structured multi-modal scene representation data packets from the scene understanding agent, and parses the scene graph, text description, and pose features in it.

[0066] Prompt Management Module: maintains a library of Prompt templates for different scene types and analysis tasks, and dynamically generates suitable analysis Prompts based on the current scene and task requirements.

[0067] RAG Module: constructs a retrieval query based on the scene representation content, initiates a query to the external knowledge base, and obtains relevant contextual auxiliary knowledge. Includes query construction, retrieval execution, result sorting and formatting, and other functions.

[0068] Large Model Reasoning Engine: encapsulates the invocation of fine-tuned large language models or multi-modal large models. Responsible for inputting the Prompt, scene representation data, and RAG retrieval results into the model, and obtaining the model's reasoning output.

[0069] Anomaly Evaluation Module: Analyze large model outputs, extract anomaly tendency scores, anomaly type judgments, and judgment basis information, and decide whether to trigger deep analysis processes according to preset thresholds.

[0070] Deep Analysis Module: When the anomaly evaluation result exceeds the threshold, perform more detailed anomaly behavior analysis, including fine-grained classification, event summary generation, and risk assessment.

[0071] Result Delivery Module: Package and deliver anomaly judgment results and deep analysis reports to the report and action agent.

[0072] S640: Report and Action Agent: As one or more independent software modules or microservices, it is used to generate and send intelligent alarms based on analysis results, and to structure the storage of related event data. Specifically, the report and action agent is composed of the following components: Alarm Generator: According to the deep analysis report, generate alarm messages customized for different recipients, including event summary, location information, risk level, and recommended measures, etc.

[0073] Multi-channel notification system: Support sending alarm messages through multiple channels (such as mobile application push, SMS, email, dedicated terminal), to ensure timely notification of relevant personnel.

[0074] Event Data Recorder: Structured storage of complete event information (including video index, skeleton sequence, scene graph, analysis result, etc.) into the system database, supporting subsequent query and analysis.

[0075] Action Suggestion Generator: For medium and high risk events, based on event type, risk level and preset response protocol, generate specific action suggestions to guide personnel receiving alarms to take appropriate measures.

[0076] Event Tracking Module: Continuously monitor relevant areas, track the development of abnormal events, and update alarms and suggestions when conditions change.

[0077] S650: Knowledge Base: An independently deployable and maintainable database system that stores domain knowledge for RAG module retrieval. Specifically, the knowledge base is composed of the following components: Knowledge Storage System: Use document databases (such as MongoDB, Elasticsearch) or vector databases (such as Pinecone, Milvus) to store structured and unstructured knowledge entries.

[0078] Vector Index: Establish a vector index based on semantic embedding for knowledge entries to support efficient similarity retrieval.

[0079] Knowledge Management Interface: Provides interfaces for adding, updating, deleting, and querying knowledge items, supporting the maintenance and expansion of the knowledge base.

[0080] Knowledge Classification System: Implements a multi-dimensional knowledge classification and tagging system for precise retrieval and filtering.

[0081] Update Mechanism: Supports regular batch updates and real-time incremental updates to ensure the latest knowledge content.

[0082] S660: Agent Communication and Orchestration Infrastructure: Includes message queue services, API gateways, service discovery mechanisms, and optional workflow engines to manage agent registration, discovery, data exchange, and task flow. Specifically, this infrastructure consists of the following components: Message Queue Service: Uses RabbitMQ, Kafka, or other message middleware to implement asynchronous communication and data transfer between agents.

[0083] API Gateway: Provides a unified API interface to manage service request routing, load balancing, authentication and authorization, etc.

[0084] Service Registration and Discovery: Uses Consul, Etcd, or other service registration centers to achieve dynamic registration and discovery of agent services.

[0085] Workflow Engine: Uses Apache Airflow, Temporal, or custom workflow systems to orchestrate and manage complex agent collaboration processes.

[0086] Monitoring and Logging System: Collects and analyzes system running status, performance indicators and log information, supports problem diagnosis and system optimization.

[0087] Configuration Management: Centralized management of system configuration parameters, supports dynamic configuration updates and environment isolation.

[0088] In this embodiment, the large model of the core reasoning agent is fine-tuned and Prompt designed to enable it to understand the complex interactions between entities in the scene graph, time evolution logic, and semantic relevance between individual behavior and overall scene context, and to evaluate the "paradigm conformity" and "context rationality" of behavior based on this. Specifically: The fine-tuning process focuses on scene graph understanding capabilities: training data contains a large number of monitoring scene samples with detailed scene graph annotations, enabling the model to learn to parse graph structure data, understand the semantics of nodes (entities) and edges (relationships), and track the changes in entity state over time.

[0089] The fine-tuning dataset contains rich interaction pattern samples: the dataset covers various human-human interactions (such as conversations, transactions, conflicts), human-object interactions (such as picking up, placing, using objects), and human-environment interactions (such as entering an area, using facilities), enabling the model to identify normal and abnormal patterns of different interactions.

[0090] Prompt design guides the model to focus on the time dimension: the prompt template contains special instructions to guide the model to analyze the time evolution logic of entity behavior and relationships, and identify abnormal timing patterns (such as behavior mutation, abnormal duration, unreasonable behavior sequence, etc.).

[0091] Prompt design emphasizes context relevance analysis: the template contains instructions to guide the model to evaluate the matching degree of individual behavior and overall scene context, for example, whether a certain behavior is reasonable in a specific place, at a specific time, and in a specific surrounding environment.

[0092] Paradigm conformity evaluation framework: during fine-tuning, the model learns to use a multi-dimensional framework to evaluate the paradigm conformity of behavior, including: social norm dimension (whether the behavior conforms to socially recognized behavior standards), place rule dimension (whether the behavior conforms to the regulations of a specific place), situational rationality dimension (whether the behavior is consistent with the current situation), and historical consistency dimension (whether the behavior is consistent with the subject's historical behavior pattern).

[0093] Situational rationality evaluation capability: the model learns a large number of behavior samples with situational annotations, developing the ability to judge the rationality of behavior in different situations. For example, it is reasonable to wait in line at a bank counter, but it may be abnormal to stay for a long time in front of an emergency exit; Multi-modal understanding and integration: fine-tuning and prompt design enable the model to effectively integrate multi-modal input such as scene graph structure information, text description, and pose features to form a comprehensive understanding of the scene, rather than relying solely on single-modal information.

Claims

1. A method for monitoring abnormal human behavior based on a large model and multiple agents, characterized in that: The process is executed by a modular multi-agent system deployed on a backend server and includes the following steps: (1) The perception agent obtains the video stream from the surveillance camera, performs key frame smoothing sampling, extracts human posture information and processes it into structured posture features; (2) The scene understanding agent uses the visual large model (VLLM) to analyze key frames, extract scene entities, relationships, and text descriptions, construct a temporal dynamic scene graph, and use the posture features as character node attributes to enhance the scene graph, and output a structured multimodal scene representation data package; (3) The core reasoning agent uses a fine-tuned text / large multimodal model (LLM / VLM) and external knowledge dynamically retrieved by the retrieval enhancement generation RAG module to perform semantic understanding of the data packet, evaluate the paradigm conformity and situational rationality of the behavior, and output an abnormal tendency score and preliminary judgment; (4) If the abnormal propensity score exceeds the threshold, the large model is driven to perform fine-grained classification of abnormal behavior, generate a natural language event summary, and assess the risk level; (5) The reporting and action agent generates customized alerts, stores structured data across the entire chain, and proposes action recommendations based on risk levels.

2. A method for monitoring abnormal human behavior based on a large model and multiple agents according to claim 1, characterized in that: In step (1), the posture feature processing is as follows: OpenPose or MediaPipe is used to extract the coordinates of 17 key skeleton points; the skeleton point sequence is processed into two types of structured data: the original coordinate sequence and the posture classification label; the posture classification label is selected from predefined discrete categories, including static posture, dynamic posture, interactive posture, abnormal posture and hand posture.

3. The method for monitoring abnormal human behavior based on a large model and multiple agents according to claim 1 is characterized in that: In step (2), the construction of the temporal dynamic scene graph is as follows: the visual large model outputs scene entity attributes, entity relationship triples and scene text description; the scene graph within 3-10 seconds is integrated into a temporal dynamic structure to capture the time evolution of entity relationships; the action state attributes of the character node are enhanced through posture classification labels.

4. The method for monitoring abnormal human behavior based on a large model and multiple agents according to claim 1, characterized in that: In step (3), the anomaly assessment mechanism is as follows: Prompt templates are dynamically selected based on the scenario type, and scenario graph data and RAG search results are inserted; the RAG module retrieves security rules, behavioral patterns, and abnormal case characteristics from the knowledge base and returns the top 3-5 relevant entries; the anomaly propensity score is a continuous value from 0 to 1, and a multi-level threshold triggers a response; among them, ATS < 0.3 is normal; 0.3 ≤ ATS < 0.7 is suspicious behavior; ATS ≥ 0.7 triggers in-depth analysis.

5. The method for monitoring abnormal human behavior based on a large model and multiple agents according to claim 1 is characterized in that: In step (4), the fine-grained classification includes: the first-level classification is property-related, personal-related or rule violation; the second-level classification is based on the refinement of the first-level category, including theft, destruction, attack, harassment, breaking into restricted areas or illegal operation; the event summary contains five elements: the subject involved, time, location, behavior description and abnormal basis.

6. A human abnormal behavior monitoring system based on a large model and multiple agents, characterized by: include: Perception agent module: used to integrate the video stream processing unit, key frame sampling module, skeleton point extraction and posture classification model; Scene understanding agent module: used for visual large model interface, temporal dynamic scene graph construction engine and posture feature integration module; Core reasoning agent module: used for built-in LLM / VLM model, Prompt management module, RAG module and anomaly assessment module; Reporting and Action Agent Module: used to realize alarm generation, multi-channel notification, event data recording and action recommendation generation functions; The knowledge base is used to store security rules, behavioral patterns, and abnormal case characteristics, supporting semantic retrieval of the RAG module; The agent communication and orchestration infrastructure is used to achieve multi-agent collaboration through message queues, API gateways, and workflow engines.

7. The human abnormal behavior monitoring system based on large model multi-agent according to claim 6 is characterized in that: The posture classification model of the perception agent is as follows: use the temporal convolutional network (TCN) or the long short-term memory network (LSTM) to process the skeleton point sequence; output composite posture pattern labels, including pushing, group gathering, or crowd evacuation.

8. The human abnormal behavior monitoring system based on large model multi-agent according to claim 6 is characterized in that: The fine-tuning of the core reasoning agent is as follows: LLM / VLM is trained using annotated multimodal scene representation data packages, with annotations including anomaly propensity scores, fine-grained classifications, and event summaries; fine-tuning focuses on scene graph parsing capabilities, covering human-human, human-object, and human-environment interaction modes; Prompt design guides the model to analyze time evolution logic and behavior-scene context matching.

9. The human abnormal behavior monitoring system based on large model multi-agent according to claim 6 is characterized in that: The specific steps for building a knowledge base are: organizing knowledge entries by scenario type, behavior category, and risk level; using BERT or Sentence-BERT to generate vector indexes and support semantic similarity retrieval; setting up a regular update mechanism to dynamically expand new abnormal patterns and security rules.

10. The human abnormal behavior monitoring system based on large model multi-agent according to claim 6 is characterized in that: The reporting and action agents are as follows: Generate specific instructions for medium and high-risk events, including dispatching security personnel, locking down areas, or activating emergency plans; continuously track the development of events and dynamically update risk ratings.

Images