Abnormal behavior real-time early warning system based on multi-agent asynchronous cooperation
Patent Information
- Application Number
- CN202510917059.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-03
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2045-07-03
AI Technical Summary
[0006]有鉴于此，本发明提供了基于多智能体异步协作的异常行为实时预警系统，以解决复杂城市监控环境下实时性不足、动态事件理解能力有限及异常行为解释性不强等技术难题
[0035]1.本发明提供了基于多智能体异步协作的异常行为实时预警系统，提出基于Redis通信的多智能体异步协作框架，将异常行为检测任务拆解为实时视频帧处理、历史信息建模、异常事件推理三大任务，由不同智能体独立执行，并通过缓存服务器实现高效并行与多阶段异步通信，本发明通过对大规模实时视频流的异步、并行处理，实现了实时性与系统性显著提升，避免了离线处理的延迟和信息丢失，显著提升了城市监控的响应速度与效率。该系统融合图像帧序列、历史事件描述等多模态信息，通过异常行为判别模型实现对连续帧行为和语义演变的动态建模和异常预警，提升了针对动态异常时间的理解能力以及针对异常行为的解释性。
Smart Images
Figure CN120808269B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent monitoring technology, specifically to a real-time early warning system for abnormal behavior based on asynchronous cooperation among multiple agents. Background Technology
[0002] With the increasing demand for urban security, how to conduct vision-based abnormal behavior detection and early warning technology for abnormal behaviors such as theft or crime detected in surveillance has become a key technical aspect of intelligent surveillance systems.
[0003] Currently, most mainstream methods rely on single visual features or offline data processing, which struggles to meet the demands of high real-time performance, strong interpretability, and understanding of dynamic and complex events in urban surveillance scenarios. In large-scale surveillance networks, cameras generate massive amounts of data, and theft often manifests as continuous actions and complex semantic relationships. How to automatically and accurately identify and warn of theft while maintaining real-time performance has become a core problem urgently needing to be solved in the field of intelligent surveillance. Existing abnormal behavior detection systems mostly employ shallow feature understanding methods based on images or video frames, lacking the ability to model and reason about historical semantic information. While methods based on Visual Language Models (VLMs) can integrate and interpret historical semantic information, they can only be applied to offline video data and cannot meet the real-time requirements of the video surveillance field.
[0004] First, existing theft detection methods based on object detection technology cannot effectively model the temporal semantic information in video streams, making it difficult to provide early warnings of theft. Second, VLM-based theft detection algorithms often rely on offline batch processing, failing to achieve real-time analysis and early warning in complex urban scenarios, which can lead to delays in handling abnormal behavior and affect the efficiency of urban security response. Finally, existing research lacks high-quality, multi-scenario theft behavior datasets, resulting in insufficient generalization ability of the systems when facing diverse urban scenarios and difficulty in adapting to environmental changes.
[0005] This demonstrates that existing monitoring systems suffer from technical challenges such as insufficient real-time performance, limited ability to understand dynamic events, and weak interpretability of abnormal behavior. Summary of the Invention
[0006] In view of this, the present invention provides a real-time early warning system for abnormal behavior based on asynchronous cooperation of multiple agents, in order to solve the technical problems of insufficient real-time performance, limited ability to understand dynamic events, and weak interpretability of abnormal behavior in complex urban monitoring environments.
[0007] To achieve the above objectives, the technical solution of the real-time early warning system for abnormal behavior based on asynchronous cooperation of multiple agents provided by the present invention includes the following steps:
[0008] Taking real-time video streams as the research object, a multi-agent asynchronous communication framework is constructed.
[0009] Based on a multimodal visual language model architecture, and by integrating long-term historical prior information, real-time identification and prediction of abnormal behavior can be achieved.
[0010] Preferably, taking real-time video streams as the research object, a multi-agent asynchronous communication framework is constructed, specifically as follows:
[0011] The abnormal behavior detection task is broken down into three main tasks: real-time video frame processing, historical information modeling, and abnormal event reasoning. These tasks are executed independently by different intelligent agents, and efficient parallelism and multi-stage asynchronous communication are achieved through a cache server.
[0012] Preferably, the multi-agent asynchronous communication framework specifically includes three agents for handling different tasks, and three Redis cache servers to support multi-stage asynchronous communication and information sharing.
[0013] The three intelligent agents are designated as the first to third intelligent agents, and the three cache servers are designated as the first to third cache servers.
[0014] First, the first cache server is used to receive and distribute real-time frame image data, and push it to the first and third intelligent agents in real time.
[0015] The first intelligent agent is responsible for processing the received image frames, extracting semantic features and generating text descriptions frame by frame, and then publishing the generated description information and corresponding timestamps through the second cache server.
[0016] The second agent, acting as a relay processing module, subscribes to frame-level image descriptions from the second cache server. After collecting a certain amount of data, it integrates the temporal structure and semantic content to construct summary information covering historical time periods. The summary information not only contains the semantic features of key image events but also reflects the evolutionary trend of events over time. The summary information and the corresponding historical time period timestamps are published to the third cache server.
[0017] The third intelligent agent subscribes to the summary information published by the third cache server, and at the same time obtains the six most recent frames of images to make a comprehensive judgment on the current scene in order to identify whether there are abnormal events or warning signals that may be abnormal in the future. If an anomaly is detected, the system will output a structured anomaly judgment result containing the event participants, the location of the anomaly, the inferred cause, and the corresponding timestamp.
[0018] Preferably, the first intelligent agent extracts semantic features and generates single-frame descriptions frame by frame in the following manner:
[0019] The BLIP model is used to perform semantic annotation on image frames, for real-time image frames I1, I2, ..., In Let the model mapping function be f(.), then the final generated single frame description is:
[0020] g i ,t i =f(I i )
[0021] Where i is the real-time image frame index, g i It is the generated image description, t i This is the timestamp of the image frame.
[0022] Preferably, the second intelligent agent constructs summary information covering historical time periods in the following manner:
[0023] The set of text sequences describing a single frame generated by the first intelligent agent is Where, N g The maximum length of the buffer sequence for a single frame description; the prompt word is "prompt", the keyword is "key", the information redundancy filtering function is "h(.)", and the Qwen-long language model is... The final generated summary information h covering historical time periods s for:
[0024] Preferably, the third intelligent agent makes a comprehensive judgment on the current scene in the following manner:
[0025] Combining summary information with information from neighboring image frames for anomaly warning and discrimination can be formally represented as follows:
[0026]
[0027] Among them, h s For long-term historical information, I i = [I1,I2,...,I6] represents the six nearest image frames; MLLM is an anomaly event discrimination model based on LLaVA-1.5. The output is a triple containing the location (where), the person (who), and the reason (why).
[0028] Preferably, MLLM is an anomaly event discrimination model based on LLaVA-1.5, specifically as follows:
[0029] The anomaly event discrimination model based on LLaVA-1.5 receives input data in two modalities: visual input and text input. The visual input consists of six adjacent image frames, representing a series of image frames sampled from a video or continuous scene; the text input is summary information.
[0030] Visual input and text input are processed by separate encoders for feature extraction. Visual input is first fed into CLIP ViT-L / 224 encoder, which generates a visual feature vector for each frame. The sequence of visual feature vectors extracted from CLIP is then processed by a multilayer perceptron (MLP). The MLP layer acts as an adapter, projecting the visual features into a new space that is compatible with the dimensions of the text features, ultimately resulting in a sequence of visual feature vectors.
[0031] The summary information first passes through a tokenizer, which segments the natural language sentence into a series of tokens. Then, the tokens are converted into dense numerical vectors through an embedding layer. This process generates two types of feature vectors: identifier feature vectors and text feature vectors. Identifier feature vectors contain positional encodings and special delimiters. Text feature vectors represent the semantic content of the tokens.
[0032] The abnormal event discrimination model based on LLaVA-1.5 includes a temporal location reconstruction module. The temporal location reconstruction module integrates feature vectors from different encoders into a unified sequence. Specifically, the processed visual feature vector sequence is embedded in the middle of the text and identifier feature vector sequences to obtain a fused sequence. The fused sequence is input to Qwen-4B.
[0033] Qwen-4B performs deep analysis on the input fused sequence and generates a structured text report; the text report includes the location (Where), the person (Who), and the reason (Why).
[0034] Beneficial effects:
[0035] 1. This invention provides a real-time early warning system for abnormal behavior based on multi-agent asynchronous collaboration. It proposes a multi-agent asynchronous collaboration framework based on Redis communication, decomposing the abnormal behavior detection task into three main tasks: real-time video frame processing, historical information modeling, and abnormal event reasoning. These tasks are executed independently by different agents, and efficient parallelism and multi-stage asynchronous communication are achieved through a caching server. This invention significantly improves real-time performance and systemicity by asynchronously and parallelly processing large-scale real-time video streams, avoiding the latency and information loss of offline processing, and significantly improving the response speed and efficiency of urban monitoring. The system integrates multimodal information such as image frame sequences and historical event descriptions, and uses an abnormal behavior discrimination model to achieve dynamic modeling and abnormal early warning of continuous frame behavior and semantic evolution, improving the understanding of dynamic abnormal events and the interpretability of abnormal behavior.
[0036] 2. This invention achieves dynamic capture and accurate early warning of behavioral evolution through multimodal fusion, long-term historical modeling and structured input design, rather than making judgments based solely on static features of a single frame.
[0037] 3. This invention decouples the anomaly detection output and includes event details (such as participants, location, speculated causes, timestamps, etc.), which facilitates subsequent tracing and law enforcement decision-making, while reducing the application risks brought about by the "black box" model. Attached Figure Description
[0038] Figure 1 The flowchart is shown for a real-time early warning system for abnormal behavior based on asynchronous cooperation of multiple agents.
[0039] Figure 2 This is a schematic diagram of the multi-stage data processing of a real-time early warning system for abnormal behavior.
[0040] Figure 3 This is a structural diagram of a multimodal temporal reasoning model. Detailed Implementation
[0041] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.
[0042] This invention proposes a real-time early warning system for abnormal behavior based on asynchronous multi-agent collaboration. Taking real-time video streams as the research object, it constructs an asynchronous multi-agent communication framework and integrates long-term historical prior information based on a multimodal visual language model architecture to achieve real-time identification and prediction of abnormal behavior events.
[0043] Asynchronous multi-agent collaboration framework based on Redis communication
[0044] To address the limitations of single-agent architectures in simultaneously meeting the demands of real-time performance, temporal modeling, and multi-task collaborative processing, this invention presents a multi-agent asynchronous collaboration framework based on Redis communication. This architecture, through task decoupling and information sharing, achieves efficient parallel processing of multi-stage processes and real-time semantic reasoning, thereby significantly enhancing the system's comprehensive perception and response capabilities to dynamic events. It enables real-time detection and semantic reasoning of abnormal behavioral events in image sequences, such as... Figure 1 As shown.
[0045] The multi-agent asynchronous communication framework specifically includes three agents for handling different tasks and three Redis cache servers to support multi-stage asynchronous communication and information sharing. The three agents are designated as Agent 1 to Agent 3, and the three cache servers are designated as Cache Server 1 to Cache Server 3.
[0046] First, the first cache server receives and distributes real-time frame image data, pushing it to the first and third intelligent agents in real time. The first intelligent agent processes the received image frames, extracts semantic features, and generates text descriptions frame by frame. It then publishes the generated descriptions and corresponding timestamps through the second cache server. The second intelligent agent, acting as a relay processing module, subscribes to frame-level image descriptions from the second cache server. After collecting a certain amount of data, it integrates the temporal structure and semantic content to construct summary information covering historical time periods. This summary not only contains the semantic features of key image events but also reflects the event's temporal evolution trend. The summary information and corresponding historical time period timestamps are published to the third cache server. The third intelligent agent subscribes to the summary content published by the third cache server and simultaneously acquires the six most recent images. It performs a comprehensive judgment on the current scene to identify whether there are any abnormal events or warning signals indicating potential future anomalies. If an anomaly is detected, the system outputs a structured anomaly judgment result containing event participants, anomaly location, speculated causes, and corresponding timestamps.
[0047] The dataset of this invention collects video data related to anomalous behavior from UCF-Crime, the largest currently available weakly supervised anomaly detection (VAD) dataset, and YouTubeClips, a dataset of normal video clips. The dataset generation process is as follows: Figure 2 As shown, the process begins with manual screening of the original videos to remove samples with poor image quality or content that does not meet requirements. Then, through manual cropping and screening, segments of unusual events covering six categories of abnormal behavior—shoplifting, theft, robbery, and burglary—are extracted. Finally, high-quality normal video clips are selected from YouTubeClips for comparative learning by the model.
[0048] The first intelligent agent extracts semantic features and generates single-frame descriptions frame by frame in the following manner: Accurately generating semantic descriptions for each frame helps improve subsequent anomaly detection and analysis. Therefore, this invention uses the BLIP model to perform semantic annotation on each frame of the video. For real-time image streams I1, I2, ..., I... n Let the model mapping function be f(.), then the final generated frame-level description is:
[0049] g i ,t i =f(I i )
[0050] Where i is the real-time image frame index, g i It is the generated image description, t i This is the timestamp of the image frame.
[0051] The second intelligent agent constructs summary information covering historical time periods in the following manner:
[0052] Based on single-frame image description, this invention designs a method for summarizing and filtering the formed historical frame cache sequences using Qwen-long prompt word engineering, keyword task information extraction, and information redundancy filtering. Let the set of formed frame-level image description text sequences be... Where, N g Let `prompt` be the prompt word, `key` be the keyword, `h(.)` be the information redundancy filtering function, and `Qwen-long` be the language model. The final generated historical summary h s for:
[0053]
[0054] The third intelligent agent makes a comprehensive judgment on the current scene using the following method:
[0055] To fully extract semantic information from event fragments, this invention combines long historical information with information from adjacent image frames for anomaly warning and discrimination, which can be formally represented as:
[0056]
[0057] Among them, h s For long-term historical information, I i = [I1, I2, ..., I6] represents neighboring image frames. MLLM is an abnormal behavior discrimination model based on LLaVA-1.5. The output is a triple containing the location (where), the person (who), and the reason (why).
[0058] Anomaly detection model based on LLaVA-1.5
[0059] To address the issues of the LLaVA-1.5 model's small context window, which hinders its ability to understand long-term time-series information and distinguish boundaries between adjacent functional modules, this invention develops an abnormal behavior discrimination model based on the LLaVA-1.5 model, such as... Figure 3 As shown.
[0060] First, the model receives input data in two modalities. The visual input, as shown in the upper left corner of the figure, consists of the current six frames, representing a series of image frames sampled from a video or continuous scene. This forms the visual basis for the model's analysis of dynamic events. The text input is as follows... Figure 3 As shown in the top right corner, the input is a historical summary. This text provides contextual information about previous events, offering background knowledge for the model's reasoning.
[0061] Secondly, to enable the model to understand data from different modalities, visual and text inputs undergo feature extraction via independent encoders. The visual input (6 frames of images) is first fed into the CLIP ViT-L / 224 encoder, generating a high-dimensional "visual feature vector" for each frame. The sequence of visual feature vectors extracted from CLIP then passes through a multilayer perceptron (MLP). This MLP layer acts as an adapter, projecting the visual features into a new space compatible with the dimensions of the text features, ultimately yielding a sequence of visual feature vectors usable by subsequent modules.
[0062] The historical summary text first passes through a tokenizer, which segments the natural language sentence into a series of tokens. These tokens are then transformed into dense numerical vectors through an embedding layer. This process generates two types of feature vectors: identifier feature vectors (containing meta-information such as positional encoding and special delimiters) and text feature vectors (representing the semantic content of the tokens).
[0063] Furthermore, to enable the language model to simultaneously understand visual events and textual context, a temporal location reconstruction module is employed. This module integrates feature vectors from different encoders into a unified sequence. Specifically, it embeds the processed visual feature vector sequence between the text and identifier feature vector sequences. This interleaved fusion strategy allows LLM to seamlessly process and associate visual and textual information within the same context, effectively combining temporal visual changes with semantic background.
[0064] Finally, leveraging the powerful contextual understanding, logical reasoning, and text generation capabilities of Qwen-4B, a deep analysis of the input fusion sequence is performed to generate a structured text report. The output text format is as follows: "Location: In an electronics store; Person: A person wearing a black jacket; Why: Detailed reasoning about the person's behavior, pointing out suspicious actions such as 'frequently looking around,' 'assessing the surrounding environment and potential surveillance,' and 'carrying a bag that could be used to hide stolen items,' ultimately concluding that 'abnormal behavior exists.'"
[0065] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. A real-time early warning system for abnormal behavior based on asynchronous cooperation of multiple agents, characterized in that, Includes the following steps: Taking real-time video streams as the research object, a multi-agent asynchronous communication framework is constructed; Based on a multimodal visual language model architecture, and by integrating long-term historical prior information, real-time identification and prediction of abnormal behavior can be achieved. The aforementioned multi-agent asynchronous communication framework, using real-time video streams as the research object, is constructed as follows: The abnormal behavior detection task is broken down into three major tasks: real-time video frame processing, historical information modeling, and abnormal event reasoning. These tasks are executed independently by different intelligent agents, and efficient parallelism and multi-stage asynchronous communication are achieved through a cache server. The multi-agent asynchronous communication framework specifically includes three agents for handling different tasks, and three Redis cache servers to support multi-stage asynchronous communication and information sharing. The three intelligent agents are designated as the first to third intelligent agents, and the three cache servers are designated as the first to third cache servers. First, the first cache server is used to receive and distribute real-time frame image data, and push it to the first intelligent agent and the third intelligent agent in real time; The first intelligent agent is responsible for processing the received image frames, extracting semantic features and generating text descriptions frame by frame, and then publishing the generated description information and corresponding timestamps through the second cache server; The second agent, acting as a relay processing module, subscribes to frame-level image descriptions from the second cache server. After collecting a certain amount of data, it integrates the temporal structure and semantic content to construct summary information covering historical time periods. The summary information not only contains the semantic features of key image events but also reflects the evolutionary trend of events over time. The summary information and the corresponding historical time period timestamps are published to the third cache server. The third intelligent agent subscribes to the summary information published by the third cache server, and at the same time obtains the six most recent frames of images to make a comprehensive judgment on the current scene in order to identify whether there are abnormal events or warning signals that may be abnormal in the future. If an anomaly is detected, the system will output a structured anomaly judgment result containing the event participants, the location of the anomaly, the inferred cause, and the corresponding timestamp.
2. The real-time early warning system for abnormal behavior as described in claim 1, characterized in that, The first intelligent agent extracts semantic features and generates single-frame descriptions frame by frame in the following manner: Semantic annotation of image frames is performed using the BLIP model for real-time image frames. I 1, I 2,…, I n Let the model mapping function be... f (.), then the final generated single frame description is: in, i For real-time image frame indexing, g i It is a description of the generated image. t i This is the timestamp of the image frame.
3. The real-time early warning system for abnormal behavior as described in claim 2, characterized in that, The second intelligent agent constructs summary information covering historical time periods in the following manner: The set of text sequences describing a single frame generated by the first intelligent agent is ,in, N g The maximum length of the buffer sequence for a single frame description; the prompt word is prompt Keywords are key The information redundancy filtering function is h (.), the Qwen-long language model is The resulting summary information covers historical time periods. h s for: .
4. The real-time early warning system for abnormal behavior as described in claim 3, characterized in that, The third intelligent agent makes a comprehensive judgment on the current scene using the following method: Combining summary information with information from neighboring image frames for anomaly warning and discrimination can be formally represented as follows: in ， For long-term historical information, These are the six nearest image frames; MLLM is an anomaly event discrimination model based on LLaVA-1.
5. The output is a triple containing the location (where), the person (who), and the reason (why).
5. The real-time early warning system for abnormal behavior as described in claim 4, characterized in that, The MLLM is an anomaly event discrimination model based on LLaVA-1.5, specifically as follows: The LLaVA-1.5-based anomaly event discrimination model receives input data in two modalities, including visual input and text input. The visual input consists of six adjacent image frames, representing a series of image frames sampled from a video or continuous scene. The text input is the summary information. Visual input and text input are processed by separate encoders for feature extraction. Visual input is first fed into CLIP ViT-L / 224 encoder, which generates a visual feature vector for each frame. The sequence of visual feature vectors extracted from CLIP is then processed by a multilayer perceptron (MLP). The MLP layer acts as an adapter, projecting the visual features into a new space that is compatible with the dimensions of the text features, ultimately resulting in a sequence of visual feature vectors. The summary information first passes through a tokenizer, which segments the natural language sentence into a series of tokens. Subsequently, the tokens are converted into dense numerical vectors through an embedding layer. This process generates two types of feature vectors: identifier feature vectors and text feature vectors. The identifier feature vectors contain positional encodings and special delimiters. The text feature vectors represent the semantic content of the tokens. The abnormal event discrimination model based on LLaVA-1.5 includes a temporal location reconstruction module; the temporal location reconstruction module integrates feature vectors from different encoders into a unified sequence. Specifically, the processed visual feature vector sequence is embedded in the middle of the text and identifier feature vector sequences to obtain a fused sequence; the fused sequence is input to Qwen-4B. Qwen-4B performs deep analysis on the input fusion sequence and generates a structured text report; the text report includes the location (Where), the person (Who), and the reason (Why).