Abnormal behavior real-time early warning system based on multi-agent asynchronous cooperation

Abstract

The invention discloses an abnormal behavior real-time early warning system based on multi-agent asynchronous cooperation, relates to the technical field of intelligent monitoring, and can solve the technical problems of insufficient real-time performance, limited dynamic event understanding ability, poor abnormal behavior interpretation and the like in a complex city monitoring environment. In order to achieve the purpose, the technical scheme of the abnormal behavior real-time early warning system based on the multi-agent asynchronous cooperation comprises the following steps: constructing a multi-agent asynchronous communication framework by taking a real-time video stream as a research object; on the basis of a multi-modal visual language model architecture, long-time-sequence historical prior information is fused, and real-time judgment and prediction of abnormal behaviors are achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent monitoring, in particular to an abnormal behavior real-time early warning system based on multi-agent asynchronous cooperation. BACKGROUND

[0002] With the continuous improvement of urban security needs, in view of some theft or criminal abnormal behaviors existing in monitoring, how to detect and warn abnormal behaviors based on vision has become a key technical link in intelligent monitoring system.

[0003] At present, the mainstream method depends on single visual feature or offline data processing, which is difficult to meet the needs of high real-time, strong interpretability and dynamic complex event understanding in urban monitoring scene. In large-scale monitoring network, the data flow generated by the camera is huge, and the theft behavior usually presents continuous action and complex semantic association. How to automatically complete the accurate identification and early warning of theft behavior under the premise of ensuring real-time has become a core problem to be solved in the field of intelligent monitoring. The existing abnormal behavior detection system mostly adopts the method of shallow feature understanding based on image or video frame, which lacks the modeling and reasoning ability of historical semantic information. Although the method based on visual language model VLM can realize the integration and interpretive description of historical semantic information, it can only be applied to offline video data and cannot meet the real-time requirements of video monitoring field.

[0004] Firstly, the existing theft behavior detection method based on target detection technology cannot effectively model the time sequence semantic information in the video stream, so it is difficult to realize the early warning of theft behavior. Secondly, the theft behavior detection algorithm based on VLM mostly relies on offline batch processing, which cannot realize real-time analysis and early warning in complex urban scenes, easily leading to delay in handling abnormal behaviors and affecting the efficiency of urban security response. Finally, the existing research lacks high-quality, multi-scene theft behavior dataset, which leads to insufficient generalization ability of the system when facing diversified urban scenes, and it is difficult to adapt to environmental changes.

[0005] Therefore, the existing monitoring system has the technical problems of insufficient real-time, limited dynamic event understanding ability and weak abnormal behavior interpretability. SUMMARY

[0006] Therefore, the present application provides an abnormal behavior real-time early warning system based on multi-agent asynchronous cooperation to solve the technical problems of insufficient real-time, limited dynamic event understanding ability and weak abnormal behavior interpretability in complex urban monitoring environment.

[0007] To achieve the above purpose, the abnormal behavior real-time early warning system based on multi-agent asynchronous cooperation provided by the present application has the technical scheme comprising the following steps:

[0008] The real-time video stream is taken as the research object, and a multi-agent asynchronous communication framework is constructed.

[0009] Based on a multi-modal visual language model architecture, long-term historical prior information is fused to realize real-time discrimination and prediction of abnormal behaviors.

[0010] Preferably, the real-time video stream is taken as the research object, and a multi-agent asynchronous communication framework is constructed, specifically as follows.

[0011] The abnormal behavior detection task is divided into three tasks of real-time video frame processing, historical information modeling and abnormal event reasoning, which are independently executed by different agents, and efficient parallel and multi-stage asynchronous communication is realized through a cache server.

[0012] Preferably, the multi-agent asynchronous communication framework specifically includes three agents for processing different tasks, and three Redis cache servers for supporting multi-stage asynchronous communication and information sharing.

[0013] The three agents are the first to third agents, and the three cache servers are the first to third cache servers.

[0014] Firstly, the first cache server is used to receive and distribute real-time frame image data, and push the data to the first agent and the third agent in real time.

[0015] The first agent is responsible for processing the received image frames, extracting semantic features and generating text descriptions frame by frame, and then publishing the generated description information and corresponding timestamps through the second cache server.

[0016] The second agent is a relay processing module, which subscribes to frame-level image descriptions from the second cache server, collects a certain amount of data, integrates the time sequence structure and semantic content, and constructs summary information covering historical periods; the summary information not only contains the semantic features of key image events, but also reflects the evolution trend of events in time; the summary information and the corresponding historical period timestamp are published to the third cache server.

[0017] The third agent subscribes to the summary information published by the third cache server, and at the same time acquires the current 6 frames of images, comprehensively judges the current scene, and identifies whether there is an abnormal event or whether there is a warning signal of a possible future abnormality; if an abnormality is detected, the system will output a structured abnormality discrimination result containing event participants, abnormal location, suspected reason and corresponding timestamp.

[0018] Preferably, the first agent extracts semantic features and generates single-frame descriptions in the following way:

[0019] The BLIP model is used to perform semantic annotation on the image frames, and for real-time image frames I1, I2, …, In , let the model mapping function be f(.), then the final generated single-frame description is:

[0020] g i ,t i = f(I i )

[0021] wherein i is the real-time image frame index, g i is the generated image description, and t i is the timestamp of the image frame.

[0022] Preferably, the second agent adopts the following way to construct the summary information covering the historical period:

[0023] The text sequence set of the single-frame description generated by the first agent is wherein N g is the maximum length of the single-frame description cache sequence; prompt is the prompt word, key is the keyword, h(.) is the information redundancy filtering function, and Qwen-long is the language model. The final generated summary information h s covering the historical period is:

[0024] Preferably, the third agent adopts the following way to comprehensively judge the current scene:

[0025] The abnormality warning and judgment are combined with the summary information and the adjacent image frame information, and the formal representation is:

[0026]

[0027] wherein h s is the long historical information, I i = [I1, I2,..., I6] is the adjacent 6 image frames; MLLM is an abnormal event judgment model based on LLaVA-1.5, the output of which is the triple information containing the place where, the person who, and the reason why.

[0028] Preferably, the MLLM is an abnormal event judgment model based on LLaVA-1.5, and specifically:

[0029] The abnormal event judgment model based on LLaVA-1.5 receives two kinds of input data, including visual input and text input. The visual input is the adjacent 6 image frames, representing a series of image frames sampled from a video or continuous scene; the text input is the summary information.

[0030] The visual input and the text input are respectively subjected to independent encoder for feature extraction; the visual input is firstly sent into a CLIP ViT-L / 224 encoder, and a visual feature vector is generated for each frame image; the visual feature vector sequence extracted from the CLIP is subjected to a multi-layer perception (MLP) layer as an adapter to project the visual features into a new space compatible with the dimension of the text features, and finally the visual feature vector sequence is obtained.

[0031] The abstract information is firstly subjected to a tokenizer to cut the natural language sentence into a series of tokens; then, the tokens are converted into dense numerical vectors through an embedding layer; this process generates two types of feature vectors: identifier feature vectors and text feature vectors; the identifier feature vectors contain position encoding and special separators; and the text feature vectors represent the semantic content of the tokens.

[0032] The abnormal event discrimination model based on LLaVA-1.5 comprises a time sequence position reconstruction module; the time sequence position reconstruction module integrates the feature vectors from different encoders into a unified sequence, specifically, embeds the processed visual feature vector sequence into the middle of the text and identifier feature vector sequence to obtain a fusion sequence; and the fusion sequence is input into Qwen-4B.

[0033] The Qwen-4B performs deep analysis on the input fusion sequence to generate a structured text report; the text report includes Where, Who and Why.

[0034] Beneficial effects:

[0035] 1. The application provides an abnormal behavior real-time warning system based on multi-agent asynchronous collaboration, proposes a multi-agent asynchronous collaboration framework based on Redis communication, and decomposes the abnormal behavior detection task into three tasks of real-time video frame processing, historical information modeling and abnormal event reasoning, which are independently executed by different agents, and efficient parallel and multi-stage asynchronous communication is realized through a cache server, the application realizes significant improvement in real-time performance and system performance through asynchronous and parallel processing of large-scale real-time video streams, avoids the delay and information loss of offline processing, and significantly improves the response speed and efficiency of city monitoring. The system integrates multi-modal information such as image frame sequences and historical event descriptions, realizes dynamic modeling and abnormal warning of continuous behavior and semantic evolution through an abnormal behavior discrimination model, and improves the understanding ability of dynamic abnormal time and the explanatory ability of abnormal behavior.

[0036] 2、The application realizes dynamic capture and accurate early warning of the behavior evolution process through multi-modal fusion, long-time sequence history modeling and structured input design, rather than only making judgments based on single-frame static features.

[0037] 3、The application decouples abnormality discrimination output, contains event details (such as participants, places, suspected reasons, timestamps, etc.), facilitates subsequent tracing and law enforcement decision-making, and reduces the application risk brought by the "black box" model. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 Flowchart of abnormal behavior real-time early warning system based on multi-agent asynchronous collaboration

[0039] Figure 2 Multi-stage data processing schematic diagram of abnormal behavior real-time early warning system

[0040] Figure 3 Multi-modal time sequence reasoning model structure diagram DETAILED DESCRIPTION

[0041] The application will be described in detail below with reference to the accompanying drawings and examples.

[0042] The application proposes an abnormal behavior real-time early warning system based on multi-agent asynchronous collaboration, which takes real-time video stream as the research object, constructs a multi-agent asynchronous communication framework, and based on a multi-modal visual language model architecture, fuses long-time sequence historical prior information, realizes real-time discrimination and prediction of abnormal behavior events.

[0043] Multi-agent asynchronous collaboration framework based on redis communication

[0044] To solve the problem that single-agent architecture cannot simultaneously meet the task requirements of real-time, time sequence modeling and multi-task collaborative processing, the application is based on a multi-agent asynchronous collaboration framework based on redis communication. The framework realizes efficient parallel and real-time semantic reasoning of multi-stage processing through task decoupling and information sharing, thereby significantly enhancing the comprehensive perception and response ability of the system to dynamic events. Real-time detection and semantic reasoning of abnormal behavior events in image sequences are realized, as shown in Figure 1 .

[0045] The multi-agent asynchronous communication framework specifically includes three agents for processing different tasks, and three Redis cache servers for supporting multi-stage asynchronous communication and information sharing. The three agents are the first to third agents, and the three cache servers are the first to third cache servers.

[0046] Firstly, the first cache server is used for receiving and distributing real-time frame image data, and pushing to the first intelligent agent and the third intelligent agent in real time. The first intelligent agent is responsible for processing the received image frames, extracting semantic features and generating frame-by-frame text descriptions, and then publishing the generated description information and the corresponding timestamp through the second cache server. The second intelligent agent acts as a relay processing module, subscribes to frame-level image descriptions from the second cache server, and after collecting a certain amount of data, integrates its time sequence structure and semantic content to construct summary information covering historical periods. The summary not only contains the semantic features of key image events, but also reflects the evolution trend of events in time. The summary information and the corresponding historical period timestamp are published to the third cache server. The third intelligent agent subscribes to the summary content published by the third cache server, and at the same time acquires the current 6 frames of images, comprehensively judges the current scene, and identifies whether there is an abnormal event or whether there is a warning signal of a possible future abnormality. If an abnormality is detected, the system will output a structured abnormality judgment result containing the event participants, abnormal location, suspected cause and corresponding timestamp.

[0047] The data set of the application collects video data related to abnormal behavior from the current largest weakly supervised anomaly detection (VAD) data set UCF-Crime and the normal video clip data set YouTubeClips. The data set generation process is as shown in Figure 2 . First, the original video is manually screened to remove samples with poor quality or content that does not meet the requirements. Then, after manual interception and screening, abnormal event clips covering six types of abnormal behaviors, such as shoplifting, theft, robbery, and theft, are obtained. Finally, high-quality normal video clips are selected from YouTubeClips to facilitate model comparison learning.

[0048] The first intelligent agent extracts semantic features and generates single-frame descriptions in the following way: During the abnormal event detection and analysis process, accurately generating semantic descriptions for each frame helps to improve the subsequent abnormality. To this end, the application uses the BLIP model to perform semantic annotation on the video frame by frame. For real-time image stream I1, I2, …, I n , let the model mapping function be f(.), then the finally generated frame-level description is:

[0049] g i ,t i =f(I i )

[0050] Where i is the index of the real-time image frame, g i is the generated image description, and t i is the timestamp of the image frame.

[0051] The second intelligent agent constructs summary information covering historical periods in the following way:

[0052] Based on the image-based single-frame description, the application designs a method based on Qwen-long prompt word engineering, keyword task information extraction and information redundancy filtering to summarize and screen the formed historical frame buffer sequence. Wherein, N g is the maximum length of the single-frame description buffer sequence. Let the prompt word be prompt, the keyword be key, the information redundancy filtering function be h(.), and the Qwen-long language model be Then the finally generated historical summary summary h s is:

[0053]

[0054] The third intelligent agent comprehensively discriminates the current scene in the following way:

[0055] In order to fully extract the semantic information in the event segment, the application combines long historical information and adjacent image frame information to perform abnormality early warning and discrimination, which can be formally expressed as:

[0056]

[0057] Wherein, h s is the long historical information, I i =[I1, I2,...,I6] is the adjacent image frame. MLLM is an abnormal behavior discrimination model based on LLaVA-1.5, The output is a triple information containing a place (where), a person (who) and a reason (why).

[0058] Abnormal event discrimination model based on LLaVA-1.5

[0059] In view of the problems that the context window of the LLaVA-1.5 model is too small, long-time sequence information cannot be understood, and adjacent functional module boundaries cannot be distinguished, the application develops an abnormal behavior discrimination model based on the LLaVA-1.5 model, as shown in Figure 3

[0060] Firstly, the model receives two kinds of input data. Among them, the visual input is shown in the upper left corner of the figure, and the input is the current 6 frames of images, which represent a series of image frames sampled from a video or a continuous scene. This constitutes the visual basis for the model to analyze dynamic events. The text input is shown in the upper right corner of Figure 3 , and the input is the historical summary summary. This part of the text provides the context information of the previous event, which provides background knowledge for the reasoning of the model.

[0061] ​Secondly, to enable the model to understand data of different modalities, visual and textual inputs are separately passed through independent encoders for feature extraction. Visual input (6 frames of images) is first sent into a CLIP ViT-L / 224 encoder and generates a high-dimensional "visual feature vector" for each frame of image. The sequence of visual feature vectors extracted from CLIP is passed through a multi-layer perceptron (MLP). This MLP layer serves as an adapter to project the visual features into a new space compatible with the dimension of textual features, resulting in a sequence of visual feature vectors available for subsequent modules.

[0062] The historical summary text is first passed through a tokenizer that splits natural language sentences into a series of tokens. Subsequently, these tokens are converted into dense numerical vectors through an embedding layer. This process generates two types of feature vectors: identifier feature vectors (containing position encoding, special separators, etc. meta-information) and text feature vectors (representing the semantic content of tokens).

[0063] Thirdly, to enable the language model to understand both visual events and textual context, the model employs a temporal position reconstruction module. This module integrates feature vectors from different encoders into a unified sequence. Specifically, it embeds the processed visual feature vector sequence into the middle of the text and identifier feature vector sequence. This interleaved fusion strategy enables the LLM to seamlessly process and correlate visual and textual information in the same context, effectively combining visual changes over time with semantic background.

[0064] Finally, the fused sequence is subjected to deep analysis using the Qwen-4B's powerful context understanding, logical reasoning, and text generation capabilities, resulting in a structured text report. The output text takes the form of "Where: In an electronics store; Who: A person wearing a black jacket; Why: Detailed reasoning of the person's behavior, pointing out suspicious actions such as "frequently looking around", "evaluating the environment and potential surveillance", "holding a bag that can be used to hide stolen items", etc., and ultimately concluding that "there is abnormal behavior".

[0065] In conclusion, the above is only a preferred embodiment of the present application, and is not intended to limit the scope of protection of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A real-time warning system for abnormal behavior based on asynchronous collaboration of multiple intelligent agents, characterized by: The steps include: Taking real-time video streaming as the research object, a multi-agent asynchronous communication framework is constructed; Based on the multimodal visual language model architecture, long-term historical prior information is integrated to achieve real-time identification and prediction of abnormal behaviors.

2. The abnormal behavior real-time warning system according to claim 1, characterized in that: The real-time video stream is used as the research object to build a multi-agent asynchronous communication framework, specifically: The abnormal behavior detection task is broken down into three major tasks: real-time video frame processing, historical information modeling, and abnormal event reasoning. These tasks are independently executed by different intelligent agents, and efficient parallel and multi-stage asynchronous communication is achieved through the cache server.

3. The real-time warning system for abnormal behavior according to claim 2, characterized in that: The multi-agent asynchronous communication framework specifically includes three agents for processing different tasks and three Redis cache servers to support multi-stage asynchronous communication and information sharing; The three intelligent agents are the first to third intelligent agents, and the three cache servers are the first to third cache servers. First, the first cache server is used to receive and distribute real-time frame image data and push it to the first agent and the third agent in real time; The first agent is responsible for processing the received image frames, extracting semantic features and generating text descriptions frame by frame, and then publishing the generated description information and corresponding timestamps through the second cache server; The second agent, acting as a relay processing module, subscribes to frame-level image descriptions from the second cache server. After collecting a certain amount of data, it integrates its temporal structure and semantic content to construct summary information covering the historical period. This summary information not only contains the semantic features of key image events but also reflects the temporal evolution trend of the events. This summary information and the corresponding historical period timestamp are then published to the third cache server. The third agent subscribes to the summary information published by the third cache server, and at the same time obtains the most recent 6 frames of images to make a comprehensive judgment on the current scene to identify whether there is an abnormal event or whether there is a warning signal that an abnormality may occur in the future; if an abnormality is detected, the system will output a structured abnormality judgment result including the event participants, the abnormal location, the presumed cause and the corresponding timestamp.

4. The real-time warning system for abnormal behavior according to claim 3, characterized in that: The first agent extracts semantic features and generates single-frame descriptions frame by frame using the following method: Use the BLIP model to semantically annotate the image frames. For real-time image frames I1, I2, ..., I n , let the model mapping function be f(.), then the final generated single frame description is: g i ,t i =f(I i ) Among them, i is the real-time image frame index, g i is the generated image description, t i The timestamp of the image frame.

5. The real-time warning system for abnormal behavior according to claim 3, characterized in that: The second agent constructs summary information covering the historical period in the following manner: The text sequence set of single-frame descriptions generated by the first agent is Among them, N g is the maximum length of the cache sequence for a single frame description; the prompt word is prompt, the keyword is key, the information redundancy filter function is h(.), and the Qwen-long language model is The final generated summary information covering the historical period h s for:

6. The real-time warning system for abnormal behavior according to claim 3, characterized in that: The third agent uses the following method to comprehensively judge the current scene: Combine summary information with adjacent image frame information to perform abnormal warning and identification, which can be formally expressed as: Among them, h s For long history information, I i =[I1,I2,...,I6] are the adjacent 6 image frames; MLLM is the abnormal event discrimination model based on LLaVA-1.5, The output is a triplet of information including the location where, the person who, and the reason why.

7. The real-time warning system for abnormal behavior according to claim 6, characterized in that: The MLLM is an abnormal event discrimination model based on LLaVA-1.5, specifically: The abnormal event discrimination model based on LLaVA-1.5 receives two modal input data, including visual input and text input, wherein the visual input is 6 adjacent image frames, representing a series of image frames sampled from a video or a continuous scene; and the text input is the summary information; The visual input and text input are each passed through independent encoders for feature extraction. The visual input is first fed into the CLIPViT-L / 224 encoder, which generates a visual feature vector for each frame. The sequence of visual feature vectors extracted from CLIP is then passed through a multi-layer perceptron (MLP) layer, which acts as an adapter to project the visual features into a new space compatible with the text feature dimension, ultimately resulting in a sequence of visual feature vectors. The summary information first passes through a tokenizer, which breaks the natural language sentence into a series of tokens. Subsequently, the tokens are converted into dense numerical vectors through an embedding layer. This process generates two types of feature vectors: identifier feature vectors and text feature vectors. The identifier feature vector contains positional codes and special delimiters, while the text feature vector represents the semantic content of the token. The abnormal event discrimination model based on LLaVA-1.5 includes a temporal position reconstruction module; the temporal position reconstruction module integrates feature vectors from different encoders into a unified sequence. Specifically, the processed visual feature vector sequence is embedded in the middle of the text and identifier feature vector sequences to obtain a fused sequence; the fused sequence is input into Qwen-4B; Qwen-4B performs an in-depth analysis of the input fusion sequence and generates a structured text report; the text report includes the location, the person, and the reason analysis.

Images