Abnormal behavior multi-modal recognition method and system based on large model agent

Abstract

The application discloses an abnormal behavior multi-modal recognition method and system based on a large model agent, relates to the field of abnormal behavior recognition, and comprises the following steps: constructing a large model agent architecture; acquiring multi-modal data flow of a target scene, generating a structured behavior semantic sequence; generating an expected result semantic description, calculating a semantic deviation vector between the expected result semantic description and an actual result semantic description as a debate anchor point; constructing an adversarial debate network of the pro and the con, generating a behavior normal hypothesis and a behavior abnormal hypothesis; according to the pro argument sequence and the con argument sequence, evaluating logical consistency, evidence sufficiency and scene rule compliance degree, and ruling whether the current behavior is abnormal; outputting a ruling result, and attaching a complete evidence chain formed by the winning party in the debate process as an abnormality judgment basis. Through multi-agent debate and counterfactual causal reasoning, accurate multi-modal abnormal behavior recognition is realized, and the method has the advantages of logical transparency, explainability, iteration and strong anti-interference.

Description

Technical Field

[0001] This invention relates to the field of abnormal behavior recognition, specifically to a method and system for multimodal recognition of abnormal behavior based on a large model intelligent agent. Background Technology

[0002] With the rapid development of security monitoring, intelligent transportation, and industrial inspection, abnormal behavior recognition technology has become an important means to ensure public safety and production safety. Traditional abnormal behavior recognition methods are mainly divided into two categories: one is supervised learning methods based on manual feature extraction and classifiers. These methods rely on a large number of labeled abnormal samples for training. However, abnormal behaviors are sporadic and diverse, making it difficult to obtain sufficient labeled data, and they cannot identify unknown abnormal types that do not appear in the training set. The other category is unsupervised learning methods based on reconstruction error or prediction error. These methods determine abnormalities by constructing a baseline of normal behavior and calculating the numerical deviation between the current behavior and the baseline. They do not require labeled abnormal samples and have strong generalization ability. However, all of the above methods share a common limitation: they treat abnormal behavior recognition as a purely numerical calculation problem. They construct a feature space by extracting low-level features (such as optical flow, trajectory, skeletal key points, etc.) and calculate distance or error within this space. This approach loses the semantic information of the behavior and cannot understand the context and intent of the behavior. For example, a person running quickly in a hospital corridor could be seeking emergency care (normal) or fighting (abnormal), which cannot be distinguished by motion features alone; a person entering a laboratory late at night could be working overtime (normal) or stealing (abnormal), which cannot be determined by trajectory features alone. This "semantic gap" leads to high false positive and false negative rates for traditional methods in complex dynamic scenarios.

[0003] In recent years, large language models and multimodal large models have demonstrated powerful semantic understanding and reasoning capabilities, providing new technical paths for abnormal behavior recognition. How to utilize large models to understand behavioral semantics, integrate multimodal information, perform causal reasoning and adversarial verification, and construct interpretable abnormal behavior recognition methods is an important direction for current technological development.

[0004] Existing abnormal behavior identification methods treat the identification process as a numerical computation problem within a feature space. By extracting low-level features to construct a normal baseline and calculating the deviation, this method loses the semantic information and contextual association of the behavior, fails to understand the behavioral intent, and struggles to distinguish between behaviors with the same semantics but different scenarios. This results in high false positive and false negative rates in complex dynamic scenarios, and the judgment results lack interpretability. Summary of the Invention

[0005] To address the aforementioned technical problems, this paper provides a method and system for multimodal recognition of abnormal behavior based on a large model intelligent agent. This technical solution solves the problem that existing abnormal behavior recognition methods mentioned in the background treat the recognition process as numerical calculation within a feature space, losing the semantic information and contextual association of the behavior, failing to understand the behavioral intent, and having difficulty distinguishing between behaviors with the same semantics but different scenarios.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for multimodal recognition of abnormal behavior based on large model intelligent agents includes:

[0008] A large-scale model intelligent agent architecture is constructed, which consists of a perception agent, a counterfactual reasoning agent, a positive agent, a negative agent, and a referee agent.

[0009] Acquire multimodal data streams of the target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences into a perceptual intelligent agent;

[0010] Based on the observation data of the first half of the structured behavioral semantic sequence, generate a semantic description of the expected result, calculate the semantic deviation vector between the expected result and the actual result semantic description as the debate anchor point, and encapsulate the functions of generating the semantic description of the expected result and calculating the semantic deviation vector into a counterfactual reasoning agent.

[0011] Based on the debate anchor points, an adversarial debate network is constructed between the affirmative and negative sides to generate normal behavior assumptions and abnormal behavior assumptions, conduct multiple rounds of debate, and encapsulate the function of generating normal behavior assumptions as the affirmative intelligent agent and the function of generating abnormal behavior assumptions as the negative intelligent agent.

[0012] Based on the sequence of arguments for and against generated during the debate, the system assesses logical consistency, sufficiency of evidence, and compliance with scenario rules to determine whether the current behavior is abnormal. The function of determining whether the current behavior is abnormal is then encapsulated as a referee agent.

[0013] The ruling will be output along with the complete chain of evidence formed by the winning party during the debate, which will serve as the basis for the anomaly determination.

[0014] Preferably, the step of acquiring the multimodal data stream of the target scene and generating a structured behavioral semantic sequence includes:

[0015] Acquire multimodal data streams of the target scene, including but not limited to: visual data, audio data, and text data;

[0016] The continuous frame sequence in the multimodal data stream is divided into several time windows of equal length according to the time order, and the multimodal semantic feature vector of each window is extracted;

[0017] Calculate the semantic aberration degree between the multimodal semantic feature vectors of adjacent windows, and mark the time points where the semantic aberration degree exceeds the dynamic threshold as semantic event boundaries;

[0018] Cross-modal semantic anchor matching is performed based on semantic event boundaries, and event pairs that meet the mutual information threshold and whose time difference is within a preset window are marked as cross-modal semantic anchors;

[0019] Based on cross-modal semantic anchor points, non-linear alignment is performed on multimodal data streams to eliminate time offset between multiple source sensors;

[0020] The aligned multimodal data stream is segmented into atomic behavior fragments based on semantic event boundaries;

[0021] Each atomic behavior fragment is subjected to structured semantic annotation, and behavior tuple fields are extracted to form a structured behavior semantic sequence.

[0022] Preferably, the step of generating a semantic description of the expected result based on the first half of the observation data in the structured behavioral semantic sequence, and calculating the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point includes:

[0023] Extract the first half of the observation data of the current behavior from the structured behavioral semantic sequence. The first half of the observation data is a subset of behavioral tuples from the start timestamp of the behavior to the current timestamp.

[0024] Combine the contextual information of the scene to perform causal inference, and generate a semantic description of the expected result that the behavior should produce in a normal scene according to the preset text template;

[0025] The scene context information includes time attributes, spatial attributes, character roles, device status, and historical normal mode;

[0026] Extract the sequence of behavioral tuples corresponding to the complete observation data of the current behavior from the structured behavioral semantic sequence, and convert it into a natural language description according to the preset text template as the semantic description of the actual result;

[0027] The expected result semantic description is encoded to generate the first semantic embedding vector, and similarity matching is performed with the semantic embedding vector in the normal behavior template library. Several templates with the highest similarity are selected, and the semantic feature vectors in the selected templates are fused according to similarity weight to generate the first semantic feature vector.

[0028] The semantic description of the actual result is encoded to generate a second semantic embedding vector, and similarity matching is performed with the semantic embedding vector in the normal behavior template library. Several templates with the highest similarity are selected, and the semantic feature vectors in the selected templates are fused according to similarity weight to generate a second semantic feature vector.

[0029] The difference between the first semantic feature vector and the second semantic feature vector is calculated to obtain the semantic deviation vector, and this semantic deviation vector is used as the debate anchor point.

[0030] Preferably, the step of constructing an adversarial debate network between the affirmative and negative sides based on debate anchor points, generating normal behavior assumptions and abnormal behavior assumptions, and conducting multiple rounds of debate includes:

[0031] Using debate anchors and scene context information as input, the system generates a normal behavior hypothesis based on the deviation value in the debate anchors and the scene context information, and retrieves multimodal evidence supporting the hypothesis from the structured behavioral semantic sequence to generate the affirmative argument.

[0032] Using debate anchors and scene context information as input, the system generates behavioral anomaly hypotheses based on deviation values ​​in the debate anchors and scene context information, and retrieves multimodal evidence supporting the hypotheses from structured behavioral semantic sequences to generate opposing arguments.

[0033] Multiple rounds of structured debate were conducted according to the pre-set debate agreement. Each round of speeches included the statement of arguments and the citation of evidence, while the rebuttal speeches included the statement of rebuttal and the citation of counter-evidence.

[0034] The debate ends prematurely if neither the affirmative nor the negative side can present new evidence or arguments in two consecutive rounds.

[0035] Preferably, the step of assessing logical consistency, sufficiency of evidence, and compliance with scenario rules based on the sequence of affirmative and negative arguments generated during the debate, and determining whether the current behavior is abnormal, includes:

[0036] Obtain the sequences of affirmative and negative arguments generated during the debate. Each argument sequence contains several rounds of arguments and corresponding evidence citations.

[0037] Logical consistency assessment is performed on the sequences of arguments for both sides, which is achieved by detecting whether there are self-contradictory statements in the sequences of arguments.

[0038] The sufficiency of evidence for the affirmative and negative argument sequences is assessed separately. The sufficiency of evidence assessment is achieved by weighting the number of multimodal evidence cited, the relevance score between the evidence and the argument, and the time coverage of the evidence.

[0039] The scenario rule compliance degree is evaluated for both the affirmative and negative argument sequences. The scenario rule compliance degree evaluation is achieved by semantically matching the arguments with scenario rules stored in an external knowledge base.

[0040] The scores for logical consistency, sufficiency of evidence, and compliance with scenario rules are weighted and summed to generate the overall scores for the affirmative and negative sides, respectively.

[0041] Compare the overall scores of the affirmative and negative sides. If the negative side's overall score is higher than the affirmative side's overall score, the current behavior is ruled as abnormal; otherwise, it is ruled as normal.

[0042] Furthermore, this solution proposes an abnormal behavior multimodal recognition system based on a large model intelligent agent, used to implement the above-mentioned abnormal behavior multimodal recognition method based on a large model intelligent agent, including:

[0043] The intelligent agent architecture module is used to construct a large model intelligent agent architecture, which consists of a perception intelligent agent, a counterfactual reasoning intelligent agent, a positive intelligent agent, a negative intelligent agent, and a referee intelligent agent.

[0044] The intelligent agent encapsulation module is used to acquire multimodal data streams of the target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences as a perceptual intelligent agent; based on the first half of the observed data in the structured behavioral semantic sequences, it generates a semantic description of the expected result, calculates the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point, and encapsulates the functions of generating the expected result semantic description and calculating the semantic deviation vector as a counterfactual reasoning intelligent agent; based on the debate anchor point, it constructs an adversarial debate network between the affirmative and negative sides, generates normal behavior assumptions and abnormal behavior assumptions, conducts multiple rounds of debate, and encapsulates the function of generating normal behavior assumptions as an affirmative intelligent agent and the function of generating abnormal behavior assumptions as a negative intelligent agent; based on the affirmative and negative argument sequences generated during the debate, it evaluates the logical consistency, sufficiency of evidence, and compliance with scene rules, and adjudicates whether the current behavior is abnormal, and encapsulates the function of adjudicating whether the current behavior is abnormal as a referee intelligent agent;

[0045] An anomaly determination module is used to output the ruling result, and to attach the complete chain of evidence formed by the winning party during the debate as the basis for anomaly determination.

[0046] Preferably, the intelligent agent encapsulation module includes:

[0047] A perceptual agent encapsulation unit is used to acquire multimodal data streams of a target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences into a perceptual agent.

[0048] The reasoning agent encapsulation unit is used to generate a semantic description of the expected result based on the first half of the observation data in the structured behavioral semantic sequence, calculate the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point, and encapsulate the functions of generating the semantic description of the expected result and calculating the semantic deviation vector into a counterfactual reasoning agent.

[0049] The positive and negative intelligent agent encapsulation unit is used to construct an adversarial debate network between the positive and negative sides based on the debate anchor point, generate normal behavior hypothesis and abnormal behavior hypothesis, conduct multiple rounds of debate, and encapsulate the function of generating normal behavior hypothesis as the positive intelligent agent and the function of generating abnormal behavior hypothesis as the negative intelligent agent.

[0050] The judge intelligent encapsulation unit is used to evaluate the logical consistency, sufficiency of evidence, and compliance with scenario rules based on the sequence of affirmative and negative arguments generated during the debate, to determine whether the current behavior is abnormal, and to encapsulate the function of determining whether the current behavior is abnormal into a judge intelligent agent.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] This invention provides a method and system for multimodal recognition of abnormal behavior based on a large-scale model intelligent agent. This solution constructs a five-in-one large-scale model intelligent agent architecture encompassing perception, counterfactual reasoning, affirmative debate, negative debate, and adjudication. Counterfactual reasoning is achieved through a structural causal model, and semantic deviation vectors serve as debate anchors to drive multiple rounds of adversarial arguments between the affirmative and negative agents. The adjudicating agent makes a quantitative ruling based on logical consistency, sufficiency of evidence, and compliance with scenario rules. This not only effectively improves the accuracy and robustness of abnormal behavior recognition in complex multimodal environments but also ensures interpretable judgment processes, traceable identification basis, and retainable evidence chains. Furthermore, it leverages an experience base to enable the reuse of historical cases and continuous iteration of agent capabilities. This effectively solves the problems of black-box reasoning, poor interpretability, susceptibility to interference, weak generalization, and lack of logical verification inherent in traditional methods, resulting in higher reliability, practicality, and scenario adaptability. Attached Figure Description

[0053] Figure 1 This is a flowchart of the abnormal behavior multimodal recognition method based on a large model intelligent agent according to the present invention;

[0054] Figure 2 To obtain the multimodal data stream of the target scene according to the present invention, a structured behavioral semantic sequence flowchart is generated;

[0055] Figure 3 The flowchart for calculating the semantic deviation vector between the expected result semantic description and the actual result semantic description is used as a debate anchor point for the semantic description of the present invention.

[0056] Figure 4 To construct the adversarial debate network between the affirmative and negative sides in this invention, a flowchart is generated to illustrate normal behavior assumptions and abnormal behavior assumptions, and to conduct multiple rounds of debate.

[0057] Figure 5 This is a flowchart illustrating the process of determining whether the current behavior is abnormal according to the present invention. Detailed Implementation

[0058] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0059] Reference Figure 1 As shown, a multimodal recognition method for abnormal behavior based on a large model intelligent agent includes:

[0060] A large-scale model intelligent agent architecture is constructed, which consists of a perception agent, a counterfactual reasoning agent, a positive agent, a negative agent, and a referee agent.

[0061] Acquire multimodal data streams of the target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences into a perceptual intelligent agent;

[0062] Based on the observation data of the first half of the structured behavioral semantic sequence, generate a semantic description of the expected result, calculate the semantic deviation vector between the expected result and the actual result semantic description as the debate anchor point, and encapsulate the functions of generating the semantic description of the expected result and calculating the semantic deviation vector into a counterfactual reasoning agent.

[0063] Based on the debate anchor points, an adversarial debate network is constructed between the affirmative and negative sides to generate normal behavior assumptions and abnormal behavior assumptions, conduct multiple rounds of debate, and encapsulate the function of generating normal behavior assumptions as the affirmative intelligent agent and the function of generating abnormal behavior assumptions as the negative intelligent agent.

[0064] Based on the sequence of arguments for and against generated during the debate, the system assesses logical consistency, sufficiency of evidence, and compliance with scenario rules to determine whether the current behavior is abnormal. The function of determining whether the current behavior is abnormal is then encapsulated as a referee agent.

[0065] The ruling will be output along with the complete chain of evidence formed by the winning party during the debate, which will serve as the basis for the anomaly determination.

[0066] This solution achieves structured, interpretable, and verifiable identification of multimodal abnormal behavior through the collaborative division of labor among five intelligent agents: the perception agent transforms the raw multimodal data into a standardized structured sequence of behavioral semantics; the counterfactual reasoning agent performs causal inference based on the first half of the behavioral data to obtain the expected result, and calculates the semantic deviation vector with the actual result to determine the debate anchor point; the affirmative and negative agents construct hypothetical arguments of normal and abnormal behavior around this anchor point and engage in multiple rounds of adversarial debate; and the adjudicating agent conducts a comprehensive quantitative evaluation of the logical consistency, evidentiary sufficiency, and compliance with scenario rules of both sides' arguments and makes a final ruling. The entire process realizes a closed-loop chain from multimodal perception, causal reasoning, adversarial verification to interpretable judgment, significantly improving the accuracy, reliability, and traceability of abnormal behavior identification.

[0067] The large-model intelligent agent architecture includes:

[0068] The large model perceptual agent is used to transform multimodal data streams into structured behavioral semantic sequences.

[0069] The large-scale counterfactual reasoning agent is used to generate a semantic description of the expected result based on the observation data of the first half of the current behavior and the context information of the scene, and to calculate the semantic deviation vector as the debate anchor point.

[0070] The affirmative and negative intelligent agents are used to generate normal behavior assumptions and abnormal behavior assumptions based on debate anchor points, structured behavioral semantic sequences and scene context information, and conduct multiple rounds of debate.

[0071] The adjudicator is used to determine whether the current behavior is abnormal based on the sequence of arguments for and against generated during the debate, and outputs the adjudication result and the winning party's evidence chain.

[0072] It can be explained that the large-scale model intelligent agent architecture constructed in this solution is the core supporting framework of the entire abnormal behavior recognition method. This architecture divides the recognition process into four stages: perception, counterfactual reasoning, adversarial debate, and adjudication. Each stage is undertaken independently by the corresponding intelligent agent and connected through standardized data interfaces. By decomposing complex tasks into multiple sub-tasks and having them processed by specialized intelligent agents, functional decoupling and modular collaboration are achieved, which facilitates the independent optimization and replacement of each intelligent agent. At the same time, it provides a clear execution subject and data flow path for the subsequent steps of structured semantic sequence generation, debate anchor construction, multi-round debate, and adjudication.

[0073] Reference Figure 2 As shown, the step of acquiring the multimodal data stream of the target scene and generating a structured behavioral semantic sequence includes:

[0074] Acquire multimodal data streams of the target scene, including but not limited to: visual data, audio data, and text data;

[0075] The continuous frame sequence in the multimodal data stream is divided into several time windows of equal length according to the time order, and the multimodal semantic feature vector of each window is extracted;

[0076] Calculate the semantic aberration degree between the multimodal semantic feature vectors of adjacent windows, and mark the time points where the semantic aberration degree exceeds the dynamic threshold as semantic event boundaries;

[0077] Cross-modal semantic anchor matching is performed based on semantic event boundaries, and event pairs that meet the mutual information threshold and whose time difference is within a preset window are marked as cross-modal semantic anchors;

[0078] Based on cross-modal semantic anchor points, non-linear alignment is performed on multimodal data streams to eliminate time offset between multiple source sensors;

[0079] The aligned multimodal data stream is segmented into atomic behavior fragments based on semantic event boundaries;

[0080] Each atomic behavior fragment is subjected to structured semantic annotation, and behavior tuple fields are extracted to form a structured behavior semantic sequence.

[0081] This step can be explained as follows: the original multimodal signal is transformed into a structured behavioral semantic sequence, and this transformation function is encapsulated into a perceptual agent. Specifically, the multimodal semantic feature vector of each window is extracted as follows: a convolutional neural network is used to extract the visual feature vector of the video frame within the window, the Mel frequency cepstral coefficient is used to extract the acoustic feature vector of the audio frame, and a pre-trained language model (such as BERT) is used to extract the semantic feature vector of the text data. The three feature vectors are then concatenated by channel to form the multimodal semantic feature vector of the window.

[0082] The semantic abrupt change degree is used to quantify the degree of semantic change in behavioral content within adjacent time windows. Its physical meaning is: when the behavior within a window changes from one state to another, the semantic abrupt change degree is larger; when the behavior within a window remains continuous and consistent, the semantic abrupt change degree is smaller. The semantic abrupt change degree is calculated by weighted summation of the cosine similarity and KL divergence of the feature vectors of adjacent windows. The cosine similarity reflects the difference in feature direction between windows, and the KL divergence reflects the difference in probability distribution between windows. By calculating the semantic abrupt change degree between adjacent windows, when the semantic abrupt change degree exceeds a dynamic threshold, the time point is marked as the semantic event boundary.

[0083] The dynamic threshold is not a fixed value, but is adaptively updated based on the statistical characteristics of the historical window abruptness. Specifically, the initial value of the dynamic threshold is determined by analyzing a calibration video known to have no abnormal behavior: extract the semantic abruptness of all adjacent windows in the calibration video, calculate their mean and standard deviation, and set the initial threshold as the mean plus three times the standard deviation, i.e., using the Laida criterion. In the actual detection process, the dynamic threshold is updated based on the moving average of the abruptness of the most recent multiple windows. The update adopts an exponentially weighted moving average method, in which the historical threshold and the current statistical features are fused according to a preset weight ratio, so that it can adapt to the normal behavior fluctuation range under different scenarios and avoid threshold failure due to scene changes.

[0084] The mutual information is used to measure the semantic association strength between video action categories and audio event categories. It is calculated as follows: for all possible combinations of video actions and audio events, the logarithm of the ratio of the product of their joint probability distribution and marginal probability distribution is calculated, and then the results are weighted and summed according to the joint probability. The mutual information ranges from zero to positive infinity; a larger value indicates a stronger association between the video action and the audio event. The mutual information threshold is determined through offline calibration: multimodal samples containing known paired events are collected, including positive and negative samples; the mutual information of all samples is calculated; and the threshold that results in the highest classification accuracy is selected as the preset threshold.

[0085] The preset time window is used to determine whether video actions and audio events belong to the same semantic event. Its value range is determined based on the inherent physical delay between human visual perception of actions and the corresponding sound generation. The lower limit of the window should be greater than the time error caused by sensor jitter, and the upper limit should be less than the minimum time interval between two adjacent different semantic events, so as to ensure that cross-modal signals within the same semantic event are correctly associated, while avoiding incorrect matching between different events.

[0086] Atomic behavior segments must simultaneously meet the following three constraints: First, the segment does not contain semantic mutation points. This condition ensures that the behavior within each segment is semantically continuous and there is no behavior state switching. Second, the segment length is within a preset frame number range. The minimum frame number is the number of frames corresponding to the shortest duration required to identify a complete atomic action, and the maximum frame number is the number of frames corresponding to the maximum reasonable duration of most atomic behaviors. Segments exceeding this range may contain multiple semantic units or have abnormal pauses. Third, the segment contains at least one complete action and object interaction. The complete action and object interaction is determined by detecting the semantic coherence between the action start frame and the action end frame. Specifically, when there is a clear action target object within the segment and the action trajectory presents a complete pattern of "approaching, contacting, leaving", it is determined to contain a complete action and object interaction.

[0087] The prompt template is used to drive the pre-trained large model to perform a structured description of atomic behavior segments. Its design principles are: to specify the input format, including video frame sequences, audio waveforms, and scene metadata; to specify the output format as JSON; and to specify the fields to be extracted, including subject, action, object, timestamp, and confidence level. The large-model perceptual agent calls the pre-trained large model according to the following prompt template: Analyze the following multimodal segments: video frame sequence; audio waveform; scene metadata including time, location, and characters. Output the analysis results in JSON format, including the following fields: subject (identifier of the person performing the action); action (action category); object (object affected by the action; fill in if no object exists); start timestamp (time when the action begins); end timestamp (time when the action ends); confidence score; output only JSON, do not output other content. The large-model perceptual agent extracts the JSON fields from the results returned by the pre-trained large model and stores them together with the corresponding multimodal feature vectors to form a structured behavioral semantic sequence. Through the above parameter definition and determination methods, this step realizes the complete conversion from the original multimodal signal to the structured behavioral semantic sequence, providing a standardized input format for subsequent counterfactual reasoning and adversarial debate.

[0088] Reference Figure 3 As shown, the process of generating a semantic description of the expected result and calculating the semantic deviation vector between it and the actual result semantic description as a debate anchor point includes:

[0089] Extract the first half of the observation data of the current behavior from the structured behavioral semantic sequence. The first half of the observation data is a subset of behavioral tuples from the start timestamp of the behavior to the current timestamp.

[0090] Combine the contextual information of the scene to perform causal inference, and generate a semantic description of the expected result that the behavior should produce in a normal scene according to the preset text template;

[0091] The scene context information includes time attributes, spatial attributes, character roles, device status, and historical normal mode;

[0092] Extract the sequence of behavioral tuples corresponding to the complete observation data of the current behavior from the structured behavioral semantic sequence, and convert it into a natural language description according to the preset text template as the semantic description of the actual result;

[0093] The expected result semantic description is encoded to generate the first semantic embedding vector, and similarity matching is performed with the semantic embedding vector in the normal behavior template library. Several templates with the highest similarity are selected, and the semantic feature vectors in the selected templates are fused according to similarity weight to generate the first semantic feature vector.

[0094] The semantic description of the actual result is encoded to generate a second semantic embedding vector, and similarity matching is performed with the semantic embedding vector in the normal behavior template library. Several templates with the highest similarity are selected, and the semantic feature vectors in the selected templates are fused according to similarity weight to generate a second semantic feature vector.

[0095] The difference between the first semantic feature vector and the second semantic feature vector is calculated to obtain the semantic deviation vector, and this semantic deviation vector is used as the debate anchor point.

[0096] Explained in this step, the antifactual reasoning agent generates a semantic description of the expected result through a large model, constructs a debate anchor point for subsequent debates, and encapsulates the functions of generating the semantic description of the expected result and calculating the semantic deviation vector into a counterfactual reasoning agent. The logic of counterfactual reasoning is as follows: assuming that the current behavior is normal behavior, after the first half of the observed behavior data, according to the normal scene rules and behavior logic, the subsequent events should occur. The preset normal behavior template library stores the standard semantic descriptions of various normal behaviors and their corresponding semantic embedding vectors. Since the semantic descriptions of the expected result and the actual result are both natural language text, they cannot be directly matched with the semantic embedding vectors. Therefore, it is necessary to encode them into semantic embedding vectors first, and then perform similarity matching with the semantic embedding vectors in the template library. The semantic deviation vector is obtained by calculating the dimensional difference between the first semantic feature vector and the second semantic feature vector. Each component in this vector represents the degree of difference between the expected result and the actual result in the corresponding semantic dimension, and it is used as a debate anchor point to provide a clear focus for subsequent adversarial debates.

[0097] The semantic embedding vector is encoded as follows: a pre-trained language model, such as BERT or Qwen, is used to encode the semantic description of the expected / actual result, and a fixed-dimensional semantic embedding vector is output. During the encoding process, the model's clstoken output is used as the final embedding vector to ensure the consistency and semantic relevance of the encoding results. The similarity matching uses cosine similarity calculation, that is, the cosine value of the semantic embedding vector and each semantic feature vector in the template library is calculated, with a value range of [-1, 1]. The closer the value is to 1, the more similar the semantics are.

[0098] The normal behavior template library is a pre-built database that stores standard semantic descriptions of various normal behaviors and their corresponding semantic embedding vectors. The template library is constructed as follows: a large amount of multimodal behavior data in normal scenarios is collected, and the data is transformed into structured behavior semantic sequences based on a perceptual agent. Each complete normal behavior sequence is used as a template, and each template contains at least the following fields: behavior type label, structured behavior semantic sequence, semantic embedding vector, and scene context label.

[0099] It should be further explained that the causal inference performed by combining scene context information is based on a structural causal model approach, which specifically includes the following steps:

[0100] A causal graph for the target scenario is pre-constructed. This causal graph is a directed acyclic graph (DAG), where nodes represent behavioral events and state variables, and directed edges represent causal relationships between nodes. Taking an access control scenario as an example, the causal graph includes the following nodes: card swipe action, authorization verification result, door lock state, and access behavior, as well as the following directed edges: card swipe action → authorization verification result, authorization verification result → door lock state, door lock state → access behavior. The causal graph is constructed based on the physical and logical rules of the scenario, obtained through expert knowledge or learned from historical data.

[0101] The behavioral tuples in the first half of the observation data are mapped to the node states in the causal graph. Specifically, the behavioral events that have occurred and the current state variables are extracted from the structured behavioral semantic sequence, and the corresponding nodes are set as observed states in the causal graph.

[0102] Based on the conditional independence relation and structure learning algorithm in the causal graph, the probability distribution of unobserved nodes is inferred given the state of the observed nodes. Specifically, Bayesian inference based on Markov blanket is used to calculate the posterior probability of each unobserved node.

[0103] Generate a semantic description of the expected result based on the reasoning result, convert the inferred subsequent event nodes and their probability distribution into natural language description according to the preset text template, and organize the description text according to the topological order in the causal graph when multiple subsequent events have a temporal relationship.

[0104] By using structural causal models, given the observation data of the first half, it is possible to infer the expected subsequent events that conform to the scene rules and causal logic, thereby generating a semantic description of the expected results.

[0105] Reference Figure 4 As shown, the construction of the adversarial debate network between the affirmative and negative sides, the generation of normal behavior assumptions and abnormal behavior assumptions, and the conduct of multiple rounds of debate include:

[0106] Using debate anchors and scene context information as input, the system generates a normal behavior hypothesis based on the deviation value in the debate anchors and the scene context information, and retrieves multimodal evidence supporting the hypothesis from the structured behavioral semantic sequence to generate the affirmative argument.

[0107] Using debate anchors and scene context information as input, the system generates behavioral anomaly hypotheses based on deviation values ​​in the debate anchors and scene context information, and retrieves multimodal evidence supporting the hypotheses from structured behavioral semantic sequences to generate opposing arguments.

[0108] Multiple rounds of structured debate were conducted according to the pre-set debate agreement. Each round of speeches included the statement of arguments and the citation of evidence, while the rebuttal speeches included the statement of rebuttal and the citation of counter-evidence.

[0109] The debate ends prematurely if neither the affirmative nor the negative side can present new evidence or arguments in two consecutive rounds.

[0110] Explained in this step, the normal behavior hypothesis and the abnormal behavior hypothesis are generated based on the debate anchor point and the context information of the scene. Evidence is retrieved from the structured behavioral semantic sequence. The function of conducting multiple rounds of debate is encapsulated into a positive intelligent agent and a negative intelligent agent. Through encapsulation, the sub-tasks of initiating the debate, refuting, retrieving evidence, and generating arguments are integrated into two independent intelligent agent modules, realizing the decoupling and adversarial collaboration of the functions of the positive and negative sides, while providing a structured debate record for subsequent judges' decisions.

[0111] The specific method for generating the normal behavior hypothesis is as follows: The deviation magnitude is calculated dimension-by-dimensionally for the semantic deviation vector. The deviation magnitude refers to the absolute value of each dimension component in the semantic deviation vector after normalization, used to quantify the degree of difference between the actual result and the expected result in each semantic dimension. The deviation magnitudes of each dimension are sorted in descending order, and the top K dimensions with the largest deviation magnitudes are selected as the main semantic dimensions, where K is the preset number of dimensions. The selected main semantic dimensions are the semantic deviation types with larger deviation degrees. A scene-to-semantic dimension threshold mapping table is pre-constructed. This mapping table is constructed by collecting a large amount of multimodal data on normal behavior in different scenes and converting it into structured behavior. The semantic sequence is matched with the normal behavior template library to calculate the deviation amplitude distribution of each semantic dimension under normal conditions. The mean μ and standard deviation σ of the deviation amplitude of each semantic dimension are statistically analyzed. The normal deviation threshold is set as μ plus k times σ, where k takes 2 or 3, corresponding to the 95% or 99% confidence interval. The allowable fluctuation range is set to zero to the normal deviation threshold. Based on the context information of the current scene, the normal deviation threshold and allowable fluctuation range of each semantic dimension in the current scene are obtained by querying the mapping table. The deviation amplitude is compared with the corresponding normal threshold. If the deviation amplitude is within the allowable fluctuation range, the semantic deviation is determined to conform to the normal behavior pattern, and a normal behavior hypothesis is generated accordingly.

[0112] The method for generating abnormal behavior hypotheses is symmetrical to the above process, but the judgment logic is reversed: if the deviation amplitude exceeds the normal threshold range and the deviation pattern is inconsistent with the historical normal behavior pattern, then the semantic deviation is determined to be inconsistent with the normal behavior pattern, and an abnormal behavior hypothesis is generated accordingly.

[0113] The specific method for retrieving multimodal evidence supporting hypotheses in structured behavioral semantic sequences is as follows: convert the hypothesis content into a semantic query vector, perform cosine similarity calculation with the semantic feature vectors of each atomic behavioral segment in the structured behavioral semantic sequence, and select segments with similarity exceeding a preset value as evidence.

[0114] The debate agreement stipulates the round structure, speaking format, and termination conditions. In the first round, the affirmative side first presents its arguments; the negative side refutes each interpretation in the affirmative side's arguments and presents its own arguments. In subsequent rounds, the affirmative side refutes the negative side's rebuttals, and the negative side continues to refute the affirmative side's new arguments. When neither side can present new evidence or new arguments in two consecutive rounds, the debate ends. After the debate ends, the complete sequence of arguments is output as input for the subsequent adjudication steps.

[0115] The maximum number of debate rounds is set at three to five. When neither the affirmative nor the negative side can present new evidence or arguments for two consecutive rounds, it indicates that the debate has converged and continuing the debate will not help to obtain new information. At this point, the debate is terminated early. After the debate ends, the affirmative and the negative sides output their complete argument sequences generated during the debate. Each argument sequence contains arguments from several rounds and corresponding evidence citations. These argument sequences, along with all the multimodal evidence cited by both sides, serve as input to the adjudicator's intelligent agent for subsequent comprehensive evaluation and adjudication.

[0116] Reference Figure 5 As shown, the determination of whether the current behavior is abnormal includes:

[0117] Obtain the sequences of affirmative and negative arguments generated during the debate. Each argument sequence contains several rounds of arguments and corresponding evidence citations.

[0118] Logical consistency assessment is performed on the sequences of arguments for both sides, which is achieved by detecting whether there are self-contradictory statements in the sequences of arguments.

[0119] The sufficiency of evidence for the affirmative and negative argument sequences is assessed separately. The sufficiency of evidence assessment is achieved by weighting the number of multimodal evidence cited, the relevance score between the evidence and the argument, and the time coverage of the evidence.

[0120] The scenario rule compliance degree is evaluated for both the affirmative and negative argument sequences. The scenario rule compliance degree evaluation is achieved by semantically matching the arguments with scenario rules stored in an external knowledge base.

[0121] The scores for logical consistency, sufficiency of evidence, and compliance with scenario rules are weighted and summed to generate the overall scores for the affirmative and negative sides, respectively.

[0122] Compare the overall scores of the affirmative and negative sides. If the negative side's overall score is higher than the affirmative side's overall score, the current behavior is ruled as abnormal; otherwise, it is ruled as normal.

[0123] This step involves a refereeing AI to quantitatively evaluate the performance of both sides in the debate, thereby making an objective ruling. The specific quantitative processes for logical consistency evaluation, evidence sufficiency evaluation, and scenario rule compliance evaluation are as follows:

[0124] Logical consistency assessment is achieved by detecting whether there are self-contradictory statements in the sequence of arguments. The specific quantification process is as follows:

[0125] Extract all argument statements from the affirmative argument sequence and parse each argument statement into a propositional form. Specifically, identify the core predicates in the argument and their affirmative or negative forms. For example, "access control equipment has no fault records" is parsed into proposition P: "access control equipment has fault records", which has a false value; "access control equipment is undergoing maintenance" is parsed into proposition Q: "access control equipment is undergoing maintenance", which has a true value.

[0126] Iterate through all proposition pairs in the sequence of arguments, checking for the following two types of logical contradictions:

[0127] The first type is direct contradiction detection: For any two propositions, if one proposition is P and the other proposition is not P (i.e., the same core predicate is affirmed and negated at the same time), then it is determined that there is a direct contradiction; for example, "access control equipment has fault records" and "access control equipment has no fault records" constitute a direct contradiction.

[0128] The second type is transitive contradiction detection: For any three or more propositions, if there is a deductive relationship of proposition A→B and B→C, and there is also a proposition that is not C, then it is determined that there is a transitive contradiction; the deductive relationship is matched by a preset common sense rule base, which stores the causal relationship between common behaviors, such as "holding valid permissions → swiping the card should pass", "swiping the card to pass → the access control opens", etc.

[0129] Calculate the logical consistency score: The initial full score is set at 100 points. 10 points are deducted for each direct contradiction detected and 5 points are deducted for each propagated contradiction detected. The final score shall not be lower than 0 points.

[0130] The logical consistency assessment of the opposing argument sequence uses the same process, and the logical consistency score of the opposing side is calculated independently.

[0131] The sufficiency of evidence assessment is achieved by weighting the number of multimodal evidence cited, the relevance score of the evidence to the argument, and the temporal coverage of the evidence. The specific quantification process is as follows:

[0132] The first dimension involves traversing all arguments in the argument sequence, counting the number of multimodal evidence cited for each argument, calculating the average number of evidence for all arguments, comparing the average number of evidence with a preset evidence quantity benchmark, and calculating the evidence quantity score. The formula is: Evidence Quantity Score = min(Average Number of Evidence / Evidence Quantity Benchmark, 1).

[0133] The second dimension involves traversing all argument-evidence pairs in the argument sequence and calculating the semantic similarity between the argument text and the evidence text. Specifically, the following method is used: convert the arguments and evidence into Term Frequency-Inverse Document Frequency (TF-IDF) feature vectors, calculate the cosine similarity between the two vectors, and output a relevance score between zero and one. After traversal, the arithmetic mean of all relevance scores is calculated as the evidence relevance score. The TF-IDF feature vector is a method for converting text into numerical vectors, where each dimension represents the importance weight of a word in the current document, which is calculated by multiplying the Term Frequency (TF) by the Inverse Document Frequency (IDF).

[0134] The third dimension involves extracting the timestamps of all evidence in the argument sequence, determining the earliest and latest evidence timestamps, and calculating the proportion of the time range covered by these two timestamps to the total duration of the entire behavioral event. The formula is: Evidence Time Coverage = (Latest Evidence Timestamp - Earliest Evidence Timestamp) / Total Duration of the Behavioral Event.

[0135] The scores of the three dimensions are weighted and summed to obtain the evidence sufficiency score; the evidence sufficiency assessment of the opposing argument sequence is carried out using the same process, and the evidence sufficiency score of the opposing side is calculated independently.

[0136] The scenario rule compliance assessment is achieved by semantically matching the arguments with scenario rules stored in an external knowledge base. The specific quantification process is as follows:

[0137] An external knowledge base is pre-built to store behavioral rules for various scenarios. Each rule is stored in the form of a semantic pair of conditions and conclusions. For example, in the access control scenario, if a person has valid permissions and completes the card swipe action, the access control should be opened.

[0138] Traverse all arguments in the argument sequence and perform semantic matching for each argument: perform keyword matching and semantic similarity calculation between the argument content and the rules in the external knowledge base. The calculation method is as follows: extract the core verbs and nouns in the argument to form a feature set, calculate the Jaccard similarity coefficient with the feature set of each rule, select the rule with the highest similarity as the matching rule, and record the similarity value as the matching score.

[0139] If the matching score exceeds the preset semantic matching threshold, the argument is determined to be consistent with the scenario rules; otherwise, it is determined to be inconsistent.

[0140] The number of arguments that conform to the scenario rules is counted, and the proportion of each argument to the total number of arguments is calculated as the scenario rule conformity score.

[0141] The same process was used to evaluate the compliance of the opposing side's argument sequence with the scenario rules, and the compliance score of the opposing side's scenario rules was calculated independently.

[0142] The logical consistency score, evidence sufficiency score, and scenario rule compliance score of the affirmative argument sequence are weighted and summed to generate the affirmative comprehensive score; the negative comprehensive score is calculated using the same weighted summation method; the affirmative comprehensive score and the negative comprehensive score are compared. If the negative comprehensive score is higher than the affirmative comprehensive score, the current behavior is ruled as abnormal; otherwise, it is ruled as normal.

[0143] The output ruling, along with the complete chain of evidence formed by the winning party during the debate, serves as the basis for anomaly determination, including:

[0144] The adjudicator AI outputs the ruling and extracts all the multimodal evidence cited by the winning party during the debate, forming a complete chain of evidence;

[0145] The complete debate record of each abnormal behavior identification is stored in the experience database. The complete debate record includes: structured behavioral semantic sequence, debate anchor point, affirmative argument sequence, negative argument sequence, affirmative comprehensive score, negative comprehensive score, judge's ruling result and winning party's evidence chain.

[0146] When a new multimodal data stream is input, the structured behavioral semantic sequence corresponding to the current behavior is encoded to generate a semantic embedding vector, and the historical debate record with the highest semantic similarity is retrieved from the experience base.

[0147] Relevant arguments and evidence from retrieved historical debate records are used as prompts and injected into the context input of the affirmative and negative agents to enhance the effect of the current debate.

[0148] This step utilizes an experience base to reuse historical success cases. After each identification, the adjudicator agent stores the structured behavioral semantic sequence, debate anchors, the sequence of arguments from both sides, the comprehensive score, the ruling result, and the winning party's evidence chain into the experience base. When new data is input, the large model perception agent extracts the retrieval vector from the current behavior and matches the most similar historical records in the experience base using cosine similarity. If the match is successful, the arguments and evidence in that record are used as prompts and injected into the input context of the affirmative and negative agents, respectively. This allows both sides to generate higher-quality arguments by referring to historical success experiences, thereby continuously improving the identification capability.

[0149] The experience base is stored in a vector database. When storing, the referee agent assembles the structured behavioral semantic sequence into prompt words according to the embedding encoding prompt template, calls the pre-trained large model, and parses the semantic feature vector from the returned result. It also assembles the scene context information into prompt words according to the same embedding encoding prompt template, calls the pre-trained large model, and parses the scene context vector from the returned result. The two vectors are then concatenated to form an index vector.

[0150] Furthermore, based on the same inventive concept as the aforementioned method for multimodal recognition of abnormal behavior based on large-model intelligent agents, this solution proposes a multimodal recognition system for abnormal behavior based on large-model intelligent agents, comprising:

[0151] The intelligent agent architecture module is used to construct a large model intelligent agent architecture, which consists of a perception intelligent agent, a counterfactual reasoning intelligent agent, a positive intelligent agent, a negative intelligent agent, and a referee intelligent agent.

[0152] The intelligent agent encapsulation module is used to acquire multimodal data streams of the target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences as a perceptual intelligent agent; based on the first half of the observed data in the structured behavioral semantic sequences, it generates a semantic description of the expected result, calculates the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point, and encapsulates the functions of generating the expected result semantic description and calculating the semantic deviation vector as a counterfactual reasoning intelligent agent; based on the debate anchor point, it constructs an adversarial debate network between the affirmative and negative sides, generates normal behavior assumptions and abnormal behavior assumptions, conducts multiple rounds of debate, and encapsulates the function of generating normal behavior assumptions as an affirmative intelligent agent and the function of generating abnormal behavior assumptions as a negative intelligent agent; based on the affirmative and negative argument sequences generated during the debate, it evaluates the logical consistency, sufficiency of evidence, and compliance with scene rules, and adjudicates whether the current behavior is abnormal, and encapsulates the function of adjudicating whether the current behavior is abnormal as a referee intelligent agent;

[0153] An anomaly detection module is used to output the ruling result, and to attach the complete chain of evidence formed by the winning party during the debate as the basis for anomaly detection.

[0154] The intelligent agent encapsulation module includes:

[0155] A perceptual agent encapsulation unit is used to acquire multimodal data streams of a target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences into a perceptual agent.

[0156] The reasoning agent encapsulation unit is used to generate a semantic description of the expected result based on the first half of the observation data in the structured behavioral semantic sequence, calculate the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point, and encapsulate the functions of generating the semantic description of the expected result and calculating the semantic deviation vector into a counterfactual reasoning agent.

[0157] The positive and negative intelligent agent encapsulation unit is used to construct an adversarial debate network between the positive and negative sides based on the debate anchor point, generate normal behavior hypothesis and abnormal behavior hypothesis, conduct multiple rounds of debate, and encapsulate the function of generating normal behavior hypothesis as the positive intelligent agent and the function of generating abnormal behavior hypothesis as the negative intelligent agent.

[0158] The judge intelligent encapsulation unit is used to evaluate the logical consistency, sufficiency of evidence, and compliance with scenario rules based on the sequence of affirmative and negative arguments generated during the debate, to determine whether the current behavior is abnormal, and to encapsulate the function of determining whether the current behavior is abnormal into a judge intelligent agent.

[0159] In summary, the advantages of this invention are: it achieves accurate identification of multimodal abnormal behavior through multi-agent debate and counterfactual causal reasoning, and has the advantages of logical transparency, interpretability, iterability and strong anti-interference.

[0160] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

Claims

1. A method for multimodal recognition of abnormal behavior based on a large model intelligent agent, characterized in that, include: A large-scale model intelligent agent architecture is constructed, which consists of a perception agent, a counterfactual reasoning agent, a positive agent, a negative agent, and a referee agent. Acquire multimodal data streams of the target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences into a perceptual agent; Based on the observation data of the first half of the structured behavioral semantic sequence, generate a semantic description of the expected result, calculate the semantic deviation vector between the expected result and the actual result semantic description as the debate anchor point, and encapsulate the functions of generating the semantic description of the expected result and calculating the semantic deviation vector into a counterfactual reasoning agent. Based on the debate anchor points, an adversarial debate network is constructed between the affirmative and negative sides to generate normal behavior assumptions and abnormal behavior assumptions, conduct multiple rounds of debate, and encapsulate the function of generating normal behavior assumptions as the affirmative intelligent agent and the function of generating abnormal behavior assumptions as the negative intelligent agent. Based on the sequence of arguments for and against generated during the debate, the system assesses logical consistency, sufficiency of evidence, and compliance with scenario rules to determine whether the current behavior is abnormal. The function of determining whether the current behavior is abnormal is then encapsulated as a referee agent. The ruling is output along with the complete chain of evidence formed by the winning party during the debate, which serves as the basis for the anomaly determination.

2. The method for multimodal recognition of abnormal behavior based on a large model intelligent agent according to claim 1, characterized in that, The large-model intelligent agent architecture includes: The large model perceptual agent is used to transform multimodal data streams into structured behavioral semantic sequences. The large-scale counterfactual reasoning agent is used to generate a semantic description of the expected result based on the observation data of the first half of the current behavior and the context information of the scene, and to calculate the semantic deviation vector as the debate anchor point. The affirmative and negative intelligent agents are used to generate normal behavior assumptions and abnormal behavior assumptions based on debate anchor points, structured behavioral semantic sequences and scene context information, and conduct multiple rounds of debate. The adjudicator is used to determine whether the current behavior is abnormal based on the sequence of arguments for and against generated during the debate, and outputs the adjudication result and the winning party's evidence chain.

3. The method for multimodal recognition of abnormal behavior based on a large model intelligent agent according to claim 2, characterized in that, The step of acquiring the multimodal data stream of the target scene and generating a structured behavioral semantic sequence includes: Acquire multimodal data streams of the target scene, including but not limited to: visual data, audio data, and text data; The continuous frame sequence in the multimodal data stream is divided into several time windows of equal length according to the time order, and the multimodal semantic feature vector of each window is extracted; Calculate the semantic aberration degree between the multimodal semantic feature vectors of adjacent windows, and mark the time points where the semantic aberration degree exceeds the dynamic threshold as semantic event boundaries; Cross-modal semantic anchor matching is performed based on semantic event boundaries, and event pairs that meet the mutual information threshold and whose time difference is within a preset window are marked as cross-modal semantic anchors; Based on cross-modal semantic anchor points, non-linear alignment is performed on multimodal data streams to eliminate time offset between multiple source sensors; The aligned multimodal data stream is segmented into atomic behavior fragments based on semantic event boundaries; Each atomic behavior fragment is subjected to structured semantic annotation, and the behavior tuple field is extracted to form a structured behavior semantic sequence.

4. The method for multimodal recognition of abnormal behavior based on a large model intelligent agent according to claim 3, characterized in that, The step of generating a semantic description of the expected result based on the first half of the observation data in the structured behavioral semantic sequence, and calculating the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point includes: Extract the first half of the observation data of the current behavior from the structured behavioral semantic sequence. The first half of the observation data is a subset of behavioral tuples from the start timestamp of the behavior to the current timestamp. Combine the contextual information of the scene to perform causal inference, and generate a semantic description of the expected result that the behavior should produce in a normal scene according to the preset text template; The scene context information includes time attributes, spatial attributes, character roles, device status, and historical normal mode; Extract the sequence of behavioral tuples corresponding to the complete observation data of the current behavior from the structured behavioral semantic sequence, and convert it into a natural language description according to the preset text template as the semantic description of the actual result; The expected result semantic description is encoded to generate the first semantic embedding vector, and similarity matching is performed with the semantic embedding vector in the normal behavior template library. Several templates with the highest similarity are selected, and the semantic feature vectors in the selected templates are fused according to similarity weight to generate the first semantic feature vector. The semantic description of the actual result is encoded to generate a second semantic embedding vector, and similarity matching is performed with the semantic embedding vector in the normal behavior template library. Several templates with the highest similarity are selected, and the semantic feature vectors in the selected templates are fused according to similarity weight to generate a second semantic feature vector. The difference between the first semantic feature vector and the second semantic feature vector is calculated to obtain the semantic deviation vector, which is then used as the debate anchor point.

5. The method for multimodal recognition of abnormal behavior based on a large model intelligent agent according to claim 4, characterized in that, The process of constructing an adversarial debate network between the affirmative and negative sides based on debate anchor points, generating normal behavior assumptions and abnormal behavior assumptions, and conducting multiple rounds of debate includes: Using debate anchors and scene context information as input, the system generates a normal behavior hypothesis based on the deviation value in the debate anchors and the scene context information, and retrieves multimodal evidence supporting the hypothesis from the structured behavioral semantic sequence to generate the affirmative argument. Using debate anchors and scene context information as input, the system generates behavioral anomaly hypotheses based on deviation values ​​in the debate anchors and scene context information, and retrieves multimodal evidence supporting the hypotheses from structured behavioral semantic sequences to generate opposing arguments. Multiple rounds of structured debate were conducted according to the pre-set debate agreement. Each round of speeches included the statement of arguments and the citation of evidence, while the rebuttal speeches included the statement of rebuttal and the citation of counter-evidence. The debate ends prematurely if neither the affirmative nor the negative side can present new evidence or arguments in two consecutive rounds.

6. The method for multimodal recognition of abnormal behavior based on a large model intelligent agent according to claim 5, characterized in that, The process of assessing logical consistency, sufficiency of evidence, and compliance with scenario rules based on the sequences of arguments presented by the affirmative and negative sides during the debate, and determining whether the current behavior is abnormal, includes: Obtain the sequences of affirmative and negative arguments generated during the debate. Each argument sequence contains several rounds of arguments and corresponding evidence citations. Logical consistency assessment is performed on the sequences of arguments for both sides, which is achieved by detecting whether there are self-contradictory statements in the sequences of arguments. The sufficiency of evidence for the affirmative and negative argument sequences is assessed separately. The sufficiency of evidence assessment is achieved by weighting the number of multimodal evidence cited, the relevance score between the evidence and the argument, and the time coverage of the evidence. The scenario rule compliance degree is evaluated for both the affirmative and negative argument sequences. The scenario rule compliance degree evaluation is achieved by semantically matching the arguments with scenario rules stored in an external knowledge base. The scores for logical consistency, sufficiency of evidence, and compliance with scenario rules are weighted and summed to generate the overall scores for the affirmative and negative sides, respectively. Compare the overall scores of the affirmative and negative sides. If the negative side's overall score is higher than the affirmative side's overall score, the current behavior is ruled as abnormal; otherwise, it is ruled as normal.

7. The method for multimodal recognition of abnormal behavior based on a large model intelligent agent according to claim 6, characterized in that, The output ruling, along with the complete chain of evidence formed by the winning party during the debate, serves as the basis for anomaly determination, including: The adjudicator AI outputs the ruling and extracts all the multimodal evidence cited by the winning party during the debate, forming a complete chain of evidence; The complete debate record of each abnormal behavior identification is stored in the experience database. The complete debate record includes: structured behavioral semantic sequence, debate anchor point, affirmative argument sequence, negative argument sequence, affirmative comprehensive score, negative comprehensive score, judge's ruling result and winning party's evidence chain. When a new multimodal data stream is input, the structured behavioral semantic sequence corresponding to the current behavior is encoded to generate a semantic embedding vector, and the historical debate record with the highest semantic similarity is retrieved from the experience base. Relevant arguments and evidence from retrieved historical debate records are used as prompts and injected into the context input of the affirmative and negative agents to enhance the effect of the current debate.

8. A multimodal recognition system for abnormal behavior based on a large model intelligent agent, characterized in that, The method for implementing the multimodal recognition of abnormal behavior based on a large model agent as described in any one of claims 1-7 includes: The intelligent agent architecture module is used to construct a large model intelligent agent architecture, which consists of a perception intelligent agent, a counterfactual reasoning intelligent agent, a positive intelligent agent, a negative intelligent agent, and a referee intelligent agent. The intelligent agent encapsulation module is used to acquire multimodal data streams of the target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences as a perceptual intelligent agent; based on the first half of the observed data in the structured behavioral semantic sequences, it generates a semantic description of the expected result, calculates the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point, and encapsulates the functions of generating the expected result semantic description and calculating the semantic deviation vector as a counterfactual reasoning intelligent agent; based on the debate anchor point, it constructs an adversarial debate network between the affirmative and negative sides, generates normal behavior assumptions and abnormal behavior assumptions, conducts multiple rounds of debate, and encapsulates the function of generating normal behavior assumptions as an affirmative intelligent agent and the function of generating abnormal behavior assumptions as a negative intelligent agent; based on the affirmative and negative argument sequences generated during the debate, it evaluates the logical consistency, sufficiency of evidence, and compliance with scene rules, and adjudicates whether the current behavior is abnormal, and encapsulates the function of adjudicating whether the current behavior is abnormal as a referee intelligent agent; An anomaly determination module is used to output the ruling result, and to attach the complete chain of evidence formed by the winning party during the debate as the basis for anomaly determination.

9. The multimodal recognition system for abnormal behavior based on a large model intelligent agent according to claim 8, characterized in that, The intelligent agent encapsulation module includes: A perceptual agent encapsulation unit is used to acquire multimodal data streams of a target scene, generate structured behavioral semantic sequences, and encapsulate the function of generating structured behavioral semantic sequences into a perceptual agent. The reasoning agent encapsulation unit is used to generate a semantic description of the expected result based on the first half of the observation data in the structured behavioral semantic sequence, calculate the semantic deviation vector between the expected result and the actual result semantic description as a debate anchor point, and encapsulate the functions of generating the semantic description of the expected result and calculating the semantic deviation vector into a counterfactual reasoning agent. The positive and negative intelligent agent encapsulation unit is used to construct an adversarial debate network between the positive and negative sides based on the debate anchor point, generate normal behavior hypothesis and abnormal behavior hypothesis, conduct multiple rounds of debate, and encapsulate the function of generating normal behavior hypothesis as the positive intelligent agent and the function of generating abnormal behavior hypothesis as the negative intelligent agent. The judge intelligent encapsulation unit is used to evaluate the logical consistency, sufficiency of evidence, and compliance with scenario rules based on the sequence of affirmative and negative arguments generated during the debate, to determine whether the current behavior is abnormal, and to encapsulate the function of determining whether the current behavior is abnormal into a judge intelligent agent.

Images