A multimodal cognitive agent video semantic analysis method

Abstract

This application discloses a video semantic analysis method for multimodal cognitive intelligent agents, belonging to the field of computer technology. The analysis method includes: acquiring natural language instructions and the constraints contained in the natural language instructions to obtain deconstructed intentions; acquiring multiple keyframes of the video stream based on the deconstructed intentions to generate a cognitive benchmark for semantic analysis; establishing dependencies between deconstructed intentions based on a thought chain reasoning mechanism and the cognitive benchmark to obtain a task graph; acquiring the tasks allocated within each heterogeneous computing node based on the task graph and advancing the task graph; and updating the posterior confidence of subsequent events in real time based on the observation evidence set continuously acquired during the advancement of the knowledge graph, and adjusting the logical links. This solves the problems of excessive resource consumption and the resulting difficulty in timely acquisition of effective information in existing technologies.

Description

Technical Field

[0001] This application belongs to the field of computer technology, specifically relating to a video semantic analysis method for multimodal cognitive intelligent agents. Background Technology

[0002] As the core carrier of digital mapping of the physical world, video surveillance data exhibits significant long-tail distribution characteristics and spatiotemporal sparsity. How to accurately extract deconstructed high-value information from massive pixel streams under limited computing power constraints has become a key scientific problem in the field of computer vision and multimedia systems.

[0003] Existing video analytics technologies primarily follow two distinct evolutionary paths, both facing insurmountable bottlenecks. The first category comprises discriminative models such as YOLO and ResNet, which, through lightweight network deconstruction, achieve efficient deployment and real-time detection at the edge, forming the cornerstone of current video surveillance systems. However, these models rely on predefined labels within closed sets, essentially acting as pixel-level pattern matching tools. They lack causal reasoning capabilities and open-domain understanding of video context, leading to frequent false positives or false negatives when dealing with complex interactions such as theft or fighting. The second category comprises recently emerging Multimodal Large Language Models (MLLMs) based on the Transformer architecture. Through massive pre-training on large datasets, they possess powerful zero-shot generalization capabilities and logical reasoning intelligence. However, their massive parameter scale, often reaching hundreds of billions, results in high memory consumption and inference latency, and they struggle to directly process ultra-long-series video streams, exhibiting a "high intelligence but low throughput" technical characteristic. This means that in practical applications, due to limitations in computing power and response speed, large models often only serve as offline analysis tools, failing to enable real-time monitoring and early warning tasks. Therefore, bridging the gap between the real-time nature of underlying visual perception and the logical nature of top-level semantic cognition, and constructing a new analytical architecture that combines efficiency and intelligence, is a pressing problem that needs to be solved. Summary of the Invention

[0004] To address the problems of excessive computational resource consumption and difficulty in timely acquisition of effective information in existing technologies, this application discloses a multimodal cognitive agent video semantic analysis method, specifically:

[0005] A multimodal cognitive agent video semantic analysis method, the analysis method comprising:

[0006] Obtain natural language instructions and the constraints contained in the natural language instructions to obtain the deconstructed intent;

[0007] Based on the deconstructed intent, multiple keyframes are acquired from the video stream to generate a cognitive benchmark for semantic analysis.

[0008] Based on the thought chain reasoning mechanism and the aforementioned cognitive benchmark, the dependency relationships between deconstructed intentions are established to obtain a task graph;

[0009] Based on the task graph, the tasks allocated within each heterogeneous computing node are obtained, and the task graph is advanced.

[0010] Based on the observation evidence set continuously acquired during the knowledge graph-based advancement process, the posterior confidence of subsequent events is updated in real time, and logical links are adjusted accordingly.

[0011] Optionally, the step of obtaining the natural language instruction and the constraints contained in the natural language instruction to obtain the deconstructed intent includes:

[0012] Natural language instructions are acquired and analyzed by a large language model;

[0013] When analyzing natural language instructions using a large language model, the visual tasks, logical dependencies, and semantic information flow content contained in the natural language are obtained to arrive at the deconstructed intent.

[0014] Optionally, the step of acquiring multiple keyframes from the video stream based on the deconstructed intent to generate a cognitive benchmark for semantic analysis includes:

[0015] Based on the deconstructed intent, the scene to be identified from the video stream is obtained;

[0016] Sampling is performed at a sparse sampling rate to extract several keyframes from the video stream;

[0017] Intra-image information analysis is performed on the aforementioned keyframes, and a scene context description is generated;

[0018] Based on the scene context description, the constraint weights within the video stream are adjusted to obtain prior knowledge of the video stream and to acquire an initialization mechanism for environmental information.

[0019] The initialization mechanism of the environmental information is associated with natural language information to generate the cognitive basis for semantic analysis.

[0020] Optionally, the step of establishing dependencies between deconstructed intentions based on the thought chain reasoning mechanism and the cognitive benchmark to obtain a task graph includes:

[0021] The visual tasks in the deconstructed intent are obtained, including object recognition tasks, trajectory extraction tasks, and behavior classification tasks.

[0022] Obtain the logical dependencies and semantic information flow from the deconstructed intent;

[0023] The visual task, logical dependencies, and semantic information flow are determined to obtain a task graph, which is represented by the following equation:

[0024] ,

[0025] Where G represents a directed acyclic graph of the task graph; N represents a set of atomic subtasks; and E represents logical dependencies and triggering relationships.

[0026] The resource and information gain relationships of all nodes in the task graph are obtained to determine the optimal scheduling strategy for task graph construction.

[0027] Optionally, the step of obtaining the resource and information gain relationships of all nodes in the task graph to obtain the optimal scheduling strategy for task graph construction includes:

[0028] The resource consumption and information gain generated in each node are obtained, and the optimal scheduling policy equation is used to select a policy. The optimal scheduling policy equation is as follows:

[0029] ,

[0030] in, Indicates the optimal scheduling strategy; This represents the computational cost of the p-th subtask under strategy π. Let π represent the expected information gain from the p-th subtask under strategy π. and These represent tradeoff coefficients, used to adjust the emphasis on computational cost and expected information gain in the scheduling strategy;

[0031] Based on the calculation results of the optimal scheduling strategy equation, the most suitable tool for each logical node is obtained from the tool set {T}, and the interpretable invocation parameters for each most suitable tool are configured.

[0032] The toolset {T} includes side-end tools and cloud-based tools.

[0033] Optionally, the step of obtaining the tasks allocated within each heterogeneous computing node based on the task graph and advancing the task graph includes:

[0034] Based on the task graph, video segments are obtained from the video stream;

[0035] Obtain the semantic density of all the video segments to quantify the information entropy value contained in the set of video frames corresponding to the video segments. The semantic density equation is as follows:

[0036] ,

[0037] Where SemDensity(x) represents semantic density; Represents the motion saliency entropy value of video segment x based on optical flow; The Intersection over Union (IoU) ratio is used to represent the degree of geometric overlap between objects within a video clip. and Indicates the adaptive weighting coefficients; x represents the video segment; and To represent different things within a video clip;

[0038] Based on the semantic density, the tasks allocated within each heterogeneous computing node are obtained, and the task graph is advanced.

[0039] Optionally, the step of obtaining the tasks allocated within each heterogeneous computing node based on the semantic density and advancing the task graph includes:

[0040] Set a threshold for judging task allocation on heterogeneous computing nodes to compare the types of tasks allocated within heterogeneous computing nodes;

[0041] The semantic density and the task allocation judgment threshold are compared to obtain the tasks allocated within each heterogeneous computing node. The task allocation judgment equation is as follows:

[0042] in, This indicates the act of uploading to the cloud; Indicates endpoint processing behavior; Represents semantic similarity, used to express the cosine similarity between the target feature vector and the user's natural language intent vector; Represents the target feature vector; Represents the user's natural language intent vector; This represents the uncertainty threshold, used to determine whether to force the video segment to be uploaded to the cloud; This represents the cloud-based threshold, used to determine whether a video segment corresponding to semantic density is a high-value segment.

[0043] Optionally, the observation evidence set continuously acquired during the knowledge graph-based advancement process is used to update the posterior confidence of subsequent events in real time and to adjust the logical links, including:

[0044] Based on the knowledge graph-based advancement process, observational evidence is acquired in real time, and an observational dataset is generated, which is represented as follows:

[0045] ,

[0046] Among them, E -t Represents the set of observational evidence at time t; e1~e kThis represents the observation data; k represents the observation data label;

[0047] Obtain the correspondence between the probability of event occurrence and the time of occurrence within the observed dataset, and obtain the posterior confidence score. The equation for determining the posterior confidence score is:

[0048] ,

[0049] in, Indicates event H i Likelihood within the set of observational evidence acquired at time t; Indicates event H i The likelihood of the event; i and j represent the sequence indices of the event; m represents the total number of sequence indices of the event; Indicates event H j The probability of occurrence; H i H represents the hypothesis of the i-th event; j P(H) represents the hypothesis of the j-th event; i ) represents event H i The prior probability; Represents time H j likelihood; E t This represents the observation dataset acquired at time t;

[0050] The logical links are adjusted based on the changing trend of the posterior probability.

[0051] Optionally, adjusting the logical link based on the changing trend of the posterior probability includes:

[0052] Set a logic link judgment threshold to control the logic link adjustment trigger;

[0053] when When the descent gradient over a consecutive video frames is not lower than the confidence threshold When the current logical link fails, select another logical link.

[0054] When the same feature exists within the current logical link, there is a mutual exclusion assumption H a and H b , making If the evidence for adjusting the current logical link is insufficient, further analysis of the current logical link is required. This represents the likelihood of a mutually exclusive hypothesis across all observed datasets. This represents the likelihood of another mutually exclusive hypothesis across all observed datasets; This indicates that the logical link adjusts the evidence judgment threshold.

[0055] The beneficial effects of this application include:

[0056] 1. Reduced computational resource consumption. In the technical solution of this application, the computational resource consumption of each logical node is analyzed, and the validity of video frames is analyzed. Then, video frames that obviously do not contain valid information are removed. In the specific processing, it can be ensured that the video frames analyzed have complete information, and the results can be achieved based on these video frames. Since irrelevant information is removed in this process, the computational resource consumption is significantly reduced.

[0057] 2. Improved timeliness of analysis results. The technical solution of this application, on the one hand, rationally filters video frames, and on the other hand, judges the validity of information in each video frame. Based on the judgment results, feature uploading and analysis are performed, or information analysis is conducted on the entire video segment. This enables the filtering and adjustment of information carried within the video stream, thereby significantly improving the timeliness of the analysis work.

[0058] 3. Improved flexibility of the analysis link. In the technical solution of this application, by analyzing the development of confidence of each node in the obtained logical link, it is possible to analyze whether there are any technical defects in the currently adopted logical link, thereby selecting the most reasonable technical link to ensure that the analysis work can be adjusted at any time according to the needs during the final analysis process. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the embodiments of this application or the prior art will be briefly introduced below. Obviously, the following description is only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. The drawings are used to provide a further understanding of this disclosure and constitute a part of the specification. They are used together with the following detailed description to explain this disclosure, but do not constitute a limitation of this disclosure. In the drawings:

[0060] Figure 1 A flowchart of a multimodal cognitive agent video semantic analysis method provided in this application embodiment;

[0061] Figure 2 This is a schematic diagram of the system execution flow of a multimodal cognitive intelligent agent video semantic analysis method provided in an embodiment of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, in the embodiments of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] Early video analysis techniques relied primarily on hand-designed feature operators. With the rise of deep learning, methods based on convolutional neural networks (CNNs) gradually became mainstream. 3D convolutional networks, such as C3D and I3D, significantly improved action recognition accuracy by learning hierarchical representations of spatiotemporal features. Subsequently, to address long-distance temporal dependencies, researchers introduced non-local neural networks and models like VideoMAE based on the Transformer architecture, utilizing self-attention mechanisms to capture global correlations between video frames. Although these discriminative models achieved excellent performance on specific datasets (such as Kinetics-400), they essentially still followed a supervised learning paradigm, relying on predefined closed-set labels for training. This "classification-centric" paradigm often leads to poor generalization ability when faced with highly complex and long-tailed anomalous behaviors in the real world. For example, while a model can easily recognize the atomic action of "running," it struggles to understand higher-level events with socio-semantic meanings, such as "theft" or "fighting," which are composed of multiple atomic actions. Traditional video understanding technologies, although solving the "pixel-to-feature" mapping problem, have consistently failed to bridge the semantic gap between "feature perception" and "logical cognition," lacking the ability to reason about causal relationships within video scenes. In recent years, multimodal large language models (MLLMs), represented by CLIP, GPT-4V, and Video-LLaMA 2, have demonstrated powerful open-domain alignment and zero-shot reasoning capabilities through large-scale cross-modal pre-training, providing a new paradigm for semantic understanding of surveillance videos. Simultaneously, research on LLM-based agents has rapidly emerged, extending models from information processors to autonomous systems with task planning, tool usage, and environmental interaction capabilities through function calls. However, directly applying MLLM to continuous long videos still faces three core bottlenecks: First, long video input leads to a linear increase in token size and triggers a sharp rise in attention computation and memory overhead, making it difficult for end-to-end slice inference to cover the complete temporal evidence chain; second, without explicit intermediate evidence and an operable inference interface, the system struggles to locate and correct false positives, false negatives, and hallucinations; third, without introducing task divide-and-conquer and key segment filtering, cloud computing power will remain under high occupancy for a long time, making large-scale deployment in multi-channel monitoring scenarios difficult. To address the problems existing in the prior art, this application discloses...

[0064] A multimodal cognitive agent video semantic analysis method, such as Figure 1 The diagram shown is a flowchart of a multimodal cognitive agent video semantic analysis method provided in an embodiment of this application. Specifically:

[0065] S110. Obtain natural language instructions and the constraints contained in the natural language instructions to obtain the deconstructed intent.

[0066] S120. Based on the deconstructed intent, multiple keyframes are acquired from the video stream to generate a cognitive benchmark for semantic analysis.

[0067] S130. Based on the thought chain reasoning mechanism and the cognitive benchmark, establish the dependency relationship between the deconstructed intentions to obtain the task graph.

[0068] S140. Based on the task graph, obtain the tasks allocated in each heterogeneous computing node, and advance the task graph.

[0069] S150. The set of observational evidence continuously acquired during the knowledge graph-based advancement process is used to update the posterior confidence of subsequent events in real time and adjust the logical links accordingly.

[0070] The following will provide a detailed explanation of all the steps above:

[0071] As described in step S110, the purpose of this step is that, during the operation of the entire system, it first needs to acquire the input natural language, and then analyze the natural language to obtain the intent contained within it, thereby obtaining the basis for information analysis. Specifically:

[0072] Natural language instructions are acquired and analyzed by a large language model;

[0073] When analyzing natural language instructions using a large language model, the visual tasks, logical dependencies, and semantic information flow content contained in the natural language are obtained to arrive at the deconstructed intent.

[0074] The so-called semantic information flow refers to the logical and dependent relationships between the various content words contained in the entire natural sentence, as well as the direction of information flow constructed by the dependency relationships. For example, the logical dependency relationship of "woman wearing red clothes" is: red - clothes - woman, and the direction of information flow is from "red" to "woman".

[0075] In scenarios involving heterogeneous computing power collaboration, video semantic analysis can be modeled as a multi-objective optimization problem. Let the input video stream be V = {f1, f2, ..., f...} T}, where f tLet represent the video frame at time t, where T is the total duration. The user's natural language query instruction is denoted as Q. The system's goal is to generate a structured report R, which contains a set S of key segments satisfying instruction Q and their corresponding semantic descriptions D. Unlike traditional single-model static inference, the core of this system lies in finding an optimal computational power scheduling strategy that minimizes the total computational cost C while satisfying semantic accuracy constraints.

[0076] As described in step S120, the purpose of this step is to obtain a contextual description of the scene, which allows for the injection of prior knowledge to construct a cognitive baseline for subsequent information analysis. Specifically:

[0077] Based on the deconstructed intent, the scene to be identified from the video stream is obtained;

[0078] Sampling is performed at a sparse sampling rate to extract several keyframes from the video stream;

[0079] Intra-image information analysis is performed on the aforementioned keyframes, and a scene context description is generated;

[0080] Based on the scene context description, the constraint weights within the video stream are adjusted to obtain prior knowledge of the video stream and to acquire an initialization mechanism for environmental information.

[0081] The initialization mechanism of the environmental information is associated with natural language information to generate the cognitive basis for semantic analysis.

[0082] Simultaneously with intent deconstruction, the system executes a panoramic initialization strategy. The agent first extracts several keyframes from the video stream V at a sparse sampling rate and generates a scene context description (SceneContext) using a lightweight image description model. This step aims to inject prior knowledge into the agent; for example, when the scene is identified as a "nighttime warehouse," the agent will automatically reduce its reliance on color features and instead increase its focus on infrared features or motion trajectories. Through this initialization mechanism that combines explicit user commands with implicit environmental priors, the system constructs a physically constrained cognitive baseline for subsequent task planning.

[0083] As described in step S130, the purpose of this step is to construct a task graph across the entire natural language processing system by acquiring cognitive benchmarks. This task graph can then be used for subsequent information analysis. Specifically:

[0084] The visual tasks in the deconstructed intent are obtained, including object recognition tasks, trajectory extraction tasks, and behavior classification tasks.

[0085] Obtain the logical dependencies and semantic information flow from the deconstructed intent;

[0086] The visual task, logical dependencies, and semantic information flow are determined to obtain a task graph, which is represented by the following equation:

[0087] ,

[0088] Where G represents the task graph; N represents the visual task; and E represents logical dependencies and semantic information flow.

[0089] Obtain the resource and information gain relationships of all nodes in the task graph to derive the optimal scheduling strategy for task graph construction, including:

[0090] The resource consumption and information gain generated in each node are obtained, and the optimal scheduling policy equation is used to select a policy. The optimal scheduling policy equation is as follows:

[0091] ,

[0092] in, Indicates the optimal scheduling strategy; This represents the computational cost of the p-th subtask under strategy π. Let π represent the expected information gain from the p-th subtask under strategy π. and These represent tradeoff coefficients, used to adjust the emphasis on computational cost and expected information gain in the scheduling strategy;

[0093] Based on the calculation results of the optimal scheduling strategy equation, the most suitable tool for each logical node is obtained from the tool set {T}, and the interpretable invocation parameters for each most suitable tool are configured.

[0094] The toolset {T} includes side-end tools and cloud-based tools.

[0095] To transform high-level abstract intentions into executable concrete operations, this application introduces a Chain of Thought (CoT) reasoning mechanism, recursively decomposing complex video analysis tasks into a series of atomized visual operation steps. The agent not only generates the reasoning steps but also explicitly constructs the dependencies between these steps, ultimately forming a Directed Acyclic Graph (DAG), denoted as the task graph. .

[0096] Where N represents the visual task, such as "detecting pedestrians", "extracting trajectories", "behavior classification", etc.

[0097] In terms of the overall technical solution, the established case study, for the instruction "check for non-motorized vehicle theft," involves a task graph generated by the agent that is not a simple linear process, but a logic tree with conditional branches. First, the root node n1 is defined as "detecting people and bicycles in the scene"; then, the child node n2 is defined as "calculating the intersection-over-union (IoU) and interaction time between people and bicycles"; only when the output of n2 meets a specific threshold will the subsequent node n3 be activated to "extract interaction segments and upload them to the cloud for action recognition." This graph-based arrangement method offers high flexibility, allowing the agent to dynamically prune or expand the graph structure according to the difficulty of the task.

[0098] To achieve optimal task orchestration, this application defines an objective function for node selection. For each node to be decided in the graph, the agent calculates the ratio of its expected information gain to computational cost. Assume the current system state is S. t The expected state after performing a certain atomic operation 'a' is S. t+1 Then the utility function U(a) of this operation is defined as the ratio of the reduction in Shannon information entropy to the execution cost. The agent uses a greedy strategy or beam search to search for each node n in the tool set T. i Choose the most suitable tool k The system configures interpretable invocation parameters (target entity, time window, threshold, etc.) for these nodes. For example, high-throughput nodes at the front end of the graph tend to be bound to edge-side perception tools, while semantic discrimination and causal attribution nodes are bound to cloud-based cognitive tools. Through this mathematical graph construction and tool mapping process, the system transforms fuzzy natural language instructions into precise, executable, and cost-controllable heterogeneous computational flows, and is naturally compatible with model implementations from different vendors and of different scales.

[0099] As described in step S140, the purpose of this step is to determine the assigned tasks within each computing node, so that information processing can be performed based on the set assigned tasks to obtain the progress results of the task graph. Specifically:

[0100] Based on the task graph, video segments are obtained from the video stream;

[0101] Obtain the semantic density of all the video segments to quantify the information entropy value contained in the set of video frames corresponding to the video segments. The semantic density equation is as follows:

[0102] ,

[0103] Where SemDensity(x) represents semantic density; Represents the motion saliency entropy value of video segment x based on optical flow; The Intersection over Union (IoU) ratio is used to represent the degree of geometric overlap between objects within a video clip. and Indicates the adaptive weighting coefficients; x represents the video segment; and To represent different things within a video clip;

[0104] Based on the semantic density, the tasks allocated within each heterogeneous computing node are obtained, and the task graph is advanced, including:

[0105] Set a threshold for judging task allocation on heterogeneous computing nodes to compare the types of tasks allocated within heterogeneous computing nodes;

[0106] The semantic density and the task allocation judgment threshold are compared to obtain the tasks allocated within each heterogeneous computing node. The task allocation judgment equation is as follows:

[0107] in, This indicates the act of uploading to the cloud; Indicates endpoint processing behavior; Represents semantic similarity, used to express the cosine similarity between the target feature vector and the user's natural language intent vector; Represents the target feature vector; Represents the user's natural language intent vector; This represents the uncertainty threshold, used to determine whether to force the video segment to be uploaded to the cloud; This represents the cloud-based threshold, used to determine whether a video segment corresponding to semantic density is a high-value segment.

[0108] After constructing a task graph based on the thought chain, the core challenge of the system lies in how to efficiently distribute atomic tasks within the graph among heterogeneous computing nodes. Monitoring data exhibits significant spatiotemporal sparsity, meaning that the vast majority of images contain low-value, repetitive information. Continuously sending redundant pixels to the cloud for end-to-end MLLM inference not only leads to token explosion and the superposition of attention computational overhead but also causes long-term full load of cloud computing power. While network bandwidth is usually not the primary bottleneck in single-path scenarios, it can still become a systemic constraint when multiple monitoring clusters converge. Therefore, this paper primarily proposes an adaptive traffic splitting and collaboration mechanism based on semantic density from the perspective of "computing power-semantic value."

[0109] Based on the obtained semantic density function, a three-level collaborative decision-making strategy was constructed:

[0110] Edge-Local Loop: When In situations such as pure background or unrelated pedestrians, the endpoint only records metadata and directly discards video frames, achieving zero bandwidth usage.

[0111] Feature Upload: When When a suspected target is discovered but its behavior is unclear, the device extracts a lightweight feature vector (such as ReID features or key point skeletons) and uploads it. The cloud then uses a feature matching algorithm for rapid analysis.

[0112] Full-Scale Inference: Only when When the edge logic detects "strong interactive behavior" (such as the IoU overlap between two targets continuously exceeding a threshold), the system triggers "deep cognitive mode". At this time, the original video slice is transmitted to the cloud without loss and MLLM is activated to perform multimodal detailed description and causal inference.

[0113] Through this mechanism, the system formalizes the boundary between "fast thinking" and "slow thinking" at the mathematical level, achieving Pareto optimality for computational efficiency and semantic capture rate.

[0114] As described in step S150, the purpose of this step is to analyze the validity of the logic links used based on the actual situation, and to determine the validity of the corresponding logic circuits based on the analysis results, thereby analyzing whether adjustments are needed. Specifically:

[0115] Based on the knowledge graph-based advancement process, observational evidence is acquired in real time, and an observational dataset is generated, which is represented as follows:

[0116] ,

[0117] Among them, E -t Represents the set of observational evidence at time t; e1~e k This represents the observation data; k represents the observation data label;

[0118] Obtain the correspondence between the probability of event occurrence and the time of occurrence within the observed dataset, and obtain the posterior confidence score. The equation for determining the posterior confidence score is:

[0119] ,

[0120] in, Indicates event H i Likelihood within the set of observational evidence acquired at time t; Indicates event H i The likelihood of the event; i and j represent the sequence indices of the event; m represents the total number of sequence indices of the event; Indicates event H j The probability of occurrence; H i H represents the hypothesis of the i-th event; j P(H) represents the hypothesis of the j-th event; i ) represents event H i The prior probability; Represents time H j likelihood; E t This represents the observation dataset acquired at time t;

[0121] Based on the changing trend of the posterior probability, the logical link is adjusted, including:

[0122] Set a logic link judgment threshold to control the logic link adjustment trigger;

[0123] when When the descent gradient over a consecutive video frames is not lower than the confidence threshold When the current logical link fails, select another logical link.

[0124] When the same feature exists within the current logical link, there is a mutual exclusion assumption H a and H b , making If the evidence for adjusting the current logical link is insufficient, further analysis of the current logical link is required. This represents the likelihood of a mutually exclusive hypothesis across all observed datasets. This represents the likelihood of another mutually exclusive hypothesis across all observed datasets; This indicates that the logical link adjusts the evidence judgment threshold.

[0125] Traditional video analytics systems often employ an "open-loop execution" model, meaning that once the process begins, intervention is impossible. However, in complex open scenarios, initial observations are often subject to noise or occlusion, causing fixed strategies to fail. To endow the system with "meta-cognitive" capabilities, this paper introduces a Bayesian Inference Network to construct a closed-loop feedback control mechanism.

[0126] Here, let the prior probability of the system at time t regarding the event hypothesis H be P(H) (obtained during the panoramic initialization phase). As the execution map progresses, the system acquires a set of observational evidence. (Such as bounding box location, attribute recognition results, VLM text description, etc.). Based on Bayes' theorem, the system updates the posterior confidence of the hypothesis in real time: .

[0127] The agent continuously monitors the changing trend of the posterior probability $P(H|E_t)$. We define two key control triggers:

[0128] Confidence Collapse: If The descent gradient exceeds the threshold within k consecutive frames. (For example, if the target enters a blind spot and tracking is lost), the system determines that the current execution path is invalid.

[0129] High Ambiguity Spike: If a mutually exclusive hypothesis H exists... a With H b , making (For example, if it is impossible to distinguish between "transferring items" and "stealing"), the system determines that the current evidence is insufficient.

[0130] Once the above conditions are triggered, the agent will activate the Adaptive Replanning module. The system will pause the current inefficient pipeline and dynamically insert new exploratory nodes into the task graph G. For example, for a "sudden drop in confidence," the agent might insert a "Temporal Backtracking" node to call the historical trajectory database for target re-identification; for "high ambiguity," the agent might insert a "Multi-view Fusion" node or generate a higher-resolution prompt to request detailed features from the MLLM. This mechanism enables the system to proactively seek new clues when evidence is insufficient, just like a human expert, thereby achieving robust semantic analysis. Figure 2 The diagram shown is a schematic representation of the system execution flow of a multimodal cognitive agent video semantic analysis method provided in an embodiment of this application.

[0131] To better illustrate the validity of this application, specific embodiments are provided below:

[0132] 1. Experiment setup and dataset

[0133] (1) Dataset Construction: Given that existing public datasets (such as UCF-Crime) mainly focus on short video clip classification and lack annotations for long-term logical reasoning, we constructed a large-scale long-term video dataset called "CityGuard-Long". This dataset contains 10 hours of high-definition surveillance video, covering six typical scenarios such as squares, shopping malls, and parking lots. The dataset not only annotates atomic actions (such as running and jumping), but also includes complex events (such as non-motorized vehicle theft, gang fights, and missing children) annotated by professional security personnel, along with their complete temporal evidence chains and textual descriptions.

[0134] (2) Baselines: We selected three representative methods as benchmarks:

[0135] High-Throughput Tool Pipeline: This pipeline uses only edge-side high-throughput sensing tools to complete detection / tracking / attribute extraction and rule-based alarms, representing the mainstream industrial baseline of "high throughput but weak semantics".

[0136] End-to-End MLLM: This approach uses a large multimodal model in the cloud to perform stage-by-stage segmented inference on the entire video (which requires segmented processing due to context window limitations). It represents a pure large model approach that is "highly semantic but computationally expensive".

[0137] Naive-Cloud: Using simple motion / threshold triggering as a traffic diversion strategy, the trigger fragments are uploaded to the cloud for large-scale model analysis, serving as a control group for end-to-cloud collaboration that lacks cognitive orchestration.

[0138] (3) Evaluation Indicators: To comprehensively evaluate system performance, we adopt a multi-dimensional indicator system:

[0139] Semantic Accuracy: The event detection accuracy is evaluated using mAP (mean precision), and the CIDEr score is introduced to evaluate the quality of the generated text description.

[0140] Computational Efficiency: The average token consumption and GPU inference latency required to process a unit of video duration.

[0141] Evidence Completeness: Evaluate the temporal overlap (IoU-Time) between the keyframe sequence output by the system and the manually annotated evidence chain.

[0142] 2. Comparative analysis of experimental results

[0143] To verify the comprehensive performance of our framework under heterogeneous computing power collaboration, we rigorously compared our proposed method (named Agent-ECA, Agent-based Edge-Cloud Adaptive Framework) with the three benchmark methods mentioned above on the CityGuard-Long dataset. At the implementation level, our prototype uses qwen3-vl-plus in the cloud as the cognitive agent model, PP series perception tools on the edge for high-throughput detection and tracking, and qwen3-8b-vl as the tool-based video clip / keyframe semantic parsing component. It is important to emphasize that the above models are only used to reproduce the experiments; the methodological implementation remains replaceable.

[0144] Experimental results demonstrate that Agent-ECA exhibits significant advantages in handling complex, long-term temporal events. In terms of semantic accuracy, Agent-ECA achieves a mAP of 78.4%, significantly outperforming the Static-CNN Pipeline's 42.1%. This is because traditional pipelines cannot understand complex temporal interaction semantics such as "theft," and can only detect isolated objects. Compared to End-to-End MLLM (mAP 81.2%), our method suffers only a less than 3% accuracy loss, primarily due to the possibility that extreme end-side filtering strategies might miss a few very weak clues. However, this small accuracy cost is compensated by an exponential improvement in computational performance.

[0145] In terms of inference latency and resource consumption, End-to-End MLLM takes an average of 45 minutes to process one hour of video, and the GPU memory is constantly at full capacity, failing to meet real-time requirements. In contrast, Agent-ECA takes only 4.2 minutes on average, achieving a speedup of nearly 15 times. More importantly, through the filtering of the semantic density decision mechanism, Agent-ECA reduces the token consumption of large cloud models by 92.6%. This means that in actual deployment, the number of monitoring channels that a single cloud server can support simultaneously increases by an order of magnitude, greatly reducing the total cost of ownership (TCO) of the system. In addition, although the Naive-Cloud strategy also reduces cloud load, due to the lack of logical orchestration of agents, its false alarm rate is high, leading to frequent invalid uploads, and its overall performance is not as good as the adaptive collaborative solution proposed in this paper.

[0146] 3. Ablation test

[0147] To further explore the contribution of each core module of the system, we designed an ablation experiment to gradually remove key components and observe performance changes.

[0148] First, removing the CoT orchestration module degenerates the system into a weakly structured keyword-triggered mode, with the overall accuracy of complex compound instructions decreasing by 46%, indicating that explicit deconstruction of the "intent-task graph" is a key prerequisite for long-term semantic reasoning. Second, removing the semantic density decision mechanism renders the system unable to distinguish between high-value evidence and redundant segments, leading to the near-full uploading of videos and triggering long-sequence reasoning by a large cloud model. This results in significant computational resource consumption and token overhead, with the system behavior degenerating into a pure large model directly recognizing the entire video, contradicting the core concept of "on-demand in-depth analysis" in this paper. Finally, removing the Bayesian closed-loop feedback mechanism leaves the system lacking posterior correction and replanning capabilities for newly added information and evidence chains, continuing to execute predetermined tasks based on early assumptions. This causes a 76% drop in accuracy on occlusion, ambiguity, and scene drift samples. These results demonstrate that CoT orchestration, semantic density splitting, and Bayesian closed-loop mechanisms together constitute necessary conditions for the effectiveness of the method.

[0149] 4. Qualitative Analysis and Visualization Cases

[0150] To visually demonstrate the interpretability of the system, we selected a typical case of "non-motorized vehicle theft" for analysis. End-to-end multimodal large model solutions often only provide event labels or brief conclusions, making it difficult to provide a traceable chain of evidence; while the method presented in this paper can generate a complete chain of reasoning evidence:

[0151] Initial perception (T=0s): The device detects that "person A" is close to "bicycle B", triggering IoU interaction detection. If the semantic density exceeds the threshold, the system wakes up the cloud cognitive agent.

[0152] Intent analysis (T=5s): The cloud-based agent observes that person A lingers next to the car for more than a set threshold, and that key body points show a "crouching" posture, which is initially inferred to be "abnormal attention".

[0153] Ambiguity resolution (T=15s): Due to the blurry image, the agent cannot determine whether it is "car repair" or "lock picking". At this time, the Bayesian confidence score is lower than the threshold, and the system automatically triggers replanning, calling the "local zoom" and "temporal correlation" tools.

[0154] Evidence locking (T=18s): The agent combines the continuity of actions in consecutive frames and detects the characteristics of "sabotage tools," ultimately determining it as "theft," and automatically captures a clear frame of the face as evidence. This process not only accurately identifies the behavior but also provides the decision-making logic for each step, greatly improving the credibility of security assessment and the efficiency of subsequent evidence collection.

[0155] The beneficial effects of this application include:

[0156] 1. Reduced computational resource consumption. In the technical solution of this application, the computational resource consumption of each logical node is analyzed, and the validity of video frames is analyzed. Then, video frames that obviously do not contain valid information are removed. In the specific processing, it can be ensured that the video frames analyzed have complete information, and the results can be achieved based on these video frames. Since irrelevant information is removed in this process, the computational resource consumption is significantly reduced.

[0157] 2. Improved timeliness of analysis results. The technical solution of this application, on the one hand, rationally filters video frames, and on the other hand, judges the validity of information in each video frame. Based on the judgment results, feature uploading and analysis are performed, or information analysis is conducted on the entire video segment. This enables the filtering and adjustment of information carried within the video stream, thereby significantly improving the timeliness of the analysis work.

[0158] 3. Improved flexibility of the analysis link. In the technical solution of this application, by analyzing the development of confidence of each node in the obtained logical link, it is possible to analyze whether there are any technical defects in the currently adopted logical link, thereby selecting the most reasonable technical link to ensure that the analysis work can be adjusted at any time according to the needs during the final analysis process.

[0159] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to computer program instructions. The aforementioned computer program can be stored in a non-volatile storage medium, and when executed, it performs the steps of the above method embodiments. Alternatively, if the integrated unit of the present invention is implemented as a software functional module and sold or used as an independent product, it can also be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a non-volatile storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, network device, etc.) to execute all or part of the methods described in the various embodiments of the present invention.

[0160] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A video semantic analysis method for multimodal cognitive intelligent agents, characterized in that, The analytical method includes: Obtain natural language instructions and the constraints contained in the natural language instructions to obtain the deconstructed intent; Based on the deconstructed intent, multiple keyframes are acquired from the video stream to generate a cognitive benchmark for semantic analysis. Based on the thought chain reasoning mechanism and the aforementioned cognitive benchmark, the dependency relationships between deconstructed intentions are established to obtain a task graph; Based on the task graph, the tasks allocated within each heterogeneous computing node are obtained, and the task graph is advanced. Based on the observation evidence set continuously acquired during the knowledge graph-based advancement process, the posterior confidence of subsequent events is updated in real time, and logical links are adjusted accordingly.

2. The multimodal cognitive intelligent agent video semantic analysis method according to claim 1, characterized in that, The process of obtaining natural language instructions and the constraints contained in those instructions to obtain the deconstructed intent includes: Natural language instructions are acquired and analyzed by a large language model; When analyzing natural language instructions using a large language model, the visual tasks, logical dependencies, and semantic information flow content contained in the natural language are obtained to arrive at the deconstructed intent.

3. The multimodal cognitive intelligent agent video semantic analysis method according to claim 1, characterized in that, Based on the deconstructed intent, the process of acquiring multiple keyframes from the video stream to generate a cognitive benchmark for semantic analysis includes: Based on the deconstructed intent, the scene to be identified from the video stream is obtained; Sampling is performed at a sparse sampling rate to extract several keyframes from the video stream; Intra-image information analysis is performed on the aforementioned keyframes, and a scene context description is generated; Based on the scene context description, the constraint weights within the video stream are adjusted to obtain prior knowledge of the video stream and to acquire an initialization mechanism for environmental information. The initialization mechanism of the environmental information is associated with natural language information to generate the cognitive basis for semantic analysis.

4. The multimodal cognitive agent video semantic analysis method according to claim 1, characterized in that, The process of establishing dependencies between deconstructed intentions based on the thought chain reasoning mechanism and the cognitive benchmark to obtain a task graph includes: The visual tasks in the deconstructed intent are obtained, including object recognition tasks, trajectory extraction tasks, and behavior classification tasks. Obtain the logical dependencies and semantic information flow from the deconstructed intent; The visual task, logical dependencies, and semantic information flow are determined to obtain a task graph, which is represented by the following equation: ， Where G represents a directed acyclic graph of the task graph; N represents a set of atomic subtasks; and E represents logical dependencies and triggering relationships. The resource and information gain relationships of all nodes in the task graph are obtained to determine the optimal scheduling strategy for task graph construction.

5. The multimodal cognitive intelligent agent video semantic analysis method according to claim 4, characterized in that, The process of acquiring the resource and information gain relationships of all nodes in the task graph to obtain the optimal scheduling strategy for task graph construction includes: The resource consumption and information gain generated in each node are obtained, and the optimal scheduling policy equation is used to select a policy. The optimal scheduling policy equation is as follows: ， in, Indicates the optimal scheduling strategy; This represents the computational cost of the p-th subtask under strategy π. Let π represent the expected information gain from the p-th subtask under strategy π. and These represent tradeoff coefficients, used to adjust the emphasis on computational cost and expected information gain in the scheduling strategy; Based on the calculation results of the optimal scheduling strategy equation, the most suitable tool for each logical node is obtained from the tool set {T}, and the interpretable invocation parameters for each most suitable tool are configured. The toolset {T} includes side-end tools and cloud-based tools.

6. The multimodal cognitive agent video semantic analysis method according to claim 1, characterized in that, The step of obtaining the tasks allocated within each heterogeneous computing node based on the task graph and advancing the task graph includes: Based on the task graph, video segments are obtained from the video stream; Obtain the semantic density of all the video segments to quantify the information entropy value contained in the set of video frames corresponding to the video segments. The semantic density equation is as follows: ， Where SemDensity(x) represents semantic density; Represents the motion saliency entropy value of video segment x based on optical flow; The Intersection over Union (IoU) ratio is used to represent the degree of geometric overlap between objects within a video clip. and Indicates the adaptive weighting coefficients; x represents the video segment; and To represent different things within a video clip; Based on the semantic density, the tasks allocated within each heterogeneous computing node are obtained, and the task graph is advanced.

7. The multimodal cognitive agent video semantic analysis method according to claim 6, characterized in that, The step of obtaining the tasks allocated within each heterogeneous computing node based on the semantic density and advancing the task graph includes: Set a threshold for judging task allocation on heterogeneous computing nodes to compare the types of tasks allocated within heterogeneous computing nodes; The semantic density and the task allocation judgment threshold are compared to obtain the tasks allocated within each heterogeneous computing node. The task allocation judgment equation is as follows: ， in, This indicates the act of uploading to the cloud; Indicates the processing behavior on the endpoint; Represents semantic similarity, used to represent the cosine similarity between the target feature vector and the user's natural language intent vector; Represents the target feature vector; Represents the user's natural language intent vector; This represents the uncertainty threshold, used to determine whether to force the video segment to be uploaded to the cloud; This represents the cloud-based threshold, used to determine whether a video segment corresponding to semantic density is a high-value segment.

8. The multimodal cognitive intelligent agent video semantic analysis method according to claim 1, characterized in that, The knowledge graph-based advancement process continuously acquires a set of observational evidence, updates the posterior confidence of subsequent events in real time, and adjusts the logical links, including: Based on the knowledge graph-based advancement process, observational evidence is acquired in real time, and an observational dataset is generated, which is represented as follows: ， Among them, E -t Represents the set of observational evidence at time t; e1~e k This represents the observation data; k represents the observation data label; Obtain the correspondence between the probability of event occurrence and the time of occurrence within the observed dataset, and obtain the posterior confidence score. The equation for determining the posterior confidence score is: ， in, Indicates event H i Likelihood within the set of observational evidence acquired at time t; Indicates event H i The likelihood of the event; i and j represent the sequence indices of the event; m represents the total number of sequence indices of the event; Indicates event H j The probability of occurrence; H i H represents the hypothesis of the i-th event; j P(H) represents the hypothesis of the j-th event; i ) represents event H i The prior probability; Represents time H j likelihood; E t This represents the observation dataset acquired at time t; The logical links are adjusted based on the changing trend of the posterior probability.

9. The multimodal cognitive intelligent agent video semantic analysis method according to claim 8, characterized in that, The adjustment of the logical link based on the changing trend of the posterior probability includes: Set a logic link judgment threshold to control the logic link adjustment trigger; when When the descent gradient over a consecutive video frames is not lower than the confidence threshold When the current logical link fails, it is determined that the current logical link has failed, and other logical links are explored and selected. When the same feature exists within the current logical link, there is a mutual exclusion assumption H. a and H b , making If the evidence for adjusting the current logical link is insufficient, further analysis of the current logical link is required. This represents the likelihood of a mutually exclusive hypothesis across all observed datasets. This represents the likelihood of another mutually exclusive hypothesis across all observed datasets; This indicates that the logical link adjusts the evidence judgment threshold.

Images