Multimodal interactive video analysis method and device, computer device and storage medium

Abstract

The application relates to the technical field of unmanned aerial vehicle video analysis, and discloses a multimodal interactive video analysis method and device, computer equipment and a storage medium. Four modes of inference layers of detection box labeling, natural language question and answer, pointing and questioning and scene description are superimposed on the collected frames of the unmanned aerial vehicle video stream, the technical problems of single interactive mode and rigid operation are solved, the user can flexibly switch the interactive mode according to the task scene and the immediate intention, the analysis process is more intuitive and efficient, the learning and operation costs are significantly reduced, the differentiated deep analysis requirements of different professional roles are met, and the intelligent level and the actual application efficiency of the unmanned aerial vehicle video analysis are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) video analysis technology, and more particularly to multimodal interactive video analysis methods, devices, computer equipment, and storage media. Background Technology

[0002] Currently, drone video analysis systems face numerous technical bottlenecks, limiting their effectiveness in complex scenarios.

[0003] For example, patent CN113221706B discloses a multi-process video stream AI analysis system. This system improves video analysis efficiency through multi-process parallel processing, but it has the following limitations: First, the system only supports a single video stream analysis mode and lacks multimodal interaction capabilities. Users cannot interact with the system through intuitive methods such as natural language or pointing.

[0004] For example, patent CN117711001B discloses a multimodal image processing technology that can process data in both image and text modes, but it still has shortcomings in UAV video analysis scenarios: the technology provides limited interaction methods and cannot meet the diverse interaction needs of users in complex task scenarios.

[0005] Therefore, such systems typically only support a single, fixed interaction mode, preventing users from flexibly choosing multiple interaction methods based on task scenarios and instantaneous intentions. This limitation of the interaction mode results in the system being less intuitive and less efficient when facing complex and ever-changing real-world tasks, and it is also difficult to meet the differentiated operating habits and in-depth analysis needs of users with different professional backgrounds (such as inspectors and commanders).

[0006] Therefore, there is an urgent need for a drone video analytics solution that supports multimodal interaction and improves the effectiveness of practical applications. Summary of the Invention

[0007] This invention provides a multimodal interactive video analysis method, apparatus, computer equipment, and storage medium to solve the technical problem of poor operational flexibility caused by the single and consistent interaction mode of existing UAV video analysis systems.

[0008] Firstly, a multimodal interactive video analysis method is provided, including:

[0009] Access the drone video stream and automatically sample it at preset time intervals to obtain a sequence of sampled frames;

[0010] An inference layer corresponding to different interaction modes is created on the sampling frame, and the interaction information input by the user based on the interaction mode is received; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, pointing local question mode, and scene description mode;

[0011] Based on the prompts in the interactive information, adaptive semantic enhancement is performed to generate a set of synonyms and update the dynamic target library.

[0012] Based on the user-selected current interaction mode and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result;

[0013] Configure a natural language question trigger to perform semantic matching on the reasoning results, and generate and execute an automatic question-answering chain alarm when the matching degree meets the preset triggering conditions.

[0014] Secondly, a multimodal interactive video analysis device is provided, including:

[0015] The video access module is used to access the drone video stream and automatically sample it at preset time intervals to obtain a sequence of sampled frames.

[0016] An interaction module is used to create inference layers corresponding to different interaction modes on the sampling frame and to receive interaction information input by the user based on the interaction mode; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, pointing to local question mode, and scene description mode;

[0017] The enhancement module is used to perform adaptive semantic enhancement based on the prompts in the interactive information, generate a set of synonyms, and update the dynamic target library;

[0018] The inference module is used to perform multimodal video inference on the sampled frame sequence based on the current interaction mode selected by the user and the dynamic target library, and obtain the inference result;

[0019] The alarm module is used to configure natural language question triggers, perform semantic matching on the reasoning results, and generate and execute automatic question-answering chain alarms when the matching degree meets the preset trigger conditions.

[0020] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described multimodal interactive video analysis method.

[0021] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the aforementioned multimodal interactive video analysis method.

[0022] The beneficial effects of this invention compared to existing technologies are as follows: By overlaying four modes of inference layers—detection box annotation, natural language question answering, pointing and questioning, and scene description—on the captured frames of the UAV video stream, this invention solves the technical problems of single interaction methods and rigid operation. This allows users to flexibly switch interaction modes according to task scenarios and immediate intentions, and makes the analysis process more intuitive and efficient. It significantly reduces learning and operation costs and meets the differentiated in-depth analysis needs of different professional roles, thereby greatly improving the intelligence level and practical application effectiveness of UAV video analysis.

[0023] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other objects, features and advantages of the present invention more obvious and understandable, preferred embodiments are described in detail below. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a multimodal interactive video analysis method according to an embodiment of the present invention;

[0025] Figure 2 yes Figure 1 A schematic diagram of a specific implementation method for step S20;

[0026] Figure 3 yes Figure 1 A schematic diagram of a specific implementation method for step S30;

[0027] Figure 4 yes Figure 1 A schematic diagram of a specific implementation method for step S50;

[0028] Figure 5 yes Figure 1 A schematic diagram of a specific implementation method for step S60;

[0029] Figure 6 yes Figure 1 A schematic diagram of a specific implementation of step S70;

[0030] Figure 7 This is a schematic diagram of a multimodal interactive video analysis device according to an embodiment of the present invention;

[0031] Figure 8 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0032] Figure 9 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The technical solutions in the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] It should be understood that, when used in this specification and the appended claims, the terms “comprising” and “including” indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0035] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0036] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0037] Please see Figure 1 As shown, Figure 1 This is a schematic flowchart illustrating a multimodal interactive video analysis method provided in an embodiment of the present invention. The multimodal interactive video analysis method includes the following steps:

[0038] S10: Access the drone video stream and automatically sample it according to a preset time interval to obtain a sampled frame sequence.

[0039] In this embodiment, the video analysis system receives real-time video streams from drones via its video access module. This video access module is responsible for receiving and automatically sampling the drone video streams. Its hardware configuration includes a video capture card, a network interface, and a storage device. The video capture card supports multiple video inputs and a resolution of 4K@60fps; the network interface uses a gigabit Ethernet interface to support high-speed video stream transmission; and the storage device uses a high-speed SSD for video caching with a capacity of at least 1TB. The video streams can be transmitted using mainstream streaming media protocols, including but not limited to RTMP, RTSP, and WebRTC. Furthermore, the video access module has protocol adaptive capabilities, automatically recognizing and adapting to the video stream formats transmitted by drones of different brands and models, ensuring broad compatibility.

[0040] In practice, the automatic sampling process is executed according to a preset time interval parameter, where the interval parameter `stepSeconds` can be configured between 0.1 seconds and 10 seconds, with a preferred value of 1 second. During sampling, the system dynamically adjusts the sampling frequency by comprehensively considering multiple factors, including but not limited to: the current video frame rate, available network bandwidth, the system's real-time processing capabilities, and the critical requirements of the specific task. For example, for high frame rate video input, the sampling interval can be appropriately increased to avoid data overload; while for scenarios requiring detailed analysis, the interval can be decreased to ensure that all important moments are captured.

[0041] To ensure the timing accuracy of subsequent analysis, the sampling process maintains strict timing synchronization. Specifically, each sampled video frame is assigned a timestamp accurate to the millisecond level. This timestamp is generated using a unified time base and is synchronized with the timestamp on the drone as much as possible. For simultaneous input of multiple video streams, the system maintains a unified timeline to ensure that video frames from different drones or camera positions are precisely aligned in the time dimension, enabling cross-viewpoint or cross-time correlation analysis.

[0042] After the sampled frames are acquired, they enter a preprocessing stage. This stage includes image decoding (restoring the compressed video data to a pixel matrix), resolution adjustment (possibly using intelligent interpolation algorithms to ensure image quality in key areas while controlling the amount of data), and color space conversion (e.g., converting common YUV format to RGB format, or vice versa). It supports processing multiple color spaces and common resolutions such as 720p and 1080p, and provides corresponding preset parameter templates to simplify the configuration process.

[0043] Furthermore, there are two main sampling modes for the acquired frames: real-time sampling mode and offline sampling mode. In real-time mode, the system continuously receives and processes real-time video streams from the drone for immediate analysis. In offline mode, the system can read data from pre-stored video files and perform subsequent processing. Both modes employ a unified core processing flow, ensuring consistency of analysis results across different operating modes and enhancing the system's flexibility.

[0044] S20: Create an inference layer corresponding to different interaction modes on the sampling frame, and receive interaction information input by the user based on the interaction mode; wherein, the interaction mode includes detection box annotation mode, natural language question answering mode, pointing local question mode and scene description mode.

[0045] It is understandable that one inference layer corresponds to one interaction mode. The inference layer uses implicit frame overlay canvas technology to overlay on the original sampling frames, presenting a transparent style, and is used to process interactive operations and provide feedback on inference results. In this embodiment, the inference function is implemented by the system's interaction module. This module uses a lightweight multimodal large model as a unified algorithm foundation to achieve model normalization for detection, question answering, and description. Its architecture design is divided into the following four layers:

[0046] Inference Engine Layer: Deploys multimodal large models based on a deep learning inference framework, supporting 4-bit / 8-bit quantization loading to improve FPS;

[0047] Prompt engineering layer: Converts user interactions (clicks, selection boxes, questions) into structured prompt word templates that the model can understand;

[0048] The scheduling and management layer manages task priorities, uses the same visual encoder features for detection and question answering, reduces redundant calculations and lowers latency;

[0049] Result decoder: Parses the text coordinate tokens output by the large model into standard pixel coordinates or structured answers.

[0050] The detection box annotation mode allows users to directly select targets on the enhanced display screen using the mouse, or receive detection boxes intelligently generated by the system, thereby achieving intuitive target location and annotation. Specifically, the detection box annotation mode supports the following functions:

[0051] Selection box interaction: Capture user mouse events and draw a rectangle in real time;

[0052] Intelligent adsorption: Based on edge detection algorithms, it automatically adjusts the boundaries of the detection box to improve annotation accuracy;

[0053] Frame editing: Supports dragging and dropping to adjust the size and position of the frame, and supports operations such as copying, deleting, and modifying the frame;

[0054] Batch operations: Supports applying the current detection box to adjacent frames or the entire video segment.

[0055] The Natural Language Question Answering (NQA) mode allows users to directly input natural language questions (such as "What is the status of the wheel in the lower left corner of the screen?") on the interface while watching videos. The system's embedded Visual Question Answering (VQA) capability will provide an answer in real time based on the visuals. Specifically, the NQA mode supports the following functions:

[0056] Text parsing: Using natural language processing techniques to parse user questions and extract keywords and semantic information;

[0057] Content retrieval: Searches for relevant content in sampled frames based on questions, supporting target retrieval, scene understanding, event detection, etc.

[0058] Answer generation: Generates natural language responses by combining visual content, supporting both simple answer and detailed explanation modes.

[0059] The "Point-to-Local Question" mode allows users to directly click on a specific device or area within the video frame using the mouse. Based on precise coordinate transformation and local feature extraction, the system generates a targeted analysis report or answer for that designated area. Specifically, the "Point-to-Local Question" mode supports the following functions:

[0060] Coordinate capture: Precisely captures the screen coordinates and corresponding video coordinates of the user's pointing gesture;

[0061] Region extraction: Extract the region of interest centered on the pointed location;

[0062] Feature analysis: Deep feature extraction is performed on the Region of Interest (ROI), including visual features such as color, texture, and shape;

[0063] Local understanding: Answers user questions by combining regional features, supporting target recognition, state analysis, anomaly detection, etc.

[0064] The scene description mode can perform global scene understanding of the entire sample frame as needed, and generate a structured text description containing a target list, spatial layout, and overall status, providing users with macro-level situational awareness. Specifically, the scene description mode supports the following functions:

[0065] Global analysis: Perform scene analysis on the entire sampled frame to identify the main targets and environmental information;

[0066] Relationship extraction: Analyzing the spatial and interactive relationships between targets;

[0067] Scene classification: Classify scenes into predefined scene types;

[0068] Description generation: Generates a structured scene description, including a list of targets, spatial layout, environmental status, and other information.

[0069] In some embodiments of the present invention, such as Figure 2 As shown, a specific inference layer creation scheme is provided. In S20, an inference layer corresponding to different interaction modes is created on the sampling frame, and the user inputs interaction information based on the interaction mode. Specifically, it includes the following steps S21-S26.

[0070] S21: Construct corresponding inference tasks for each interaction mode on the sampling frame.

[0071] For step S21, the system will instantiate a specific inference task in the background for the current interaction mode selected by the user (for example, the user draws a detection box on the screen with the mouse, which means the detection box annotation mode has been selected). The inference tasks corresponding to each mode have different internal logic, computational requirements and output formats, but they are all uniformly incorporated into the system's task management framework for scheduling.

[0072] S22: Create an independent canvas context for the inference task, associate the canvas context with the frame identifier of the sampled frame sequence, and output a canvas context containing task attributes and frame association relationships.

[0073] For step S22, a virtual, independent workspace, namely a canvas context, is provided for each inference task. The canvas context not only contains basic information about the inference task, such as task ID, creation time, and task type, but more importantly, it is precisely associated with a specific sequence of sampled frames. For example, the system records information such as the video stream ID currently being processed by the task, and the timestamps of the corresponding start and end frames. This association ensures that all subsequent inference operations are based on clear spatiotemporal coordinates.

[0074] S23: Based on the canvas context, configure the corresponding inference layer for the interaction mode, and use the canvas context as the execution environment of the inference layer to output the instantiated inference layer.

[0075] For step S23, above the soft canvas (i.e., the canvas context), the system configures a corresponding hard canvas for each interaction mode, which is an inference layer in the implicit frame overlay canvas. This is a logical processing layer that is invisible to the user, but it carries the core visual understanding and inference logic. Computational resources such as multimodal large models and visual encoders all run on this layer.

[0076] Among them, the Implicit Frame Overlay Canvas is a technology that uses virtual overlay to dynamically create multiple transparent and parallel inference layers on top of the original video frames, thereby enabling a single video frame to support multiple interaction modes and corresponding analysis modes.

[0077] S24: For the instantiated inference layer, associate the timestamp and attitude information of the sampling frame, and output the inference layer with timing synchronization information.

[0078] For step S24, to ensure the accuracy and spatiotemporal consistency of the inference results, each inference layer must not only know which frame it is in, but also when and from what angle it was captured. In addition to the video timestamp associated in S22, the system also associates the attitude information recorded by the UAV (such as GPS position, flight altitude, camera pitch angle, and yaw angle) with the corresponding frame and inference layer, providing crucial metadata for subsequent analysis, especially for spatial relationship judgment, coordinate transformation, and viewpoint calibration.

[0079] S25: Allocate independent computing resources to the inference layer with timing synchronization information, schedule the execution of the inference task through a priority queue, and output the inference layer with completed resource scheduling.

[0080] In this embodiment, the system architecture design supports resource isolation and mutual exclusion scheduling for concurrent multi-task operations. Specifically, the system allocates independent processor, memory, or GPU computing resources to different types of interaction modes (or inference layers) to prevent performance jitter or latency caused by resource contention between different tasks. Simultaneously, a central task scheduler manages a priority queue. For example, an emergency alarm task triggered by a trigger has a higher priority than a user-triggered, non-real-time scene description task. This scheduling mechanism ensures that high-priority tasks can be responded to quickly.

[0081] S26: Execute the inference task in the inference layer after the resource scheduling is completed, and overlay and render the intermediate and final inference results onto the original sampling frame in real time to generate and present an enhanced display screen to the user.

[0082] Understandably, in the resource-secure inference layer, specific multimodal analysis and inference are performed on video frames. The results of the inference (such as object detection boxes in images, visual evidence regions used to answer text, etc.) are seamlessly overlaid on the original video screen that the user is watching as a transparent layer through a real-time rendering engine. This technology is called augmented reality display or Simply Augmented Display. It allows users to intuitively obtain the AI's analysis conclusions while maintaining a complete perception of the original video screen, greatly improving interaction efficiency and user experience.

[0083] S30: Based on the prompts in the interactive information, perform adaptive semantic enhancement, generate a set of synonyms, and update the dynamic target library.

[0084] For step S30, by expanding the single, fixed prompt word (such as "insulator") into a rich semantic network (such as insulator, bushing, disconnector, etc.), the subsequent video reasoning steps can understand the video content from a broader semantic perspective, thereby significantly improving the system's target detection recall rate, question answering accuracy, and enhancing the system's adaptability to diverse user language expressions.

[0085] The semantic enhancement function in step S30 is implemented by the enhancement module of this system. Key technologies include using multilingual-BERT as the word embedding model and using a graph database to store the dynamic target database relationships. An example of the dynamic target database data structure is as follows:

[0086] {

[0087] "target_id": "vehicle",

[0088] "chinese_term": "vehicles",

[0089] "english_terms": ["vehicle", "car", "truck", "automobile"],

[0090] "synonyms": {

[0091] "zh": ["automobile", "sedan", "truck", "motor vehicle"],

[0092] "en": ["auto", "motor vehicle", "transportation"]

[0093] },

[0094] "attributes": ["color", "type", "brand", "license_plate"],

[0095] "weight": 0.95,

[0096] "last_updated": "2024-01-15T10:00:00Z"

[0097] }

[0098] Before implementation, the dynamic target library can be pre-trained. During or after each inference task, the dynamic target library is updated through the synonym set so that it can continuously learn and meet the user's application needs in various complex scenarios.

[0099] In some embodiments of the present invention, such as Figure 3 As shown, a specific semantic enhancement scheme is provided. In S30, adaptive semantic enhancement is performed based on the prompt words of the interaction information to generate a set of synonyms and update the dynamic target library. Specifically, it includes the following steps S31-S34.

[0100] S31: Extract at least one Chinese prompt word from the interactive information.

[0101] Understandably, the sources of prompts are diverse and depend on the current interaction mode. For example, in bounding box annotation mode, prompts might be labels assigned to the target by the user, such as "insulator"; in natural language question-and-answer or point-and-answer mode, prompts might be key entities or nouns extracted from the user's questions and instructions, such as "tower crane" and "crane arm" extracted from "Is the boom angle of this tower crane normal?"; in scene description mode, prompts might be keywords for scene classification, such as "construction site".

[0102] It is also understandable that the interactive information is natural language text, such as "What is the device status in the lower left corner?". In specific implementation, for step S31, the obtained natural language text will first be preprocessed, including cleaning irrelevant characters, performing Chinese word segmentation, part-of-speech tagging, and named entity recognition, so as to accurately extract the core prompt words to be analyzed.

[0103] S32: Input the Chinese prompt word into a pre-trained word embedding model and calculate the semantic similarity between the Chinese prompt word and multiple candidate words in the candidate word library.

[0104] In this embodiment, the system employs a powerful pre-trained multilingual word embedding model as the semantic computation engine, such as BERT or its variants. This model can transform words or phrases in text into vectors in a high-dimensional space, i.e., word vectors. In this vector space, words with similar semantics have closer vector distances (such as cosine similarity). When a user inputs a suggestion word, the system calculates its vector similarity to each word in a large candidate vocabulary (such as a domain terminology database or a general corpus). The process of calculating word embeddings and semantic similarity is the core of adaptive semantic enhancement. It does not rely on a fixed dictionary but utilizes the deep learning model's understanding of the deep semantics of language for matching.

[0105] S33: Based on the calculated semantic similarity, candidate words with semantic similarity higher than a preset threshold are selected to generate a set of synonyms for the Chinese prompt words.

[0106] In this embodiment, a configurable similarity threshold is pre-set. All candidate words whose semantic similarity to the original prompt word exceeds this threshold are considered potential synonyms or semantically related words and are collected into a set. For example, the set of synonyms for "vehicle" may include "car," "sedan," "truck," "motor vehicle," "transportation vehicle," etc. Setting a threshold can filter out a large number of irrelevant words, ensuring the purity and effectiveness of the set. Expanding the synonym set based on the similarity calculation results can ensure the completeness and diversity of semantics.

[0107] S34: Update the dynamic target library using the generated set of synonyms.

[0108] In this embodiment, the Dynamic Target Library is stored and maintained using a graph data structure. Nodes in the graph represent target concepts (such as "vehicle"), and edges represent semantic relationships between concepts (such as synonyms, hierarchical relationships, component relationships, etc.). When a new set of synonyms is generated, the system merges it into the library according to predefined rules. For example, "sedan" and "truck" are used as synonym nodes for "vehicle," and an association is established. This library is dynamic because it is continuously updated and expanded as the system is used (user input, batch test feedback), thereby constantly learning new semantic knowledge, improving the system's ability to recognize different expressions, and maintaining the version history of the Dynamic Target Library for rollback and difference comparison. Understandably, the dynamic update mechanism includes adding new words, adjusting the weights or associations of existing words, and even removing unused low-frequency words.

[0109] S40: Based on the current interaction mode selected by the user and the dynamic target library, perform multimodal video inference on the sampled frame sequence to obtain the inference result.

[0110] It is understood that this embodiment is based on a unified underlying multimodal large model based on the Transformer architecture, and provides different specific inference schemes for different interaction modes. The implementation process of the four modes is described in detail below. It is important to emphasize that these interaction modes share the same visual encoder and multimodal fusion model base, which realizes model normalization, thereby significantly reducing redundant calculations and reducing system latency.

[0111] In some embodiments of the present invention, a specific inference scheme is provided when the current interaction mode is the detection box annotation mode. In S40, that is, based on the current interaction mode selected by the user and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result, specifically including the following steps:

[0112] The sampled frames are encoded into visual feature vectors, and the target descriptions in the dynamic target library are used as text prompts. Both are input into the multimodal large model to perform cross-modal alignment.

[0113] Based on the aligned cross-modal features, the normalized bounding box coordinates corresponding to the target description text are output through a multimodal large model;

[0114] The text sequence containing the bounding box coordinates in the output of the multimodal large model is parsed, or the cross-modal attention weights generated by the multimodal large model in the process of generating the bounding box coordinates are analyzed to obtain the prediction confidence of each bounding box coordinate.

[0115] Filter out low-confidence detection results below a preset threshold and obtain high-confidence bounding box coordinates as inference results.

[0116] In practice, firstly, the sampled frame (e.g., frame_00123) is input into the visual encoder of the multimodal large model and encoded into a set of dense visual feature vectors, Visual Tokens. Simultaneously, target description text obtained from a dynamic target library (e.g., "detecting insulators, bushings, and disconnectors in the image" after step S30 expansion) is input as a text prompt. The model achieves cross-modal alignment by performing cross-attention computation on these visual and text tokens internally (typically built on a Transformer architecture), that is, semantically associating textual concepts with image regions.

[0117] Based on aligned cross-modal features, the model does not rely on predefined categories or additional bounding box regression heads. Instead, it directly outputs a sequence of bounding box coordinates normalized to the range [0, 1] corresponding to each target description in the prompt words, for example, 0.12, 0.45, 0.30, 0.65, where the four numbers represent relative coordinates [ymin, xmin, ymax, xmax]. This is a key manifestation of zero-shot / few-shot generalization ability. Relying on the semantic knowledge obtained from its massive pre-training and combined with synonym prompts from a dynamic target library, the model can detect specific category targets that it may not have seen during the training phase.

[0118] Since the model's output is a text sequence containing coordinates, or generates cross-modal attention weight maps during its internal computation, this embodiment designs a flexible confidence acquisition method. It can parse the confidence score accompanying the output text sequence, or analyze the cross-modal attention weights between the corresponding text prompt and the image patch when the model generates each coordinate, using these as the prediction confidence. Subsequently, the system sets a preset confidence threshold (e.g., 0.5) to filter all detection results, retaining the bounding box coordinates with high confidence as the final inference result.

[0119] The inference results are ultimately organized into structured JSON data. Below is a specific example of the data structure for the inference results:

[0120] {

[0121] "frame_id": "frame_00123",

[0122] "timestamp": "2024-01-15T10:30:45.123Z",

[0123] "mode": "detection",

[0124] "results": [

[0125] {

[0126] "object_id": "obj_001",

[0127] "class": "vehicle",

[0128] "confidence": 0.95,

[0129] "bbox": [100, 200, 150, 250],

[0130] "attributes": {

[0131] "color": "red",

[0132] "type": "truck",

[0133] "state": "moving"

[0134] }

[0135] }

[0136] ],

[0137] "metadata": {

[0138] "processing_time": 0.056,

[0139] "model_version": "v2.1.0"

[0140] }

[0141] }

[0142] In some embodiments of the present invention, a specific inference scheme is provided when the current interaction mode is a natural language question-and-answer mode. In S40, that is, based on the current interaction mode selected by the user and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result, specifically including the following steps:

[0143] The sampled frames are encoded into visual feature vectors, and the natural language questions in the user's interactive information are used as text prompts, which are then input into the multimodal large model.

[0144] By using the multimodal large model, combined with the global visual features of the sampled frames and the semantic information in the dynamic target library, the natural language problem is understood and reasoned.

[0145] Generate a textual response in structured or natural language form as the result of the inference.

[0146] Understandably, the natural language question answering model reuses the visual encoder from the bounding box annotation model to encode sampled frames into visual features. Simultaneously, the user's input interaction information, i.e., the natural language question (such as "What is the device status in the lower left corner?"), is input as an independent text prompt into the multimodal large model. The model also performs cross-modal alignment, but the purpose of this alignment is to understand what the question is asking and to associate the linguistic logic in the question (such as the spatial location "lower left corner") with the visual scene.

[0147] Then, through a Visual Question Answering (VQA) mechanism, the model combines encoded global visual features to understand the overall scene and combines local visual features to analyze the specific areas mentioned in the user's question. The model can understand relatively simple logical reasoning, such as "determine whether the device in the image is working." It does not simply retrieve keywords but performs genuine reasoning based on visual content.

[0148] Finally, based on the inference results, the model generates a text response in natural language. This response can be a concise one-sentence summary (such as "Equipment status is normal, no abnormalities found") or a detailed explanation generated according to task requirements. For example, for the question "Are there any abnormalities at the connection points of the insulator string?", a detailed explanation might be "A total of 15 insulators were detected in the string, and the metal fittings of the connector between the 3rd and 4th insulators have slight corrosion. Manual re-inspection is recommended."

[0149] To improve the smoothness of interaction, this embodiment adopts a variety of underlying technologies to optimize the reasoning of natural language question answering mode: (1) Caching mechanism: caching is established for high-frequency questions and their visual features, so that the same question can be responded to quickly next time; (2) Parallel computing: multiple question requests or detection requests are processed in batches at the same time, making full use of the parallel computing capabilities of hardware resources such as GPUs; (3) Incremental reasoning: due to the continuity of video frames, the visual features of adjacent frames are highly similar. The system can reuse some features of the previous frame and reuse the KV (Key-Value) cache in Transformer, thereby greatly accelerating the processing speed of subsequent frames.

[0150] In some embodiments of the present invention, a specific reasoning scheme is provided when the current interaction mode is a pointing-and-asking mode. In S40, that is, based on the current interaction mode selected by the user and the dynamic target library, multimodal video reasoning is performed on the sampled frame sequence to obtain the reasoning result, specifically including the following steps:

[0151] Obtain the coordinates of the user pointing at the screen in the interaction information, and convert the coordinates into image coordinates on the sampling frame;

[0152] Generate a region of interest based on the image coordinates;

[0153] Deep feature extraction is performed on the region of interest to obtain the visual features of the local region;

[0154] The region of interest is comprehensively analyzed by combining the visual features of the local region, the visual context information around the region of interest, and the semantic information in the dynamic target library.

[0155] Generate a response text for the region of interest, which serves as the reasoning result.

[0156] Understandably, for the point-and-click local question mode, the first step in system implementation is to capture the specific screen coordinates (Px_screen, Py_screen) of the user's point on the augmented display. The system needs to accurately convert these screen coordinates into image coordinates (Px_image, Py_image) on the sampled frame based on the current display ratio, the original resolution of the video frame, and the drone's attitude information. Furthermore, using known camera intrinsic and extrinsic parameters (attitude), a precise conversion from screen coordinates to image coordinates, and even to real-world coordinates, can be achieved. Then, using the converted image coordinates as the center, the system defines a region of interest (ROI). For this ROI, the system performs deep feature extraction, which involves more than simply cropping the image; it includes more refined analysis, such as superpixel segmentation to understand texture consistency within the region, or edge detection to delineate the target outline. Simultaneously, the system supports region analysis at different scales. For example, it can analyze a small window (detail) centered on the click point, and a larger surrounding window (context), then fuse the features from the two scales—this is multi-scale analysis. It is understood that the analysis of local areas in this embodiment is not performed in isolation. The system combines visual context containing information about the surrounding area of ​​the region of interest with semantic information from the dynamic target library to perform a comprehensive analysis of the area. For example, when a user clicks on a valve, the system not only needs to identify the valve itself, but also needs to analyze its state by combining the types of its adjacent pipes and equipment, as well as knowledge about the normal state of the valve from the dynamic target library. The system also dynamically adjusts the ROI extraction strategy and analysis model based on the UAV attitude information (such as pitch angle and altitude) associated with the sampled frames to ensure stable analysis results under different shooting angles. Based on the above comprehensive feature extraction and analysis, the system generates specific and in-depth answer text for the region of interest as the reasoning result. The answer can cover multiple aspects such as target recognition (e.g., "This is a high-voltage disconnect switch"), state analysis (e.g., "The switch is in the closed state"), and anomaly detection (e.g., "There is a suspected oil leak on the valve surface").

[0157] In some embodiments of the present invention, a specific inference scheme is provided when the current interaction mode is scene description mode. In S40, that is, based on the current interaction mode selected by the user and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result, specifically including the following steps:

[0158] Perform global scene analysis on the sampled frames to identify the main targets and target categories in the global scene;

[0159] A semantic segmentation algorithm is used to divide the global scene into several analysis regions, and the functional and spatial relationships between the main targets identified based on the analysis regions are analyzed.

[0160] Reasoning is performed based on a pre-set scene rule base to identify the type and state of the global scene;

[0161] Based on the semantic information in the dynamic target library, the main targets, functions and spatial relationships, and the types and states of the global scene are described in a structured manner;

[0162] A structured scene description text is generated as the reasoning result.

[0163] Understandably, in scene description mode, the system employs a semantic segmentation algorithm to comprehensively segment the entire sampled frame, classifying image pixels into different semantic categories (such as sky, road, building, vehicle, and people) to understand the overall scene composition. Based on this, and combined with a dynamic target library, it identifies the main targets in the image and their categories (e.g., target 1: truck; target 2: crane; target 3: construction site fence). Then, the system further analyzes the spatial layout and relationships between the identified targets. This is not just a simple parallel relationship, but also includes spatial relationships (e.g., "the truck is parked to the right of the crane"), functional relationships (e.g., "the crane is lifting steel"), and even inference relationships based on prior knowledge. Next, based on a pre-defined scene rule base, it performs inference to identify the type and state of the global scene. For example, by analyzing "the presence of a crane, construction site fence, and a large amount of steel," combined with the rule base, it can infer that this is a construction site scene, and that it is currently under construction. Finally, based on the semantic classification system in the dynamic target library, the system organizes and describes the main targets, spatial layout and relationships, and the types and states of the global scene in a structured manner, generating a structured scene description text that is easy to understand and parse as the reasoning result.

[0164] S50: Configure a natural language question trigger to perform semantic matching on the reasoning results, and generate and execute an automatic question-answering chain alarm when the matching degree meets the preset triggering conditions.

[0165] For step S50, the alarm configuration is flexible and efficient, the triggering is accurate and traceable. The triggering conditions are defined in natural language to adapt to the diverse alarm needs of complex scenarios. The multi-dimensional semantic matching algorithm is combined to improve the matching accuracy, and the adaptive threshold and condition combination mechanism reduces false alarms and missed alarms. The generated automatic question and answer chain contains complete triggering information and reasoning basis, which not only makes it easy for users to quickly understand the alarm background, but also provides a clear evidence chain for subsequent review. At the same time, it supports multiple triggers in parallel and historical statistics, further improving the intelligence and practicality of alarm management.

[0166] Specifically, the intelligent alarm function in step S50 is implemented by the alarm module of this system. Key technologies include using Sentence-BERT to calculate semantic similarity, implementing flexible triggering logic based on a rule engine, and a notification service that supports multi-channel notification push. The trigger configuration example is as follows:

[0167] triggers:

[0168] - id: "fire_detection"

[0169] Question: "Was a source of fire or smoke detected?"

[0170] threshold: 0.85

[0171] priority: "high"

[0172] actions:

[0173] - type: "alert"

[0174] channels: ["sms", "email"]

[0175] - type: "auto_mark"

[0176] color: "red"

[0177] In some embodiments of the present invention, such as Figure 4 As shown, a specific alarm scheme is provided. In S50, a natural language question trigger is configured to perform semantic matching on the reasoning result. When the matching degree meets the preset triggering condition, an automatic question-answering chain alarm is generated and executed. Specifically, it includes the following steps S51-S55.

[0178] S51: Create multiple parallel natural language question triggers and configure independent matching thresholds, priorities, and notification rules for each trigger.

[0179] Users can freely create multiple triggers according to their needs, and these triggers run in parallel through a background service. Each trigger is an independently configured probe, and its configuration items include:

[0180] Independent matching threshold: determines the sensitivity of triggering an alarm; for example, a threshold of 0.85 is more sensitive than 0.9.

[0181] Priority: Determines the urgency of the alarm, thereby affecting subsequent scheduling strategies and notification methods.

[0182] Notification rules: Define how to notify the recipient, and at what frequency, after an alarm is triggered, through which channel (SMS, email, APP push, audible and visual alarm), to whom, and at what frequency.

[0183] Trigger conditions: You can set time, location, or specific interaction mode as the prerequisites for triggering, enabling more granular control.

[0184] S52: Configure the triggering question and the corresponding matching threshold defined in natural language as the triggering condition in the trigger.

[0185] In this embodiment, the definition of the trigger question is highly flexible and based on natural language. Specifically, the system provides a visual question editor and a built-in library of commonly used templates (such as "Are there any suspicious persons?", "Is the device malfunctioning?") to reduce configuration difficulty. The core logic of the trigger is: when the system's inference result (the output of step S40) highly matches the natural language question semantically, the event is determined to have occurred. The construction of the question also supports complex logic, such as supporting logical combinations of multiple trigger conditions (AND, OR, NOT) to achieve the detection of complex compound events.

[0186] The following is an implementation example of trigger configuration for a fire detection scenario:

[0187] The trigger's trigger question is "Is a fire source or smoke detected?"; its matching threshold is 0.85; its notification rule is to issue an immediate alarm and summarize the data hourly; after being triggered, it automatically executes actions such as marking relevant frames and notifying on-duty personnel.

[0188] S53: During the reasoning process, the semantic similarity matching value between the reasoning result and the triggering question is calculated in real time.

[0189] In practice, as the system continuously generates inference results—whether it's the description text of the detection boxes or the VQA response text—the background trigger service continuously monitors these results. For each trigger question, the system uses a semantic matching engine, such as a sentence vector model based on Sentence-BERT, to calculate the semantic similarity match between the text description of the inference result and the trigger question. In addition, the system uses keyword matching for rapid initial screening, or for structured inference results, such as "state":"moving" in JSON, it employs pattern matching methods such as regular expressions. Finally, it combines multiple matching results to calculate a final match confidence score, forming a unified match score.

[0190] S54: When the matching value exceeds the corresponding matching threshold, an alarm is automatically triggered.

[0191] Understandably, the trigger service compares the matching value calculated in step S53 with the threshold configured in step S52 in real time. Once the matching value exceeds the threshold, the trigger is activated, entering the alarm generation stage. To improve accuracy, the system introduces intelligent optimization mechanisms, such as adaptive thresholds, specifically, the algorithm dynamically fine-tunes the threshold based on historical false alarms and missed alarms; and false alarm learning, specifically, when a user manually processes and provides feedback on the alarm nature (such as "false alarm"), this feedback is recorded and used to adjust relevant parameters.

[0192] S55: Generate and send a structured question-and-answer chain alert containing the trigger time, matching value, and relevant reasoning basis.

[0193] In this embodiment, once an alarm is triggered, the system assembles a structured alarm message. This message not only contains the alarm itself, but also a question-and-answer chain of evidence. It automatically collects and integrates relevant information to form one or more chained messages, i.e., an automatic question-and-answer chain alarm.

[0194] The following is an example of the data structure for the question-and-answer chain alarm in this embodiment:

[0195] {

[0196] "alert_id": "alert_00456",

[0197] "trigger_id": "fire_detection",

[0198] "trigger_time": "2024-01-15T14:25:30.000Z",

[0199] "match_score": 0.92,

[0200] "evidence": [

[0201] {

[0202] "frame_id": "frame_04567",

[0203] "timestamp": "2024-01-15T14:25:28.100Z",

[0204] "location": [320, 240],

[0205] "description": "Suspected fire source detected"

[0206] }

[0207] ],

[0208] "actions_taken": ["auto_mark", "notify_operator"],

[0209] "status": "active"

[0210] }

[0211] The question-and-answer chain alert system organizes multiple related evidence events into a chain to demonstrate the complete evolution of an event. Finally, through a notification push unit, this complete and structured alert information is pushed to relevant personnel via multiple channels according to the predefined rules of the triggers. Advanced features also support cascading triggers, meaning that an alert from one trigger can automatically activate another trigger configured with related logic, thus achieving chain-like detection and response to complex events.

[0212] S60: Maintain a historical frame cache containing historical sampled frames, and based on the temporal correlation of related frames in the historical frame cache, construct a thought chain for explaining the reasoning process and visualize it.

[0213] Step S60 ensures that the reasoning task possesses both interpretability and traceability. By maintaining historical frame caches and mining inter-frame temporal relationships, the complex reasoning process is broken down into a thought chain containing evidence, intermediate results, and conclusions. Combined with tree structures, interactive folding, and other visualization methods, the AI ​​decision-making logic becomes intuitively visible. This supports users in exploring reasoning details, assisting in responsibility identification and process auditing, while also improving reasoning reliability through temporal change analysis. Simultaneously, it enhances user trust in the system, providing strong support for task review and experience accumulation. Specifically, step S60 is implemented by a visualization module. Key technologies include constructing a thought chain based on temporal data and logical reasoning, using D3.js for interactive visualization, maintaining dialogue context and history, and designing the visualization interface as a video player on the left, a thought chain display area on the right, a dialogue input and output area at the bottom, and a toolbar and operation buttons at the top.

[0214] In some embodiments of the present invention, such as Figure 5 As shown, a specific scheme for constructing a thinking chain is provided. In S60, a historical frame cache containing historical sampling frames is maintained, and a thinking chain for explaining the reasoning process is constructed based on the temporal correlation of related frames in the historical frame cache and then visualized. Specifically, it includes the following steps S61-S63.

[0215] S61: Use a circular buffer to store the most recent N historical sampled frame data.

[0216] In this embodiment, to achieve efficient temporal analysis, the system must have short-term memory of the video data that has just been processed. Technically, the system uses a circular buffer as the cache data structure. Its size is a preset value of N, which typically ranges from 20 to 100, with a preferred value of 50. This means that at any given time, the buffer retains complete information on the N most recent sampled frames arranged chronologically, including the frame's image data (or encoded visual features), precise timestamps, relevant UAV attitude information, and the inference results for that frame.

[0217] To optimize the memory usage of the circular buffer, this embodiment incorporates the Least Recently Used (LRU) algorithm for intelligent cache management. When cache capacity is limited, it prioritizes evicting frames that have not been accessed for the longest time or have lower importance, thus balancing storage space and processing demands. Simultaneously, it employs an efficient compression storage algorithm for historical frame data to reduce memory and storage consumption while preserving image quality in critical areas as much as possible to support subsequent visualization and backtracking.

[0218] To enable rapid retrieval of acquired frames, this embodiment establishes a time-based index structure (time index) for the frames in the cache, enabling the system to quickly locate and retrieve frame data and its associated analysis results at a specific time point based on the timestamp.

[0219] S62: Analyze the temporal changes and semantic relationships between adjacent historical sampling frames.

[0220] The core of the thought chain lies in understanding the development of events over time, rather than simply analyzing individual frames. Specifically, after acquiring a reliable short-term memory, the system enters the temporal correlation analysis phase, first comparing adjacent frames in the cache, such as frame_t-1 and frame_t.

[0221] More specifically, temporal change detection is performed first. Temporal change detection includes the appearance, disappearance, and movement of targets, as well as subtle changes in global attributes such as scene lighting and color. This is achieved by calculating inter-frame differences, target tracking algorithms, or feature matching.

[0222] Then, target tracking and association are performed. For targets that persist in consecutive frames (such as a moving vehicle), the system performs target tracking analysis to form its motion trajectory in the video.

[0223] Next, semantic association mining is performed. A deeper analysis involves mining the semantic relationships between frames. For example, in frame_t-1, a person walks towards a car, and in frame_t, the car starts and leaves. The system needs to connect these independent events to understand the cause and effect.

[0224] S63: Based on the temporal changes and semantic correlations, generate a chain-like thought process record containing reasoning basis, intermediate results and final results.

[0225] Based on the analysis in step S62, the system begins to construct a structured, chain-like internal reasoning log, called the thought chain. This process mimics the human approach to problem analysis, breaking down complex reasoning into a series of logical steps—the thought chain construction algorithm.

[0226] Here is a pseudocode example of building a chain of thoughts:

[0227] def build_thought_chain(history_frames, current_frame):

[0228] chain = []

[0229] # Step 1: Analyze time series changes

[0230] changes = detect_changes(history_frames, current_frame)

[0231] chain.append({

[0232] "step": "change_detection",

[0233] "description": "A change in the screen was detected",

[0234] "details": changes

[0235] })

[0236] # Step 2: Identify key objects

[0237] objects = identify_objects(current_frame)

[0238] chain.append({

[0239] "step": "object_identification",

[0240] "description": "Identifying key objects",

[0241] "objects": objects

[0242] })

[0243] # Step 3: Analyze the event

[0244] events = analyze_events(objects, changes)

[0245] chain.append({

[0246] "step": "event_analysis",

[0247] "description": "Analyzing the nature of the event",

[0248] "events": events

[0249] })

[0250] # Step 4: Generate Conclusion

[0251] conclusion = generate_conclusion(events)

[0252] chain.append({

[0253] "step": "conclusion",

[0254] "description": "Generate final conclusion",

[0255] "conclusion": conclusion

[0256] })

[0257] return chain

[0258] This chain clearly records the basis for each inference (such as frame changes), intermediate results (identified target categories), and final conclusion, greatly enhancing the interpretability of the system's decisions. It also embodies the logic of uncertainty handling and hypothesis testing. For example, if there are multiple possibilities in the identification stage, the chain of thought will record them simultaneously along with their respective confidence levels, and compare and eliminate them in subsequent steps.

[0259] S64: Present the thought process record in an interactive, visual format.

[0260] In this embodiment, the thought chain must be clearly presented in order for users to understand and trust the AI's decision-making. The system uses advanced visualization technology to render the structured thought chain data generated in step S63 into a user-friendly graphical interface.

[0261] The visualization employs a tree structure or timeline view to display the logical hierarchy and temporal evolution of the reasoning. Users can click, collapse / expand tree nodes to gain a deeper understanding of the details of each reasoning step. For example, clicking the "Target Recognition" node will display the bounding box of the recognized target, along with multiple candidate categories and confidence levels provided by the model. This achieves a collapsible visualization. The system can also dynamically demonstrate how the reasoning chain unfolds step by step through animation, making the logic more intuitive.

[0262] The following is an example of the thought process chain structure for anomaly event analysis:

[0263] Root node: An abnormal event was detected.

[0264] Sub-node 1: Identify anomaly types (equipment failure / personnel intrusion / environmental anomaly).

[0265] Child node 2: Locates the abnormal location.

[0266] Sub-node 3: Analyze the severity of the anomaly.

[0267] Leaf nodes: Generate processing suggestions.

[0268] Each node can interact, and the node color may change depending on the level of confidence or the degree of certainty.

[0269] S70: Perform human-machine hybrid annotation on the inference results, and conduct online accuracy evaluation on the inference results and annotation results, and optimize the system parameters or model based on the evaluation results.

[0270] For step S70, the closed loop from result labeling to performance evaluation and then to parameter / knowledge optimization constitutes an online accuracy evaluation and proactive optimization mechanism. This ensures that the system can continuously improve as it is used more extensively, rather than being a statically deployed system.

[0271] Specifically, the human-machine hybrid annotation in step S70 is implemented by the annotation module of this system. The hardware of this module adopts a distributed processing architecture, supports multi-GPU parallel inference, and the single-machine performance can reach 10 frames / second (1080p video), and the cluster performance can reach 100 frames / second (10-node cluster), which significantly improves the annotation efficiency compared with the traditional method.

[0272] In some embodiments of the present invention, such as Figure 6 As shown, a specific optimization scheme is provided. In S70, human-machine hybrid annotation is performed on the inference results, and the accuracy of the inference results and annotation results is evaluated online. Based on the evaluation results, the system parameters or model are optimized, specifically including the following steps S71-S73.

[0273] S71: Receive the inference result, and verify and correct the inference result based on the human-machine hybrid annotation mode to generate a high-confidence labeled truth value.

[0274] The initial inferences generated in step S40 may contain errors or uncertainties and require verification by human experts before they can be transformed into reliable ground truth. This embodiment defines a highly efficient human-in-the-Loop annotation workflow, whose core features include:

[0275] Batch processing and parallelization: The system supports the creation of batch test tasks to perform parallel inference on a complete historical video dataset, generating a large number of prediction results at once, which provides a foundation for subsequent batch verification.

[0276] Two-level collaborative annotation modes: Simple Mode and Detailed Mode. In Simple Mode, for target detection boxes or answers with high confidence and easy-to-judge results, annotators can confirm with one click or perform batch corrections. The system supports an intelligent function that copies prediction results as true values; when the user clicks "Confirm," the system's predictions (such as a list of boxes) are directly marked as rectified_labels and stored in the database, greatly improving the processing speed of simple cases. In Detailed Mode, for complex, ambiguous, or low-confidence results, annotators can access the Frame-by-Frame Refinement interface. In this interface, not only can the position and size of the detection boxes be adjusted, but the category attributes of the targets can also be modified, and even supplementary annotations can be added, such as adding descriptions of relationships between targets ("a person is holding a phone") or adding free text notes describing abnormal states.

[0277] Quality control and version iteration: All annotation results support quality spot checks and version control to ensure the reliability and consistency of the dataset.

[0278] S72: Compare the inference result with the generated labeled true value, and calculate evaluation metrics including at least precision and recall.

[0279] After obtaining a high-quality ground truth dataset, the system initiates an online accuracy evaluation process, the core of which is to perform quantitative calculations to objectively measure the current model performance.

[0280] In practice, the first step is to determine the match: following standard methods in object detection, the Intersection over Union (IoU) is used as the metric to measure the overlap between the predicted bounding box and the ground truth bounding box. The system sets an IoU threshold (e.g., 0.5), and a match is considered complete only when the IoU between the predicted and ground truth bounding boxes exceeds this threshold. Next, based on the matching results, the number of true positives (TP), false positives (FP), and false negatives (FN) is counted.

[0281] Then, based on the statistical results, the following series of evaluation indicators were calculated:

[0282] Precision, calculated as TP / (TP + FP), measures the percentage of correct positive predictions made by the system. Its detailed formula provides a quantitative basis for this judgment. TP (True Positive) represents the number of correctly detected positive samples, and FP (False Positive) represents the number of falsely detected negative samples.

[0283] Recall = TP / (TP + FN) measures how many of the actual ground truth targets the system successfully identifies. FN (False Negative) represents the number of false positives missed.

[0284] The F1 score, which is F1 = 2 * (Precision * Recall) / (Precision + Recall), is a harmonic mean that takes into account both precision and recall, and can better evaluate the overall detection level of a system.

[0285] Intersection over Union (IoU) = Area(B_pred ∩ B_gt) / Area(B_pred ∪ B_gt), where B_pred represents the predicted bounding box and B_gt represents the ground truth bounding box.

[0286] Understandably, this evaluation system provides a multi-dimensional, quantitative view of performance, enabling developers to accurately diagnose in which categories and scenarios the model performs poorly.

[0287] S73: Based on the calculated evaluation indicators, optimize and adjust the parameters of the multimodal large model or the construction strategy of the dynamic target library.

[0288] Based on the detailed evaluation report output in step S72 (which may include multi-dimensional statistics by time period / target category), the system enters the optimization phase. The optimization directions mainly include the following two:

[0289] Model parameter and algorithm optimization: In practice, self-supervised learning (SSL) or incremental learning strategies can be used to perform lightweight retraining of the last few layers of the model or low-fit modules using the newly confirmed labeled ground truth data from step S71. Furthermore, key thresholds during inference (such as the object detection confidence threshold and the alarm trigger matching threshold) will be adaptively adjusted according to changes in the false positive rate (FPR) and false negative rate (FNR) to achieve dynamic balance.

[0290] Iteration of the dynamic target library: Evaluation results are also directly fed back to the knowledge layer. For example, if a word in a certain synonym set fails to effectively recall any target in a large number of queries, the system will automatically reduce the weight of that word in the library or mark it as pending review. This may guide manual intervention to consider whether its definition needs to be updated or more typical visual examples added. Conversely, new words that are frequently triggered and match accurately may be proactively suggested to be added to the target library.

[0291] S80: Based on the optimized system parameters or model, perform retrospective analysis of the historical reasoning process.

[0292] For step S80, the reliability of the tracing results is ensured by relying on the optimized system parameters and model. Historical data is organized through time series and reasoning information is overlaid on video thumbnails, making the development of events intuitive and clear. It supports quick jump to details, multi-user collaborative annotation and commenting, and data export and sharing, which not only provides a complete chain of evidence for task review and responsibility determination, but also helps to summarize experience and optimize subsequent tasks, significantly improving the practicality and decision support capabilities of the application.

[0293] Specifically, the task tracing in step S80 is implemented by the tracing module of this system. The key technologies of this system include using a timeline rendering library for timeline rendering, using multimedia processing tools to generate thumbnails, and analyzing the correlation between events.

[0294] In some embodiments of the present invention, a specific traceability analysis scheme is provided. In S80, that is, based on the optimized system parameters or model, the historical reasoning process is traced and analyzed, which specifically includes the following steps:

[0295] Organize historical sampling frames and their corresponding inference results according to time sequence to generate a result timeline;

[0296] The inference information of key frames in the result timeline is overlaid and rendered onto the video thumbnail of the sampled frame to form an interactive visual traceability interface.

[0297] Understandably, the system organizes all historical sampling frames and their corresponding inference results, annotation comparisons, and alarm records in a time-series manner, forming a structured result timeline. This timeline forms the basis for user tracing and navigation. For efficient display, the system automatically extracts keyframes representing key event changes from the video stream. For these keyframes, the system uses an image compression algorithm to generate uniformly sized video thumbnails. Then, the system uses an information overlay engine to overlay and render the inference information related to each keyframe (such as detection boxes, category labels, confidence levels, alarm markers, etc.) onto its corresponding thumbnail, ensuring that this overlay information is clear, orderly, and avoids mutual occlusion. Finally, these thumbnails with rich annotation information are arranged in chronological order to form a visual tracing interface.

[0298] The following is an implementation example of a power line inspection task review:

[0299] Its timeline is shown as follows:

[0300] 09:00: Mission begins, drones take off.

[0301] 09:15: First insulator damage detected (automatic detection).

[0302] 09:32: Manually confirm the first fault point.

[0303] 09:45: Foreign object detected in wire (user-triggered detection).

[0304] 10:10: Mission complete, drone returns to base.

[0305] The overlaid content in the thumbnail includes: fault point label box, confidence level display, severity mark, and processing status indicator.

[0306] In this embodiment, the interactive functions supported by the timeline include: time navigation, which allows users to quickly jump to any point in time; zoom control, which allows users to zoom in and out of the timeline to view different time granularities; filters, which allow users to filter the displayed content based on event type, target category, etc.; and export function, which allows users to export timeline data as reports or presentation materials.

[0307] The following is an example of a timeline data structure:

[0308] {

[0309] "timeline_id": "tl_001",

[0310] "task_id": "task_power_inspection_001",

[0311] "start_time": "2024-01-15T09:00:00Z",

[0312] "end_time": "2024-01-15T10:10:00Z",

[0313] "events": [

[0314] {

[0315] "event_id": "evt_001",

[0316] "timestamp": "2024-01-15T09:15:23Z",

[0317] "event_type": "fault_detected",

[0318] "description": "Insulator damage",

[0319] "frame_id": "frame_00923",

[0320] "thumbnail": "thumb_00923.jpg",

[0321] "annotations": [...],

[0322] "confidence": 0.96

[0323] } ]

[0325] }

[0326] S90: Based on the results of the traceability analysis, perform batch testing and active learning.

[0327] In this embodiment, step S90 is the highest-level loop in the system capability closed loop, upgrading from passive tracing to active learning. Its specific implementation is as follows:

[0328] In the visual traceability interface provided in step S80, users can easily select a batch of historical inference data (e.g., a video segment with a high false negative rate in a certain task). The system then initiates a new batch test, specifically, using the latest model parameters optimized in step S73 and the updated dynamic target library to perform asynchronous, offline new inference on this batch of historical data.

[0329] Because it's an offline replay, users can easily review the new inference results and use the S71's human-machine hybrid annotation tool for more refined second-round verification. Ultimately, the high-confidence, high-quality new predictions and new corrections generated from this batch testing and active annotation will be intelligently and incrementally added to the annotation truth library automatically.

[0330] Thus, a dynamic and continuously evolving closed loop is formed: new ground truths supplement richer and more accurate training data; this new data provides a more solid benchmark for the next round of online evaluation; the new evaluation results drive a new round of model and knowledge optimization; and the optimized system capabilities will directly improve the efficiency of subsequent real-time video inference.

[0331] Understandably, through the optimization mechanism of steps S60 to S90, this system has far exceeded a tool and evolved into an intelligent agent with self-reflection, explainability, and evolutionary characteristics. Its value has been clearly demonstrated in long-term and complex practical applications.

[0332] Furthermore, this embodiment can persistently store analysis results and supports data export and integration with external systems. Specifically, through a systematic data management method, it securely stores and flexibly reuses all analysis data, including raw videos, inference results, annotation information, and evaluation data, while establishing collaborative channels with external systems to ensure maximum data value and support the business closed loop.

[0333] The data storage employs a tiered approach, combining hot data SSD storage with cold data HDD storage. Storage resource allocation is optimized based on data access frequency, ensuring fast read / write responses for high-frequency data while accommodating massive archived data with low-cost storage media, achieving a balance between storage efficiency and cost. A hybrid storage approach using relational and NoSQL databases is employed. Structured data utilizes relational databases to ensure consistency and query efficiency, while unstructured data leverages NoSQL databases to adapt to flexible formats and massive storage needs, ensuring storage adaptability for different data types. A dual backup method, combining scheduled full backups and real-time incremental backups with off-site storage, effectively mitigates risks such as system failures and hardware damage, ensuring data integrity and recoverability, and providing security for task review and data traceability. Highly efficient data compression methods are used, with dedicated compression algorithms adapted for different data types such as video and text, significantly reducing storage space usage and storage hardware investment costs without affecting data quality.

[0334] Among its features, data export supports multiple standard formats such as JSON, XML, CSV, and Excel, adapting to different scenarios such as data interaction, statistical analysis, and document archiving, improving the flexibility of data reuse, and reducing the manual cost of cross-platform data conversion; it provides a custom export template method, allowing users to select export fields and adjust data arrangement formats as needed to generate personalized export files that fit business requirements, reducing the workload of subsequent data processing; it adopts a batch asynchronous export method, supporting batch export operations of large-scale data, avoiding the occupation of real-time system analysis resources through asynchronous processing, and ensuring that the export process does not affect the normal operation of core business; it supports an incremental export method, exporting only newly added or modified data, reducing the overhead of repeated data transmission and storage, improving the efficiency of data update synchronization, and adapting to the needs of high-frequency data update scenarios.

[0335] The system integrates with external systems, providing standardized RESTful APIs and WebSocket interfaces to establish data interaction channels with third-party business systems. This enables real-time push and cross-system calls of data such as inference results and alarm information, improving the collaborative efficiency of multiple business systems. It supports bidirectional data synchronization, allowing the system to synchronize analysis results to external systems such as UAV flight control systems and emergency command platforms, while also receiving task instructions and configuration parameters from external systems, achieving a closed-loop cross-system business process. Furthermore, it integrates single sign-on methods from the enterprise's unified authentication system (OAuth2.0, SAML protocol), allowing users to access systems across systems without repeated authentication, simplifying operations, improving efficiency, and ensuring unified management and security of account permissions. Finally, it supports integration with external permission management systems, linking the system's permission system with the enterprise's existing permission management framework to achieve unified allocation, modification, and auditing of permissions, adapting to enterprise-level multi-system permission control needs, and improving the standardization and security of permission management.

[0336] It is understood that the multimodal interactive video analysis method of this embodiment can be widely applied to multiple fields such as power line inspection, border monitoring, and emergency rescue. The following is a detailed description in conjunction with specific scenarios.

[0337] Power line inspection scenario: A power company needs to conduct regular inspections of high-voltage transmission lines to promptly detect potential hazards such as damaged insulators, foreign objects in conductors, and tilted towers. The multimodal interactive video analysis method described in this example can significantly improve inspection efficiency and accuracy. In terms of system configuration, the hardware includes a drone, an aerial vehicle gimbal camera, and a 5G communication module. In the software configuration, the sampling interval is set to 0.5 seconds, the dynamic target library is configured with a dedicated terminology for power equipment (including core targets such as insulators, conductors, and towers, as well as synonym extensions), the triggers are configured with equipment fault detection rules, and the evaluation cycle is set to once per hour.

[0338] The implementation process is divided into three stages: the preparation stage involves importing route data and inspection plans, configuring dynamic target libraries and triggers, and completing system calibration; the execution stage involves the UAV flying along the predetermined route, with the system receiving and processing UAV video streams in real time, and notifying relevant personnel through an automatic question-and-answer chain alarm when an anomaly is triggered; and the review stage involves generating an inspection report, analyzing the fault distribution pattern based on the result timeline, and optimizing subsequent inspection strategies.

[0339] As can be seen, the above solution solves the technical problems of single interaction mode and rigid operation by superimposing four modes of inference layers on the acquisition frames of the UAV video stream: detection box annotation, natural language question answering, pointing question and scene description. This allows users to flexibly switch the interaction mode according to the task scenario and real-time intention, and makes the analysis process more intuitive and efficient. It significantly reduces the learning and operation costs and meets the differentiated in-depth analysis needs of different professional roles, thereby greatly improving the intelligence level and practical application effectiveness of UAV video analysis.

[0340] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0341] In one embodiment, the present invention provides a multimodal interactive video analysis device 100, which corresponds one-to-one with the multimodal interactive video analysis method described in the above embodiments. For example... Figure 7 As shown, the multimodal interactive video analysis device 100 includes a video access module 101, an interaction module 102, an enhancement module 103, an inference module 104, an alarm module 105, a visualization module 106, an annotation module 107, a tracing module 108, and a learning module 109. Detailed descriptions of each functional module are as follows:

[0342] The video access module 101 is used to access the video stream of the drone and automatically sample it according to a preset time interval to obtain a sampled frame sequence;

[0343] The interaction module 102 is used to create an inference layer corresponding to different interaction modes on the sampling frame and to receive interaction information input by the user based on the interaction mode; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, pointing local question mode and scene description mode.

[0344] Enhancement module 103 is used to perform adaptive semantic enhancement based on the prompt words of the interaction information, generate a set of synonyms and update the dynamic target library;

[0345] Inference module 104 is used to perform multimodal video inference on the sampled frame sequence based on the current interaction mode selected by the user and the dynamic target library, and obtain inference results;

[0346] Alarm module 105 is used to configure natural language question triggers, perform semantic matching on the reasoning results, and generate and execute automatic question-answering chain alarms when the matching degree meets preset triggering conditions;

[0347] The visualization module 106 is used to maintain a historical frame cache containing historical sampling frames, and to construct a thought chain for explaining the reasoning process based on the temporal correlation of related frames in the historical frame cache, and to visualize the result.

[0348] The annotation module 107 is used to perform human-machine hybrid annotation on the inference results, and to evaluate the accuracy of the inference results and the annotation results online, and to optimize the system parameters or model based on the evaluation results;

[0349] The tracing module 108 is used to perform tracing analysis on the historical reasoning process based on optimized system parameters or models;

[0350] Learning module 109 is used to perform batch testing and active learning based on the results of the traceability analysis.

[0351] Specific limitations regarding the multimodal interactive video analysis device 100 can be found in the limitations of the multimodal interactive video analysis method described above, and will not be repeated here. Each module in the multimodal interactive video analysis device 100 can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0352] In one embodiment, a computer device 200 is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown. The computer device 200 includes a processor 220, memory, and a network interface 250 connected via a system bus 210. The processor 220 provides computing and control capabilities. The memory of the computer device 200 includes non-volatile and / or volatile storage media and internal memory 240. The non-volatile storage media 230 stores an operating system 231, computer programs 232, and a database 233. The internal memory 240 provides an environment for the operation of the operating system and computer programs in the non-volatile storage media 230. The network interface 250 of the computer device 200 is used to communicate with external clients via a network connection. When the computer program is executed by the processor 220, it implements the functions or steps of a multimodal interactive video analysis method server. That is, when the processor 220 executes the computer program, it implements the following steps:

[0353] Access the drone video stream and automatically sample it at preset time intervals to obtain a sequence of sampled frames;

[0354] An inference layer corresponding to different interaction modes is created on the sampling frame, and the interaction information input by the user based on the interaction mode is received; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, pointing local question mode, and scene description mode;

[0355] Based on the prompts in the interactive information, adaptive semantic enhancement is performed to generate a set of synonyms and update the dynamic target library.

[0356] Based on the user-selected current interaction mode and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result;

[0357] Configure a natural language question trigger to perform semantic matching on the reasoning results, and generate and execute an automatic question-answering chain alarm when the matching degree meets the preset triggering conditions.

[0358] In one embodiment, a computer device 300 is provided, which may be a client, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor 320, memory, network interface 350, display screen 370, and input device 360 ​​connected via a system bus 310. The processor 320 provides computing and control capabilities. The memory includes a non-volatile storage medium 330 and internal memory 340. The non-volatile storage medium 330 stores an operating system 331 and a computer program 332. The internal memory provides an environment for the operation of the operating system 331 and the computer program 332 in the non-volatile storage medium 330. The network interface 350 of the computer device 300 is used for communication with an external server via a network connection. When the computer program is executed by the processor 320, it implements the functions or steps of a multimodal interactive video analysis method on the client side. That is, when the processor 320 executes the computer program 332, it implements the following steps:

[0359] Access the drone video stream and automatically sample it at preset time intervals to obtain a sequence of sampled frames;

[0360] An inference layer corresponding to different interaction modes is created on the sampling frame, and the interaction information input by the user based on the interaction mode is received; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, pointing local question mode, and scene description mode;

[0361] Based on the prompts in the interactive information, adaptive semantic enhancement is performed to generate a set of synonyms and update the dynamic target library.

[0362] Based on the user-selected current interaction mode and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result;

[0363] Configure a natural language question trigger to perform semantic matching on the reasoning results, and generate and execute an automatic question-answering chain alarm when the matching degree meets the preset triggering conditions.

[0364] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0365] Access the drone video stream and automatically sample it at preset time intervals to obtain a sequence of sampled frames;

[0366] An inference layer corresponding to different interaction modes is created on the sampling frame, and the interaction information input by the user based on the interaction mode is received; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, pointing local question mode, and scene description mode;

[0367] Based on the prompts in the interactive information, adaptive semantic enhancement is performed to generate a set of synonyms and update the dynamic target library.

[0368] Based on the user-selected current interaction mode and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result;

[0369] Configure a natural language question trigger to perform semantic matching on the reasoning results, and generate and execute an automatic question-answering chain alarm when the matching degree meets the preset triggering conditions.

[0370] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0371] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0372] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0373] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multimodal interactive video analysis method, characterized in that, include: Access the drone video stream and automatically sample it at preset time intervals to obtain a sequence of sampled frames; An inference layer corresponding to different interaction modes is created on the sampling frame, and the interaction information input by the user based on the interaction mode is received; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, pointing local question mode, and scene description mode; Based on the prompts in the interactive information, adaptive semantic enhancement is performed to generate a set of synonyms and update the dynamic target library. Based on the user-selected current interaction mode and the dynamic target library, multimodal video inference is performed on the sampled frame sequence to obtain the inference result; Configure a natural language question trigger to perform semantic matching on the reasoning results, and generate and execute an automatic question-answering chain alarm when the matching degree meets the preset triggering conditions; The step of creating an inference layer corresponding to different interaction modes on the sampling frame and obtaining user-input interaction information includes: On the sampling frame, a corresponding inference task is constructed for each interaction mode; Create an independent canvas context for the inference task, associate the canvas context with the frame identifier of the sampled frame sequence, and output a canvas context containing task attributes and frame association relationships; Based on the canvas context, configure the corresponding inference layer for the interaction mode, and use the canvas context as the execution environment of the inference layer to output the instantiated inference layer. When the current interaction mode is the detection box annotation mode, the step of performing multimodal video inference on the sampled frame sequence based on the user-selected current interaction mode and the dynamic target library to obtain the inference result includes: The sampled frames are encoded into visual feature vectors, and the target descriptions in the dynamic target library are used as text prompts. Both are input into the multimodal large model to perform cross-modal alignment. Based on the aligned cross-modal features, the normalized bounding box coordinates corresponding to the target description text are output through a multimodal large model; The text sequence containing the bounding box coordinates in the output of the multimodal large model is parsed, or the cross-modal attention weights generated by the multimodal large model in the process of generating the bounding box coordinates are analyzed to obtain the prediction confidence of each bounding box coordinate. Filter out low-confidence detection results below a preset threshold and obtain high-confidence bounding box coordinates as inference results.

2. The multimodal interactive video analysis method according to claim 1, characterized in that, The step of creating an inference layer corresponding to different interaction modes on the sampling frame and obtaining user-input interaction information further includes: For the instantiated inference layer, associate the timestamp and attitude information of the sampling frame, and output the inference layer with timing synchronization information; Allocate independent computing resources to the inference layer with timing synchronization information, schedule the execution of the inference tasks through a priority queue, and output the inference layer with completed resource scheduling. The inference task is executed in the inference layer after the resource scheduling is completed, and the intermediate and final inference results are superimposed and rendered onto the original sampling frame in real time to generate and present an enhanced display screen to the user.

3. The multimodal interactive video analysis method according to claim 1, characterized in that, The adaptive semantic enhancement of prompts based on the interaction information, generating a synonym set and updating the dynamic target library, includes: Extract at least one Chinese prompt word from the interactive information; The Chinese prompt word is input into a pre-trained word embedding model to calculate the semantic similarity between the Chinese prompt word and multiple candidate words in the candidate word library; Based on the calculated semantic similarity, candidate words with semantic similarity higher than a preset threshold are selected to generate a set of synonyms for the Chinese prompt words; The dynamic target library is updated using the generated set of synonyms.

4. The multimodal interactive video analysis method according to claim 1, characterized in that, When the current interaction mode is a natural language question-and-answer mode, the step of performing multimodal video inference on the sampled frame sequence based on the user-selected current interaction mode and the dynamic target library to obtain the inference result includes: The sampled frames are encoded into visual feature vectors, and the natural language questions in the user's interactive information are used as text prompts, which are then input into the multimodal large model. By using the multimodal large model, combined with the global visual features of the sampled frames and the semantic information in the dynamic target library, the natural language problem is understood and reasoned. Generate a textual response in structured or natural language form as the result of the inference.

5. The multimodal interactive video analysis method according to claim 1, characterized in that, When the current interaction mode is the pointing-to-question mode, the step of performing multimodal video inference on the sampled frame sequence based on the user-selected current interaction mode and the dynamic target library to obtain the inference result includes: Obtain the coordinates of the user pointing at the screen in the interaction information, and convert the coordinates into image coordinates on the sampling frame; Generate a region of interest based on the image coordinates; Deep feature extraction is performed on the region of interest to obtain the visual features of the local region; The region of interest is comprehensively analyzed by combining the visual features of the local region, the visual context information around the region of interest, and the semantic information in the dynamic target library. Generate a response text for the region of interest, which serves as the reasoning result.

6. The multimodal interactive video analysis method according to claim 1, characterized in that, When the current interaction mode is a scene description mode, the step of performing multimodal video inference on the sampled frame sequence based on the user-selected current interaction mode and the dynamic target library to obtain the inference result includes: Perform global scene analysis on the sampled frames to identify the main targets and target categories in the global scene; A semantic segmentation algorithm is used to divide the global scene into several analysis regions, and the functional and spatial relationships between the main targets identified based on the analysis regions are analyzed. Reasoning is performed based on a pre-set scene rule base to identify the type and state of the global scene; Based on the semantic information in the dynamic target library, the main targets, functions and spatial relationships, and the types and states of the global scene are described in a structured manner; A structured scene description text is generated as the reasoning result.

7. The multimodal interactive video analysis method according to claim 1, characterized in that, The configured natural language question trigger performs semantic matching on the inference results, and generates and executes an automatic question-answering chain alarm when the matching degree meets a preset trigger condition, including: Create multiple parallel natural language question triggers, and configure independent matching thresholds, priorities, and notification rules for each trigger; The trigger is configured with a trigger question defined in natural language and a corresponding matching threshold as the trigger condition. During the reasoning process, the semantic similarity matching value between the reasoning result and the triggering question is calculated in real time. When the matching value exceeds the corresponding matching threshold, an alarm is automatically triggered; Generate and send a structured question-and-answer chain alert containing the trigger time, matching value, and relevant reasoning.

8. The multimodal interactive video analysis method according to claim 1, characterized in that, The configuration of the natural language question trigger, after semantic matching of the inference result and generating and executing an automatic question-answering chain alarm when the matching degree meets the preset triggering conditions, also includes: Maintain a historical frame cache containing historical sampled frames, and construct a thought chain to explain the reasoning process based on the temporal association of related frames in the historical frame cache, and then visualize it.

9. The multimodal interactive video analysis method according to claim 8, characterized in that, The maintenance includes a historical frame cache containing historical sampled frames, and based on the temporal correlation of related frames in the historical frame cache, a thought chain for explaining the reasoning process is constructed and visualized, including: A circular buffer is used to store the most recent N historical sampled frames; Analyze the temporal changes and semantic relationships between adjacent historical sampling frames; Based on the temporal changes and semantic relationships, a chain-like thought process record containing reasoning basis, intermediate results and final results is generated; The thought process record is presented in an interactive and visual format.

10. The multimodal interactive video analysis method according to claim 8, characterized in that, The process of maintaining a historical frame cache containing historical sampled frames, constructing a thought chain for explaining the reasoning process based on the temporal correlation of related frames in the historical frame cache, and visualizing the result, further includes: The inference results are subjected to human-machine hybrid annotation, and the inference results and annotation results are evaluated online for accuracy. Based on the evaluation results, the system parameters or model are optimized.

11. The multimodal interactive video analysis method according to claim 10, characterized in that, The process of performing human-machine hybrid annotation on the inference results and evaluating the accuracy of the inference results and annotation results online, and optimizing system parameters or models based on the evaluation results, includes: The inference results are received, and the inference results are verified and corrected based on the human-machine hybrid annotation mode to generate high-confidence labeled truth values. The inference results are compared with the generated labeled true values, and evaluation metrics including at least precision and recall are calculated. Based on the calculated evaluation metrics, the parameters of the multimodal large model or the construction strategy of the dynamic target library are optimized and adjusted.

12. The multimodal interactive video analysis method according to claim 10, characterized in that, The process of performing human-machine hybrid annotation on the inference results, evaluating the accuracy of the inference results and annotation results online, and optimizing the system parameters or model based on the evaluation results further includes: Based on the optimized system parameters or model, the historical reasoning process is traced and analyzed.

13. The multimodal interactive video analysis method according to claim 12, characterized in that, The retrospective analysis of the historical reasoning process based on the optimized system parameters or model includes: Organize historical sampling frames and their corresponding inference results according to time sequence to generate a result timeline; The inference information of key frames in the result timeline is overlaid and rendered onto the video thumbnail of the sampled frame to form an interactive visual traceability interface.

14. The multimodal interactive video analysis method according to claim 12, characterized in that, The optimized After tracing and analyzing the historical reasoning process, the system parameters or model also include: Based on the results of the traceability analysis, batch testing and active learning are performed.

15. A multimodal interactive video analysis device, characterized in that, include: The video access module is used to access the drone video stream and automatically sample it at preset time intervals to obtain a sequence of sampled frames. An interaction module is used to create inference layers corresponding to different interaction modes on the sampling frame and to receive interaction information input by the user based on the interaction modes; wherein, the interaction modes include detection box annotation mode, natural language question answering mode, point-and-answer mode, and scene description mode; wherein, creating inference layers corresponding to different interaction modes on the sampling frame and obtaining user-input interaction information includes: On the sampling frame, a corresponding inference task is constructed for each interaction mode; Create an independent canvas context for the inference task, associate the canvas context with the frame identifier of the sampled frame sequence, and output a canvas context containing task attributes and frame association relationships; Based on the canvas context, configure the corresponding inference layer for the interaction mode, and use the canvas context as the execution environment of the inference layer to output the instantiated inference layer. The enhancement module is used to perform adaptive semantic enhancement based on the prompts in the interactive information, generate a set of synonyms, and update the dynamic target library; The inference module is used to perform multimodal video inference on the sampled frame sequence based on the current interaction mode selected by the user and the dynamic target library, and obtain the inference result; The alarm module is used to configure natural language question triggers, perform semantic matching on the inference results, and generate and execute automatic question-answering chain alarms when the matching degree meets preset trigger conditions; when the current interaction mode is the detection box annotation mode, the step of performing multimodal video inference on the sampled frame sequence according to the current interaction mode selected by the user and the dynamic target library to obtain the inference results includes: The sampled frames are encoded into visual feature vectors, and the target descriptions in the dynamic target library are used as text prompts. Both are input into the multimodal large model to perform cross-modal alignment. Based on the aligned cross-modal features, the normalized bounding box coordinates corresponding to the target description text are output through a multimodal large model; The text sequence containing the bounding box coordinates in the output of the multimodal large model is parsed, or the cross-modal attention weights generated by the multimodal large model in the process of generating the bounding box coordinates are analyzed to obtain the prediction confidence of each bounding box coordinate. Filter out low-confidence detection results below a preset threshold and obtain high-confidence bounding box coordinates as inference results.

16. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal interactive video analysis method as described in any one of claims 1 to 15.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the multimodal interactive video analysis method as described in any one of claims 1 to 15.

Images