Video semantic question-answering system oriented to monitoring scene

Abstract

The invention discloses a video semantic question-answering system for a monitoring scene, and the system comprises a video collection module which is composed of a plurality of cameras and forms a monitoring network; the video preprocessing module is used for optimizing original video data through three-stage processing of key frame extraction, noise reduction and time alignment; the multi-modal feature extraction module comprises a video feature extraction channel and a text feature extraction channel; the multi-modal interactive reasoning module adopts a cross-modal neural network model; the answer generation module is used for outputting corresponding structured answer branches and natural language branches based on joint representation prediction answers; and an interaction and visualization interface module. The invention provides a video semantic question-answering system oriented to a monitoring scene, which supports a user to carry out semantic query on monitoring video contents in a natural language form, and the system automatically analyzes problems, analyzes video data and generates structured or natural language answers. And the combined understanding and response capability of open semantics, dynamic behaviors and attribute states is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of video semantic interaction, and in particular to a video semantic question-answering system for monitoring scenarios. Background Art

[0002] Current surveillance video analysis systems are primarily used in areas such as campus security, traffic management, and factory supervision. Commonly used intelligent video analysis systems are primarily based on traditional computer vision technology, and their technical architecture typically consists of the following three layers:

[0003] (1) Data acquisition layer: Deploy a network of surveillance cameras with fixed viewing angles, use the H.265 encoding format to achieve real-time acquisition and transmission of 1080P resolution video streams, and upload them to the central server or edge device as data for subsequent analysis.

[0004] (2) Analysis and processing layer: Relying on classic target detection models such as YOLOv3 / v4 and Faster R-CNN, specific targets such as people, vehicles, and objects are identified and labeled through a preset feature extraction network; combined with trajectory tracking algorithms such as SORT and DeepSORT, specific behaviors such as gathering of people in the area, climbing over fences, and entering restricted areas are identified; while the natural language processing (NLP) module usually exists as an independent subsystem and is only used for simple log analysis or alarm text generation, and cannot be deeply integrated with visual detection.

[0005] (3) Application display layer: Displays real-time images through the client or web interface to achieve static display of detection results and basic alarm functions

[0006] Existing technologies present challenges: Traditional systems typically process visual and structured data separately, using visual features solely for object detection and text modules solely for logging. These systems lack the ability to deeply integrate and reason with video and language information. Consequently, they lack the ability to understand and respond to open-ended natural language questions. For example, operators struggle to directly ask, "Were there any visitors at the South Gate yesterday without their badges?" These complex monitoring requirements are difficult to address. Summary of the Invention

[0007] The purpose of the present invention is to provide a video semantic question-answering system for surveillance scenarios, which supports users to perform semantic queries on surveillance video content in natural language, and the system automatically parses questions, analyzes video data, and generates structured or natural language answers, thereby realizing the joint understanding and response capabilities of open semantics, dynamic behaviors and attribute states.

[0008] The technical solution adopted by the video semantic question answering system for surveillance scenarios disclosed in the present invention is:

[0009] A video semantic question answering system for surveillance scenarios, including:

[0010] The video acquisition module, as the basic input end, is composed of multiple cameras to form a monitoring network. Through standardized access and unified management, it collects video data in real time.

[0011] The video preprocessing module optimizes the raw video data through three-level processing of keyframe extraction, noise reduction, and time alignment to improve the quality of subsequent analysis;

[0012] The multimodal feature extraction module includes a video feature extraction channel: it adopts an improved ViT architecture and builds a multi-scale attention mechanism. Through a spatial layering strategy, it extracts detailed features and global structures at different spatial resolution levels, and maintains the semantic flow between upper and lower layers through a residual connection mechanism. In the temporal dimension, the model introduces a cross-frame attention mechanism and position encoding to capture the semantic evolution of continuous behavior, and ultimately outputs video features V; and a text feature extraction channel: it uses a pre-trained language model to semantically encode the user input question to obtain text features T;

[0013] The multimodal interactive reasoning module uses a cross-modal neural network model to effectively combine video data and text data;

[0014] Answer generation module: predicts the answer based on the joint representation and outputs the corresponding structured answer branch and natural language branch;

[0015] Interactive and visual interface module: supports interactive operations. Users can enter query content in text form, which will automatically parse the question, call the reasoning module and return multimodal visual results, forming a complete question-answering closed loop.

[0016] As a preferred solution, the key frame extraction: adopts a key frame extraction algorithm based on motion detection to extract a representative frame sequence containing core information from the video stream, effectively reducing the amount of video data while retaining key spatiotemporal information;

[0017] Denoising: Utilizing deep learning-based non-local mean denoising technology, this technology effectively removes image noise that may occur in low-light conditions, ensuring that image quality meets high standards in all environments.

[0018] The time alignment method uses a time alignment algorithm to correct the time delay differences between different cameras and ensure the time consistency of the video stream during cross-view analysis.

[0019] As a preferred solution, constructing a cross-modal joint representation of video features and text features includes:

[0020] Mapping the video features to a unified semantic space through a first projection matrix;

[0021] Mapping the text features to the unified semantic space through a second projection matrix;

[0022] Based on the text features as the query and the video features as the key and value, a cross-attention mechanism is used to calculate the association weights to generate an adjusted video semantic representation;

[0023] The adjusted video semantic representation and text semantic representation are concatenated to form a cross-modal joint representation.

[0024] As a preferred solution, the structured answer branch: for location query questions, uses a trainable regression and classification network to predict the bounding box coordinates of the target in the video coordinate system;

[0025] Natural language branch: For open problems such as behavior description, a transformer-based sequence generation model is used to obtain natural language answers while retaining the corresponding evidence information to enhance the interpretability of the results.

[0026] The beneficial effects of the video semantic question answering system for surveillance scenarios disclosed in the present invention are: innovatively introducing a multi-scale, hierarchical spatiotemporal attention mechanism in traditional ViT, extracting key features of a single frame through local window attention, and using cross-frame global attention to capture long-range temporal correlations, significantly improving the accuracy and efficiency of video spatiotemporal feature extraction, proposing a learnable dynamic projection matrix and text-guided attention mechanism to achieve semantic alignment of video and text features, developing a spatiotemporal constrained natural language parser, and converting it into an executable visual detection task, by removing traditional redundant design and retaining only a single-stage attention mechanism to complete feature interaction, enabling the system to run in real time on resource-constrained edge devices, providing a feasible technical solution for the deployment of intelligent monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of a video semantic question answering system for surveillance scenarios according to the present invention.

[0028] Figure 2 This is a video feature extraction structure diagram of a video semantic question-answering system for surveillance scenarios according to the present invention.

[0029] Figure 3 This is a flow chart of a cross-modal feature interaction module of a video semantic question-answering system for surveillance scenarios according to the present invention. DETAILED DESCRIPTION

[0030] The present invention will be further described and explained below in conjunction with specific embodiments and accompanying drawings:

[0031] Please refer to Figure 1, a video semantic question answering system for surveillance scenarios, including:

[0032] The video acquisition module, as the basic input terminal, is composed of multiple high-definition cameras deployed at key locations in the park, forming a comprehensive monitoring network. Through standardized access and unified management, video data is collected in real time.

[0033] The video preprocessing module optimizes the original video data through three-level processing of key frame extraction, noise reduction and time alignment, improving the quality of subsequent analysis.

[0034] Key frame extraction: A key frame extraction algorithm based on motion detection is used to extract representative frame sequences containing core information from the video stream, effectively reducing the amount of video data while retaining key spatiotemporal information.

[0035] Noise Reduction: Utilizing deep learning-based non-local means denoising technology, we effectively remove image noise that may occur in low-light conditions, ensuring high image quality in all environments.

[0036] Time alignment: Apply a time alignment algorithm to correct for delay differences between different cameras and ensure temporal consistency of video streams during cross-view analysis.

[0037] Multimodal feature extraction module, please refer to Figure 2 , including the video feature extraction channel: using the improved Vi T architecture, building a multi-scale attention mechanism, through the spatial layering strategy, respectively extracting detailed features and global structures at different spatial resolution levels, and maintaining the semantic flow of upper and lower layers through the residual connection mechanism. In the temporal dimension, the model introduces a cross-frame attention mechanism and position encoding to capture the semantic evolution in continuous behavior, and finally outputs the video feature V; and the text feature extraction channel: using a pre-trained language model such as BERT to semantically encode the user input question and obtain the text feature T;

[0038] The multimodal interactive reasoning module adopts a cross-modal neural network model to construct a cross-modal union of video features and text features, realizing the effective combination of video data and text data.

[0039] Mapping the video features to a unified semantic space through the first projection matrix;

[0040] Map the text features to the unified semantic space through the second projection matrix;

[0041] Based on text features as queries and video features as keys and values, a cross-attention mechanism is used to calculate the association weights and generate adjusted video semantic representations;

[0042] The spliced ​​and adjusted video semantic representation is combined with the text semantic representation to form a cross-modal joint representation.

[0043] Please refer to Figure 3 , cross-modal neural network model:

[0044] The video features V∈Rdv×d and Wt∈Rdt×d are respectively transformed into N ×dv and text features T∈R M ×dt are unified into a semantic space of the same dimension and normalized to obtain features Fv = VWv and Ft = TWt to ​​eliminate modality differences;

[0045] Introducing a text-guided video attention mechanism, using text features as queries and video features as key-value pairs to calculate cross-modal attention weights Where Aij represents the association strength between the i-th text word and the j-th video region, and the adjusted video feature F'v is obtained;

[0046] The adjusted video feature F'v is concatenated with the text feature Ft to generate a joint representation Fjoint = [F'v; Ftt], which serves as the input for downstream answer generation.

[0047] Answer generation module: predicts the answer based on the joint representation and outputs the corresponding structured answer branch and natural language branch.

[0048] The structured answer branch: For location query questions, a trainable regression and classification network is used to predict the bounding box coordinates (x, y, w, h) of the target in the video coordinate system;

[0049] Natural language branch: For open questions such as behavior description, a transformer-based sequence generation model is used to obtain natural language answers while retaining corresponding evidence information, such as the corresponding time point, camera number, and screenshot area, to enhance the interpretability of the results.

[0050] Interactive and visual interface module: supports operation through terminal pages, client software or smart large screens. Users can enter query content in text form. The system will automatically parse the question, call the reasoning module and return multimodal visual results, forming a complete question-answering closed loop.

[0051] Example:

[0052] The user enters a question through the multimodal interactive reasoning module: "Between 3:00 and 4:00 p.m., were there any people at the south gate of the park without their work badges?"

[0053] System execution process:

[0054] (1) Video data processing stage:

[0055] Spatiotemporal positioning: Matching the "South Gate" to the camera node based on the digital twin map of the park (CAM-07 / 08)

[0056] Time period capture: Automatically align with the NTP time server and accurately capture the original video stream from 15:00:00 to 16:00:00

[0057] Keyframe sampling: Adopts motion-aware adaptive sampling algorithm (base sampling rate 1fps, automatically increased to 3fps when motion is intense)

[0058] (2) Visual feature extraction stage: ID card detection based on improved ViT

[0059] Regional Focus: Enhancing Neck Region Feature Extraction Through Attention Mechanism

[0060] Multi-scale verification: Simultaneously detect auxiliary features such as the badge lanyard and card text

[0061] Anti-interference design: effectively handles complex situations such as work badge being blocked or reflected

[0062] (3) Semantic Understanding Stage: Text Encoder Workflow

[0063] Time and space expression analysis: "3pm to 4pm" -> [15:00,16:00]

[0064] Spatial semantic understanding: "South Gate" -> geographic coordinates

[0065] Attribute condition extraction: "Not wearing work badge" -> Work badge status: Missing

[0066] (4) Cross-modal matching stage: CM-GNN implementation mechanism

[0067] Node features: Fusion of visual appearance features (ViT output) and spatiotemporal coordinate edge relationships: Establishing cross-frame associations of people and temporal continuity constraints Iterative optimization: Output matching results after 3 rounds of network propagation

[0068] 3. Output generation phase:

[0069] Result integration: Aggregate the spatiotemporal occurrence records of all matching targets Confidence filtering: Only retain detection results with a confidence level > 85%

[0070] Natural language generation: "Two people were found without badges.

[0071] ① Appeared in the South Gate security area at 15:23 (91% confidence level)

[0072] ② Passed through the South Gate at 15:47 (confidence level 87%)"

[0073] (You can click on the timestamp to view the original video clip)

[0074] Technical features:

[0075] 1. Processing efficiency: compared with manual inspection, it can save more than 90% of time cost

[0076] 2. Accuracy guarantee: A multi-level verification mechanism ensures a false alarm rate of <5%

[0077] 3. Traceability: All conclusions can be traced back to the original video evidence frames.

[0078] Significantly improves the accuracy of video semantic understanding. Through the improved ViT architecture and multi-scale spatiotemporal attention mechanism, the system can more accurately extract global semantic features and behavioral intentions in videos, solving the problem of traditional CNN models' insufficient understanding of complex scenes.

[0079] Achieve deep multimodal fusion. Through semantically driven feature alignment and joint reasoning, the system can accurately understand and respond to complex multi-condition queries such as "Find people wearing red shirts without work badges."

[0080] Enhanced naturalness and intelligence in human-computer interaction. The system can directly understand and process various open-ended questions posed by operators in natural language, such as "Were there any visitors at the South Gate without badges yesterday?" This transforms video retrieval, which previously required professional operation, into an intuitive question-and-answer interaction.

[0081] It is feasible for actual engineering deployment. Through algorithm optimization and architecture streamlining, the system significantly reduces computing resource requirements while maintaining high performance, providing reliable technical support for the implementation of intelligent monitoring in edge computing scenarios.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the scope of protection of the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A video semantic question answering system for surveillance scenarios, characterized by: include: The video acquisition module, as the basic input end, is composed of multiple cameras to form a monitoring network. Through standardized access and unified management, it collects video data in real time. The video preprocessing module optimizes the raw video data through three-level processing of keyframe extraction, noise reduction, and time alignment to improve the quality of subsequent analysis; The multimodal feature extraction module, including the video feature extraction channel, adopts an improved ViT architecture and builds a multi-scale attention mechanism. Through a spatial layering strategy, it extracts detailed features and global structures at different spatial resolution levels, and maintains the semantic flow between upper and lower layers through a residual connection mechanism. In the temporal dimension, the model introduces a cross-frame attention mechanism and position encoding to capture the semantic evolution of continuous behavior, and ultimately outputs the video feature V. And the text feature extraction channel: using the pre-trained language model to semantically encode the user input question to obtain the text feature T; The multimodal interactive reasoning module uses a cross-modal neural network model to build a cross-modal combination of video features and text features, achieving an effective combination of video data and text data; Answer generation module: predicts the answer based on the joint representation and outputs the corresponding structured answer branch and natural language branch; Interactive and visual interface module: supports interactive operations. Users can enter query content in text form, which will automatically parse the question, call the reasoning module and return multimodal visual results, forming a complete question-answering closed loop.

2. The video semantic question answering system for surveillance scenarios according to claim 1, characterized in that: The key frame extraction method uses a motion detection-based key frame extraction algorithm to extract representative frame sequences containing core information from the video stream, effectively reducing the amount of video data while retaining key spatiotemporal information. Denoising: Utilizing deep learning-based non-local mean denoising technology, this technology effectively removes image noise that may occur in low-light conditions, ensuring that image quality meets high standards in all environments. said time alignment; A time alignment algorithm is applied to correct the time delay differences between different cameras and ensure the temporal consistency of the video stream during cross-view analysis.

3. The video semantic question answering system for surveillance scenarios according to claim 1, characterized in that: The constructing of a cross-modal joint representation of video features and text features includes: Mapping the video features to a unified semantic space through a first projection matrix; Mapping the text features to the unified semantic space through a second projection matrix; Based on the text features as the query and the video features as the key and value, a cross-attention mechanism is used to calculate the association weights to generate an adjusted video semantic representation; The adjusted video semantic representation and text semantic representation are concatenated to form a cross-modal joint representation.

4. The video semantic question answering system for surveillance scenarios according to claim 1, characterized in that: The structured answer branch uses a trainable regression and classification network to predict the bounding box coordinates of the target in the video coordinate system for location query questions. Natural language branch: For open problems such as behavior description, a transformer-based sequence generation model is used to obtain natural language answers while retaining the corresponding evidence information to enhance the interpretability of the results.

Images