Universal scene retrieval analysis method and system based on multi-modal feature fusion

Abstract

The invention discloses a universal scene retrieval analysis method and system based on multi-modal feature fusion, the method comprises a video analysis step and an application service step, and the application service step comprises the steps of receiving a user input request, describing a multi-dimensional standardized video tag based on a video summary, and obtaining a multi-dimensional standardized video tag; the steps of cross-modal video retrieval, dynamic knowledge enhancement question answering and interactive enhancement analysis can be executed, efficient video preprocessing is achieved by constructing an offline feature library, the retrieval precision is improved by adopting a cross-modal feature fusion technology, the analysis authority is enhanced in combination with a dynamic knowledge base, and the interactive enhancement analysis is supported to achieve abnormal early warning. The method has the advantages that the offline video processing efficiency is improved, cross-modal feature fusion retrieval is realized, and the authority of an analysis result is enhanced.

Description

Technical Field

[0001] The present application relates to the field of video analysis and intelligent retrieval technology, and specifically to a general scene retrieval and analysis method and system based on multimodal feature fusion. Background Art

[0002] In the context of multimodal features, general scene retrieval and analysis usually have the following problems: First, existing technologies rely too much on online databases, resulting in low efficiency of video processing in offline scenarios and failure to meet real-time requirements; second, traditional methods lack an effective cross-modal feature fusion mechanism, making it difficult to achieve correlation analysis across video files, especially for the insufficient ability to locate related events scattered in different time periods; in addition, existing systems generally lack dynamic knowledge base support, resulting in a lack of authoritative basis for the interpretation of video events, affecting the accuracy of analysis results. These problems have seriously restricted the in-depth application of video analysis technology in security monitoring, smart cities and other fields. Existing technical solutions usually use single modal features for retrieval, which cannot effectively integrate multi-dimensional information such as spatiotemporal parameters and semantic labels, and lack intelligent linkage mechanisms with industry standards in terms of abnormal event warnings. In response to the above problems, existing technologies are in urgent need of improvement. Summary of the Invention

[0003] One of the purposes of this application is to provide a general scene retrieval and analysis method and system based on multimodal feature fusion, which has the advantages of improving offline video processing efficiency, realizing cross-modal feature fusion retrieval, and enhancing the authority of analysis results.

[0004] One of the objectives of the present invention is achieved through the following technical solutions: This general scene retrieval and analysis method based on multimodal feature fusion includes the following steps: A general scene retrieval and analysis method based on multimodal feature fusion, the method comprising a video analysis step and an application service step; wherein, The video analysis step includes preprocessing the offline video to generate structured video description information; the preprocessing includes: Extract frames from the video to obtain multiple frames of images; Performing image understanding on the multiple frames of image to generate a frame-level understanding description and a corresponding timestamp for each frame of image; Based on the frame-level understanding description and timestamp of each frame image, summarizing processing is performed to generate a video summary description of the entire video content; Extracting semantic labels from the multiple frames of image to obtain an initial label set, and processing the initial label set using a normalization cleaning algorithm to eliminate redundancy and form multi-dimensional standardized video labels covering scene types, components, and environmental parameters; The application service step includes receiving a user input request and performing at least one of the following operations based on the video summary description and the multi-dimensional standardized video tags: Cross-modal video retrieval: parses the semantic content of user queries, matches the query semantics with the video summary description and multi-dimensional standardized video tags based on the Euclidean distance algorithm, and outputs the associated video clips and timestamps; Dynamic knowledge-enhanced question answering: Retrieve video description information related to the user's question through RAG technology, associate it with normative information in the external knowledge base, and generate enhanced answers that integrate internal and external information; After executing the application service step, an optional interactive enhanced analysis step is also included. The interactive enhanced analysis step responds to the circle instruction initiated by the user in the result of the application service step, and calls the visual question and answer model to perform image understanding and question and answer on the specified target area, and returns the question and answer results to the user.

[0005] As an optional technical solution, the cross-modal video retrieval specifically includes: Analyze the spatiotemporal parameters and semantic content of user queries based on user prompt words; Filter candidate frames in the feature library based on the time range and location tags in the query; Calculate the Euclidean distance between the query semantic vector and the candidate frame feature vector; Output a list of video clips based on the similarity threshold, and annotate them with confidence and timestamp.

[0006] As an optional technical solution, the dynamic knowledge enhanced question answering includes: Use RAG technology to associate video analysis results with external knowledge bases; Retrieve the regulatory provisions and historical records related to the query event in the knowledge base; Fusion of video analysis results with external norms to generate enhanced answers.

[0007] As an optional technical solution, the interactive enhanced analysis includes: Retrieve all clips where the target appears in the entire video library based on feature similarity; Output the tracking results across videos in time series.

[0008] A second object of the present invention is to propose a general scene retrieval and analysis system based on multimodal feature fusion, the system comprising: Video analysis module, used to preprocess offline videos to generate structured video description information; An application service module is configured to receive user input requests and perform operations based on the structured information generated by the video analysis module. The application service module includes: A cross-modal video retrieval submodule is used to parse the semantic content of user queries, match the query semantics with the video summary description and multi-dimensional standardized video tags based on the Euclidean distance algorithm, and output the associated video clips and timestamps; The dynamic knowledge enhancement submodule is used to retrieve video description information related to the user's question through RAG technology, and associate it with the normative information in the external knowledge base to generate an enhanced answer that integrates internal and external information; The interactive enhanced analysis submodule is used to respond to the circle instruction initiated by the user, call the visual question-answering model to perform image understanding and question-answering on the target area, and return the question-answering results to the user.

[0009] As an optional technical solution, the video analysis module includes: A frame extraction unit, used to extract frames from the video to obtain multiple frames of images; An image understanding unit, configured to understand the plurality of frames of image, generate a frame-level understanding description and a corresponding timestamp for each frame of image, and perform summary processing to generate a video summary description; The label processing unit is used to extract semantic labels from the multiple frames of images and process the initial label set using a normalized cleaning algorithm to form multi-dimensional standardized video labels covering scene types, components and environmental parameters.

[0010] As an optional technical solution, the image understanding unit includes: The metadata parsing subunit is used to parse and extract the timestamp corresponding to each frame of the video stream; A feature extraction subunit, for extracting a depth feature vector of a frame image using a CLIP model; The semantic generation subunit is used to generate the frame-level understanding description and the video summary description by using a visual language model.

[0011] As an optional technical solution, the dynamic knowledge enhancement submodule performs the following when generating answers: Linked retrieval and query of relevant specifications and event records.

[0012] As an optional technical solution, the interactive enhanced analysis submodule includes: Target delineation tool, responding to user annotations to trigger enhanced analysis; An intelligent response unit, configured to call a visual question-answering model to perform image understanding and question-answering on the target area, and return the question-answering results to the user; The warning push unit automatically pushes warnings to the management terminal when abnormal target behavior is detected.

[0013] As an optional technical solution, the enhanced analysis further includes: The associated segment sorting unit is used to retrieve associated segments in the entire video library based on target features and output all video segments where the target appears in time sequence.

[0014] The beneficial effects of the present invention are: The present invention realizes efficient video preprocessing by constructing an offline feature library, improves retrieval accuracy by adopting cross-modal feature fusion technology, enhances analysis authority by combining with a dynamic knowledge base, and supports interactive enhanced analysis to realize abnormal warning. It has the advantages of improving offline video processing efficiency, realizing cross-modal feature fusion retrieval, and enhancing the authority of analysis results.

[0015] Other advantages, objects and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art based on an examination of the following or may be learned from the practice of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which: Figure 1 Schematic diagram of the method flow of the present invention; Figure 2 A logical diagram for realizing the overall functions of the present invention; Figure 3 A schematic diagram of the video retrieval logic of the present invention; Figure 4 It is a question-answering logic diagram of the present invention; Figure 5 Schematic diagram of video analysis logic of the present invention. DETAILED DESCRIPTION

[0017] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, and are not intended to limit the scope of protection of the present invention.

[0018] In existing technologies, general scene retrieval and analysis have long faced the challenges of low offline scene processing efficiency, difficulty locating events across time periods, and inaccurate event interpretation. Traditional methods rely on online databases for real-time processing, making them difficult to handle instability or large data volumes. Video retrieval often uses single-modal feature matching, which cannot effectively link related events scattered across different time periods. Event interpretation lacks professional and standardized support, limiting the credibility of analysis results. For example, in security monitoring scenarios, when investigators need to trace the target's movements across multiple days of surveillance footage, existing technologies are unable to quickly complete offline video analysis, accurately link clips across multiple cameras, and automatically reference relevant external standards to assist in event analysis.

[0019] To address these issues, we first proposed a solution for building a localized feature library, effectively resolving the conflict between feature extraction and storage. To address the challenge of cross-video retrieval, we discovered that multimodal feature fusion can overcome the limitations of single-modal retrieval. To address the issue of interpretation accuracy, we recognized that a dynamic knowledge association mechanism can enhance professional judgment. By organically combining offline feature library construction, cross-modal retrieval algorithms, and knowledge enhancement mechanisms, we have formed a complete offline analysis and intelligent interaction system.

[0020] Therefore, this application proposes a general scene retrieval analysis method based on multimodal feature fusion, such as Figure 1 and Figure 2 As shown, the following steps are included: The method includes a video analysis step and an application service step; wherein, The video analysis step includes preprocessing the offline video to generate structured video description information; the preprocessing includes: Extract frames from the video to obtain multiple frames of images; Performing image understanding on the multiple frames of image to generate a frame-level understanding description and a corresponding timestamp for each frame of image; Based on the frame-level understanding description and timestamp of each frame image, summarizing processing is performed to generate a video summary description of the entire video content; Extracting semantic labels from the multiple frames of image to obtain an initial label set, and processing the initial label set using a normalization cleaning algorithm to eliminate redundancy and form multi-dimensional standardized video labels covering scene types, components, and environmental parameters; The application service step includes receiving a user input request and performing at least one of the following operations based on the video summary description and the multi-dimensional standardized video tags: Cross-modal video retrieval: parses the semantic content of user queries, matches the query semantics with the video summary description and multi-dimensional standardized video tags based on the Euclidean distance algorithm, and outputs the associated video clips and timestamps; Dynamic knowledge-enhanced question answering: Retrieve video description information related to the user's question through RAG technology, associate it with normative information in the external knowledge base, and generate enhanced answers that integrate internal and external information; After executing the application service step, an optional interactive enhanced analysis step is also included. The interactive enhanced analysis step responds to the circle instruction initiated by the user in the result of the application service step, and calls the visual question and answer model to perform image understanding and question and answer on the specified target area, and returns the question and answer results to the user.

[0021] Among them, the frame extraction in the preprocessing adopts preset interval frame extraction. Preset interval frame extraction means dynamically adjusting the frame extraction frequency according to the complexity of the video content. A fixed time interval or a dynamic frame extraction algorithm based on scene changes can be used to ensure the balance between key frame coverage and processing efficiency; environmental parameters can be obtained by parsing video metadata through the OpenCV library; RAG technology integrates the knowledge base retrieval results into the question and answer generation process through the retrieval enhancement generation mechanism.

[0022] Specifically, when building an offline feature library (the feature library can be understood as Figure 1 The video library and video analysis are two parts in it. The video library stores the original video files, while the video analysis stores the relevant feature data obtained after analysis). The preset frame extraction strategy balances information integrity and storage overhead, and simultaneously extracts image visual features, environmental parameters and semantic labels to form a structured feature index; in the cross-modal retrieval stage, after the spatiotemporal parameter screening narrows the retrieval scope, the Euclidean distance algorithm is used to realize the joint matching of semantic content and multimodal features, and output related clips across videos; in the dynamic knowledge enhanced question and answer process, RAG technology retrieves the regulatory provisions related to the event in real time, and integrates them with the video analysis results to generate a professional report; interactive enhanced analysis supports users to circle suspicious targets, realize cross-video tracking through feature matching, and detect abnormal behavior in combination with the preset rule library.

[0023] Compared with existing technologies, traditional methods rely on online databases, resulting in processing delays. This solution utilizes a local feature library for efficient offline processing. Existing single-modal retrieval struggles to correlate events across time periods, while this solution improves retrieval accuracy through multimodal feature fusion. Existing question-answering systems lack standardized support, while this solution leverages RAG technology to dynamically enhance knowledge. In particular, in cross-camera tracking scenarios, existing technologies require manual comparison of multiple video streams. This solution automatically outputs all associated segments and time series of target occurrences.

[0024] Through the above technical solution, this application effectively improves offline scene processing efficiency, achieves multi-modal precision retrieval across videos, and enhances the professionalism and credibility of event interpretation. For example, in security investigation scenarios, investigators can quickly locate the target object's activity trajectory at different monitoring points and automatically generate analysis reports that comply with industry standards. In equipment inspection scenarios, operation and maintenance personnel can retrieve historical similar events by simply entering the abnormal component and quickly determine the fault level based on the knowledge base specifications.

[0025] The specific implementation methods of each step of the above method will be further explained through the following embodiments.

[0026] Example 1 like Figure 3 As shown, in this embodiment, it is further proposed to parse the spatiotemporal parameters and semantic content of the user query based on the user prompt words; filter the candidate frames in the feature library according to the time range and location tags in the query; calculate the Euclidean distance between the query semantic vector and the candidate frame feature vector; output the video clip list according to the similarity threshold, and mark the confidence level and timestamp.

[0027] User prompt word parsing refers to extracting the spatiotemporal parameters and semantic content in queries through natural language processing technology. Specifically, the BERT model can be used to vectorize text features, thereby accurately capturing the multi-dimensional needs of user queries. Time range and location tag screening refers to quickly locating candidate data in the feature library based on query conditions. Specifically, database indexing technology can be used to achieve joint retrieval of spatiotemporal parameters, thereby reducing subsequent computational complexity. Similarity threshold filtering refers to setting a dynamic threshold to filter highly relevant results. Specifically, statistical distribution analysis can be used to determine the optimal cutoff value to prevent low-quality fragments from interfering with the output results.

[0028] Specifically, the text query entered by the user is first converted into a vector representation containing spatiotemporal parameters and semantic features. The video frame data stored in the feature library is indexed using spatiotemporal labels, enabling the candidate frame screening process to quickly locate a subset that meets the time range and location criteria. During the semantic matching phase, the query vector and the candidate frame feature vectors are calculated using the Euclidean distance algorithm to obtain a relevance score. The score results are then filtered through a threshold to generate a ranked list. The final output video clip list contains not only timestamp information but also a confidence indicator to assess the reliability of the retrieval results.

[0029] Compared with existing technologies, traditional video retrieval methods usually rely only on single modal features or fixed time windows for matching, resulting in inefficient cross-time event location. The similarity calculation based on full database traversal in existing technologies generates a large number of redundant operations and lacks a joint spatiotemporal label screening mechanism, making it difficult to meet the retrieval needs of large-scale video data. This solution, through a hierarchical retrieval strategy, first narrows the candidate range based on spatiotemporal labels and then performs semantic matching calculations, significantly reducing computational complexity. At the same time, the dynamic threshold mechanism can adapt to the data distribution of different scenarios, avoiding retrieval bias caused by manually setting fixed thresholds.

[0030] Through the above technical solutions, this application solves the problems of high computing resource consumption and low result relevance during cross-video event positioning. The joint screening of spatiotemporal labels effectively reduces the size of candidate data, thereby improving the efficiency of subsequent semantic matching calculations. The Euclidean distance algorithm is combined with dynamic threshold filtering to ensure that the output video clips meet both spatiotemporal constraints and have high semantic relevance. The confidence annotation mechanism provides a quantitative basis for the credibility of the results, helping users to quickly judge the effectiveness of the retrieval. The multimodal feature fusion strategy realizes event association analysis across video clips, meeting the needs of precise positioning in complex scenarios.

[0031] Example 2 like Figure 4 As shown, in this embodiment, a method for implementing dynamic knowledge-enhanced question and answer is further proposed, including establishing an association mechanism between video analysis results and an external knowledge base through retrieval enhancement generation technology, retrieving normative clauses and historical records related to the query event in the knowledge base, and fusing the video analysis results with external specifications to generate a question and answer report. Among them, retrieval enhancement generation technology refers to an artificial intelligence technology that combines semantic retrieval and text generation. Specifically, it can be implemented by jointly using a pre-trained language model and a vector database to extract relevant information from the knowledge base through semantic matching and generate structured text.

[0032] Among them, the external knowledge base refers to a standardized document storage system independent of the video feature library. It can be implemented using a hybrid architecture of relational database and document database, and is used to store structured and unstructured data such as industry standards and historical event records.

[0033] Among them, normative provisions and historical records refer to industry standard documents related to the query event and similar event handling cases. They can be screened through keyword matching and semantic similarity calculation to ensure the accuracy and timeliness of knowledge references.

[0034] Among them, the question and answer report refers to a composite output document that contains video analysis conclusions and regulatory basis. It can be implemented using template filling and natural language generation technology, and a double verification mechanism is used to ensure the objectivity and compliance of the content. Specifically, when a user initiates an event query, the system first analyzes the key elements in the video analysis results, and dynamically retrieves the current specifications and historical processing records related to the event from the external knowledge base through retrieval enhancement generation technology. The retrieval process uses a semantic vector matching algorithm to calculate the similarity between the query content and the knowledge base entries, and screens out the specification clauses with a relevance higher than the preset threshold. Subsequently, the system cross-validates the objective facts obtained from the video analysis with the retrieved specification requirements, and integrates the verification results into a question-and-answer report containing factual descriptions, specification references, and processing suggestions through a text generation model. During this process, the update status of the knowledge base affects the retrieval results in real time, ensuring that the referenced specification version is always the latest valid standard. Compared to existing technologies, traditional approaches rely on static knowledge bases, resulting in delayed regulatory citations and an inability to dynamically link video analysis results with external knowledge. This solution leverages retrieval-enhanced generation technology to dynamically access the knowledge base, ensuring a precise alignment between regulatory retrieval scope and video analysis content. It also incorporates historical records as a supporting decision-making tool, effectively eliminating interpretation errors caused by delayed knowledge updates or deviations in scene understanding. Through the above technical solution, this application addresses the issue of interpretation accuracy caused by a lack of dynamic knowledge support. It utilizes a dual verification mechanism to compare video analysis results with external specifications in real time, ensuring that event interpretations align with both objective facts and industry standards. By dynamically linking to the latest specification versions, it avoids compliance risks arising from lagging knowledge base updates and enhances the professionalism and credibility of Q&A reports.

[0035] Example 3 like Figure 5 As shown in the figure, the specific steps of video analysis are revealed. For each frame of image, the system performs multiple analysis tasks in parallel to extract rich semantics and information, including: (1) Video frame extraction: The system extracts frames from the input video file at a preset time interval (e.g., 1 frame per second), discretizing the continuous video stream into a series of continuous image frames.

[0036] (2) Frame-level image understanding: Use large-scale visual language models (such as GPT-Vision, CLIP, BLIP-2, etc.) to perform deep semantic understanding of a single frame image and generate a natural language description of the frame (i.e., "each frame image understanding description"), for example, "a man in blue clothes is approaching a white door."

[0037] (3) Environmental parameter analysis: Using a computer vision library (e.g., OpenCV), the timestamps embedded in video frames or metadata are analyzed to accurately obtain the shooting time (date and time) of each frame. In some embodiments, the model can also be used to analyze the first frame or key frames to infer comprehensive environmental information (e.g., weather "sunny", lighting "nighttime", location "intersection") to form a "video environment" summary.

[0038] Semantic label annotation: Use advanced visual recognition models (such as ResNet, ViT, VLM) or semantic segmentation models to identify objects, scenes, and activities in the image and generate a series of preliminary semantic labels for each frame (such as [pedestrian, blue shirt, white gate, walking]).

[0039] (4) Video-level information integration and refinement: Based on the frame-level analysis, a higher-level integration is performed to form a macroscopic description of the entire video file. This includes: (4.1) Video description generation: Based on the understanding description of all frames, summary processing is performed through a temporal model or a large language model to generate a coherent "video description" that summarizes the entire video content.

[0040] (4.2) Label system construction and cleaning: a. Label extraction: Gather the preliminary semantic labels of all frames to form an initial set of labels and the frame indexes in which they appear.

[0041] b. Label cleaning and normalization: The system uses a dedicated normalization cleaning algorithm to process the initial label set. Its purpose is to: Eliminate redundancy: merge synonyms (such as "car" and "sedan") and antonyms.

[0042] Hierarchical organization: Build a structured labeling system that covers at least the following dimensions: Scene Type: such as "traffic intersection", "indoor lobby", and "perimeter protection".

[0043] Object / Parts: such as "vehicle", "face", "license plate", and "tire".

[0044] Environmental Parameters: such as "Visibility: Low", "Temperature: High", "Weather: Rain or Snow".

[0045] (4.3) Ensure accuracy and interpretability: After cleaning and normalization, the final output is an accurate and structured set of "video tags", which can greatly improve the accuracy and efficiency of subsequent retrieval.

[0046] (5) Feature library storage: Ultimately, all extracted and generated structured information, including image feature vectors, frame understanding descriptions, timestamps, video descriptions, video environments, cleaned video label sets and their correspondence with frames, are uniformly stored in the local offline feature library.

[0047] Video analysis is the foundation of the overall solution of the present invention, which aims to transform unstructured raw video data into a structured, machine-understandable and retrievable multimodal feature information library.

[0048] Example 4 This embodiment further proposes receiving a user's instruction to circle a target area in a video frame, using the DetectWithAttributes algorithm to extract features of the circled area, retrieving all clips in which the target appears in the entire library of videos based on feature similarity, and outputting cross-video tracking results in a time series. Among them, receiving the user's instruction to circle the target area in the video frame refers to obtaining the coordinate information of the user's focus area through an interactive annotation tool. Specifically, it can be achieved by rectangular box selection or free drawing. This feature enables the analysis process to focus on a specific target area. Using the DetectWithAttributes algorithm to extract the features of the circled area refers to simultaneously extracting the visual features and semantic attributes of the target. Specifically, it can be achieved by jointly outputting the target detection model and the attribute classification model. This feature enhances the target representation ability by generating a composite feature vector. Retrieving all clips where the target appears in the entire video library based on feature similarity refers to matching the extracted feature vectors with all video frames in the offline feature library for similarity. Specifically, it can be achieved by using the Euclidean distance algorithm threshold screening method. This feature breaks through the limitations of single video analysis and realizes cross-file retrieval. Outputting cross-video tracking results in time series refers to sorting and reorganizing the retrieved discrete video clips by timestamps. Specifically, it can be achieved by using a timeline alignment algorithm. This feature forms a continuous behavior trajectory to restore the complete event chain. Specifically, when a user circles a target area in a video frame, the coordinate information of that area is transmitted to the feature extraction module. For example, the DetectWithAttributes algorithm can be used to superimpose attribute analysis on target detection, extracting features such as the target's color, shape, and motion state to generate a feature vector containing multidimensional information. This vector is input into the offline feature library for full comparison. By calculating the similarity score with the features of each video frame, candidate segments whose similarity exceeds a preset threshold are screened. All eligible segments are sorted by the original video timestamp, and a time series tracking report across the video is ultimately generated. Compared with existing technologies, traditional methods typically rely on fixed algorithms for automatic target identification, are unable to handle user-defined regions of interest, and rely solely on single visual features when searching across videos, resulting in a high false positive rate. This solution introduces user-defined delineation instructions to enable human-machine collaborative analysis, combines the DetectWithAttributes algorithm to generate composite feature vectors to improve search accuracy, and uses full-database search and time series reconstruction techniques to address the problem of disconnected events across time periods. Through the above technical solution, this application can accurately lock onto a user-specified target, quickly locate all spatiotemporal segments where the target appears in an offline video library, and integrate the scattered search results into a continuous behavioral trajectory. This solution effectively improves the ability to correlate and analyze cross-video events, providing a complete spatiotemporal evidence chain for tracking abnormal behavior.

[0049] Example 5 Based on the design principles of the previous method, this embodiment further proposes a system comprising a video analysis module and an application service module, wherein the application service module includes a cross-modal video retrieval submodule, a dynamic knowledge enhancement submodule, and an interactive enhanced analysis submodule. The video analysis module is used to preprocess offline videos to generate structured video description information; the application service module is used to receive user input requests and perform operations based on the structured information generated by the video analysis module. The functions of each submodule of the application service module are as follows: The cross-modal video retrieval submodule is used to parse the semantic content of the user query and match the query semantics with the video summary description and multi-dimensional standardized video tags based on the Euclidean distance algorithm, and output the associated video clips and timestamps; The dynamic knowledge enhancement submodule is used to retrieve video description information related to user questions through RAG technology, and associate it with normative information in the external knowledge base to generate enhanced answers that integrate internal and external information; The interactive enhanced analysis submodule is used to respond to the circle instruction initiated by the user, call the visual question-answering model to perform image understanding and question-answering on the target area, and return the question-answering results to the user.

[0050] In this embodiment, the video analysis module refers to the unit that performs structured processing on the raw video. Specifically, this can be achieved by extracting frames at preset intervals, using the CLIP model to extract image features, parsing timestamps with OpenCV to obtain environmental parameters, and generating semantic labels using the VLM model. This module is used to build an offline feature library, eliminating reliance on online databases. This module supports custom configuration, allowing for the setting of a default frame extraction interval. In specific implementations, a normalized cleaning algorithm can be used to uniformly label scene types, device components, and temperature and humidity parameters, achieving data standardization to eliminate feature differences in multi-source data.

[0051] The cross-modal video retrieval submodule refers to the core algorithm module for implementing cross-video retrieval. Specifically, it can be implemented by combining spatiotemporal feature fusion technology with Euclidean distance calculation to solve the problem of difficulty in locating events across time periods.

[0052] The dynamic knowledge enhancement submodule refers to the knowledge enhancement processing unit, which can be implemented by using RAG technology to establish vector associations between video features and external knowledge bases. By jointly searching the video analysis results with the regulatory provisions, it is used to improve the accuracy of event interpretation.

[0053] The interactive enhanced analysis sub-module includes a human-computer collaborative analysis interface, supports multi-screen interaction, and supports adaptation to multiple terminals such as PC / tablets. It provides conventional interactive components such as a floating search box and a dynamic loading list. More importantly, it provides a target delineation tool. Specifically, it can use the DetectWithAttributes algorithm to extract user-annotated area features and perform cross-video tracking to trigger enhanced analysis and push warning information.

[0054] Specifically, the video analysis module extracts video frames at fixed intervals, such as one frame every 1 or 5 seconds, and extracts image feature vectors using the CLIP model. It also parses the GPS coordinates and timestamps in the video metadata as environmental parameters and then uses the BERT model to generate semantic description tags. The cross-modal video retrieval submodule receives the temporal and spatial conditions and semantic keywords input by the user, screens the feature library for candidate frames that meet the time range and location tags, concatenates the query semantic vector with the candidate frame feature vector, calculates the Euclidean distance, and outputs the associated fragment when the threshold is exceeded. The dynamic knowledge enhancement submodule simultaneously retrieves video analysis results and external knowledge bases during the question-answering process. For example, it associates and matches detected equipment anomalies with industry safety regulations to generate an analysis report containing regulatory references. The multi-dimensional label processing module cleans the original labels, for example, converting expressions such as "high temperature" and "hot" into a standardized label such as "temperature ≥ 35°C." The intelligent interaction module responds to the target area circled by the user on the video frame, extracts the color histogram and texture features of the area, and searches for the same target in all videos based on feature similarity.

[0055] Through the above technical solutions, this application realizes the efficient processing capability of offline videos, supports the completion of feature extraction and retrieval tasks in an environment without network connection; enhances the positioning accuracy of cross-time events, and can accurately identify related video clips scattered in different time periods; establishes a dynamic knowledge association mechanism, and automatically quotes the latest regulatory provisions during the event analysis process; unifies the multi-source data labeling standards, and eliminates retrieval errors caused by expression differences; provides human-computer collaborative analysis functions, and realizes closed-loop processing of abnormal events through target feature tracking.

[0056] Example 6 This embodiment is an extension and expansion of the technical content of Example 5. This embodiment further proposes that the video processing module includes a frame extraction unit, an image understanding unit and a label processing unit, wherein the frame extraction unit is used to extract frames from the video to obtain multiple frames of images; the image understanding unit is used to understand the multiple frames of images, generate a frame-level understanding description and a corresponding timestamp for each frame of the image, and perform summary processing to generate a video summary description; the label processing unit is used to extract semantic labels from the multiple frames of images, and use a normalized cleaning algorithm to process the initial label set to form multi-dimensional standardized video labels covering scene types, components and environmental parameters.

[0057] Furthermore, in this embodiment, the image understanding unit includes a metadata parsing subunit, a feature extraction subunit and a semantic generation subunit, wherein the metadata parsing subunit is used to parse and extract the timestamp corresponding to each frame image from the video stream, such as using a computer vision library (such as OpenCV) to extract timestamp metadata from the video file; the feature extraction subunit is used to extract the depth feature vector of the frame image using a model (such as CLIP); and the semantic generation subunit is used to generate a frame-level understanding description and a video summary description using a visual language model (such as VLM).

[0058] Among them, the frame extraction adopted by the frame extraction unit can adopt the method of preset time interval. The preset time interval refers to the frame extraction frequency dynamically adjusted according to the video content. Specifically, it can be implemented by a fixed interval or a dynamic interval strategy based on scene changes to balance data coverage and processing efficiency; the CLIP model is used to extract high-dimensional visual features of the image; OpenCV parsing timestamp refers to associating environmental parameters with time information in video metadata to establish a mapping relationship between spatiotemporal dimensions and the physical environment; the label processing unit can use the BERT model to generate semantic labels, that is, a semantic understanding model based on natural language processing, which specifically uses text descriptions to automatically generate standardized labels to eliminate subjective biases in manual labeling.

[0059] Specifically, the frame extraction unit reduces the number of redundant video frames through a preset interval strategy, providing structured input for subsequent processing. In the feature extraction unit, the CLIP model extracts deep image features through a residual structure, addressing the problem of traditional models' insufficient representation of complex scenes. OpenCV parses timestamps and associates them with environmental parameters such as lighting and temperature to form a multi-dimensional description of spatiotemporal physical properties. The BERT model performs semantic understanding of video content and generates text labels in a unified format. The three work together to convert video data into a multimodal feature vector containing visual features, environmental parameters, and semantic labels, constructing a standardized offline feature library.

[0060] Through the above technical solution, this application realizes the automated feature extraction and standardized processing of offline video data, solving the problem of low processing efficiency caused by relying on online databases. By fusing multimodal features to build a structured feature library, it provides a unified data foundation for subsequent cross-modal retrieval, while eliminating the subjective errors caused by manual annotation and improving the scalability of offline scene analysis.

[0061] Example 7 This embodiment further proposes a working method for the cross-modal video retrieval submodule. First, a preliminary screening is performed in the feature library based on the time range and location information of the user's query, and the query semantics and video frame features are matched using the Euclidean distance algorithm. A list of video clips sorted by confidence and the corresponding timestamps are returned.

[0062] The cross-modal video retrieval submodule refers to a multi-dimensional matching system that integrates text semantics and image features. It can be implemented using a multimodal feature fusion framework to simultaneously process association analysis of different data types. The time range and location information of the user query refers to the retrieval conditions that include time intervals and geographic coordinates. It can be implemented using a timestamp parsing module and a location tag matching algorithm to quickly filter irrelevant data in the feature library. Confidence sorting refers to prioritizing results based on matching similarity. It can be implemented using a threshold filtering and sorting algorithm to ensure the quality controllability of the output results.

[0063] Specifically, when the user inputs a query condition containing a time interval and geographic coordinates, the system first performs a preliminary screening of the spatiotemporal dimensions in the offline feature library, such as filtering out video frame data within a specific date range and located in a specified area. Subsequently, a cross-modal similarity calculation is performed between the text semantic vector entered by the user and the image feature vector of the candidate video frame, where the text semantic vector can be generated by a natural language processing model and the image feature vector is extracted by a convolutional neural network. The vector space distance between the two is calculated using the Euclidean distance algorithm to obtain a matching score for each candidate frame. The system filters low-scoring results based on a preset confidence threshold, and sorts the qualified results in descending order of score, while associating the timestamp information of the original video to form a spatiotemporal correlation event chain containing multiple video clips.

[0064] Through the above technical solution, this application can effectively solve the problem of cross-video event location difficulties, reduce data processing volume through spatiotemporal filtering mechanisms, improve the accuracy of fuzzy queries through cross-modal feature matching, and ultimately output a chain of related events with time series annotations. This solution allows users to quickly obtain related event fragments scattered across different video files, providing reliable data support for subsequent behavioral analysis and decision-making.

[0065] Example 8 This embodiment further proposes that the dynamic knowledge base module performs linkage retrieval of specifications and event records related to the query when generating an answer. Among them, linked retrieval refers to actively calling external knowledge base resources through multi-dimensional retrieval strategies. Specifically, it can be implemented by using a retrieval algorithm based on semantic relevance. By simultaneously matching keywords and event context relevance, it ensures that the retrieval scope covers industry regulations and historical event records.

[0066] Furthermore, a structured analysis report can be generated by combining video analysis results and knowledge base information. The generation of a structured analysis report refers to the cross-modal alignment of video features with external knowledge. Specifically, this can be achieved by using a knowledge fusion algorithm. By spatially mapping the video feature vector with the knowledge base text vector, an association relationship between video behavior and regulatory provisions can be established. Specifically, after receiving a user query request, the dynamic knowledge base module first extracts the core elements of the query through semantic analysis, such as event type, time range, spatial location and other parameters. Based on these parameters, the module initiates a linkage retrieval mechanism to search in parallel in the external knowledge base for industry regulations that match the event type, as well as recorded cases with similar spatiotemporal attributes in the historical database. A multi-dimensional matching strategy is adopted in the retrieval process. For example, for the query "wearing of safety helmets at construction sites", not only keywords such as "safety helmets" and "construction" are matched, but also extended concepts such as "height work" and "labor protection" are associated. After completing the retrieval, the module fuses the feature data obtained from the video analysis with the retrieval results. For example, the coordinate information of people not wearing safety helmets detected in the video is associated with Article 3.2.1 of the "Technical Code for Construction Safety". A structured report containing references to illegal clauses and comparisons with similar historical cases is generated through the knowledge fusion algorithm. Compared with existing technologies, traditional video analysis systems rely solely on internal feature databases for event interpretation, lacking external knowledge support, resulting in conclusions that lack a normative basis. This solution uses a multi-dimensional retrieval strategy that integrates event context to expand the search scope. For example, when searching for safety helmet specifications, it simultaneously links labor protection equipment inspection procedures. In the report generation process, existing technologies often use a simple result splicing method, while this solution uses cross-modal vector alignment technology to achieve deep correlation between video detection data and textual clauses. For example, it establishes a mapping relationship between frame sequences of people not wearing safety helmets and specific clauses in the regulations. Through the above-mentioned technical solution, this application solves the problem of limited accuracy in event interpretation due to the lack of dynamic knowledge base support, and effectively links video analysis conclusions with industry standards. Specifically, this solution automatically matches video behavior with relevant regulations and generates analysis conclusions with clear regulatory basis, avoiding omissions or misjudgments that may occur through manual comparison. Furthermore, through correlation analysis of historical cases, it provides reference examples for event interpretation, enhancing the credibility and practicality of the conclusions.

[0067] Embodiment 9 This embodiment further proposes that the intelligent interaction module includes a target delineation tool, an intelligent response unit, and an early warning push unit. The target delineation tool is used to trigger enhanced analysis in response to user annotations; the intelligent response unit is used to call a visual question-answering model to perform image understanding and question-answering on the target area and return the question-answering results to the user; and the early warning push unit is used to automatically push an early warning to the management terminal when abnormal target behavior is detected. The triggering condition for enhanced analysis is that after performing a search or viewing the question and answer results, the user has doubts about certain content (such as discovering a suspicious target or event) and actively initiates a request through the interactive interface (such as the target circle tool).

[0068] The execution process includes: The system extracts visual features based on the specific target or area circled by the user. Subsequently, rather than simply performing a similarity search across the entire database, the target and its contextual information (the image frame, existing labels, etc.) are submitted to a large-scale visual language model for specialized "image understanding question answering." The large model performs in-depth reasoning and analysis based on the specific target (this may involve the system invoking the model to extract the visual features and semantic attributes of the area, generating a feature vector containing information such as color distribution, shape outline, and object category. Specifically, a specific model can be used to extract vector features of the circled area, as described in the feature extraction method described in the previous embodiment). The system then answers the user's specific question (e.g., "What is this person doing?", "What model is this vehicle?", "Is this behavior unusual?"). Output and Response: The system returns the large model's question-and-answer results to the user. Meanwhile, the system backend continuously monitors the results of this analysis. If it is determined to meet pre-set anomaly criteria (e.g., identifying "illegal intrusion"), an alert is automatically sent to the management terminal.

[0069] This solution uses a target delineation tool to trigger enhanced analysis through user delineation instructions, combining human experience with algorithm feature extraction to effectively improve the pertinence of target features; through the early warning push unit, early warning signals are discovered and pushed in a timely manner, ensuring the timeliness of monitoring.

[0070] Example 10 This embodiment further proposes that the enhanced analysis also includes a related segment sorting unit, which is used to retrieve related segments in the entire video library based on the target features of the circled area, and output all video segments where the target appears in time series. Among them, full-library video retrieval refers to cross-file matching based on a pre-built multimodal feature library. Specifically, feature vector comparison can be achieved through the Euclidean distance algorithm, and correlation analysis across video files can be achieved using the spatiotemporal labels of the offline feature library.

[0071] Time series output refers to the temporal reorganization of retrieval results, which can be achieved by using timestamp alignment technology. By establishing a spatiotemporal coordinate chain of the target's appearance time, a complete trajectory tracking record is formed. Time series output strengthens the continuity of the behavior trajectory and provides structured time series data for anomaly detection. Specifically, when the user circles the target area in the video frame through the interactive interface, the system calls the model to extract the visual features and semantic attributes of the area, and generates a feature vector containing information such as color distribution, shape outline and object category (the model can be used to extract the vector features of the circled area, and the feature extraction method of the aforementioned embodiment can be referred to for details). The feature vector is input into the pre-built multimodal feature library and matched with the feature data of all offline video frames for similarity. During the matching process, the Euclidean distance algorithm is used to calculate the Euclidean distance between feature vectors. The smaller the distance, the more similar it is. When the preset threshold is exceeded, the system records the timestamp and location information of the corresponding video clip. All qualified retrieval results are sorted and reorganized according to the timestamps, and the time axis alignment technology is used to connect the target appearance times scattered in different videos, and finally a spatiotemporal trajectory map across videos is generated.

[0072] Compared with existing technologies, traditional video analysis systems only support single-file retrieval and rely on online feature extraction, making cross-video correlation analysis in offline scenarios impossible. This solution utilizes a pre-built multimodal feature library and an offline similarity matching mechanism to perform cross-file retrieval without requiring a real-time database connection. It also uses interactive feature extraction instead of full-frame computation, focusing computing resources on the user's area of ​​interest and significantly increasing target tracking efficiency. Timestamp alignment technology breaks through the traditional file-based retrieval model, enabling true cross-temporal and spatial event correlation. Through the above technical solutions, this application effectively solves the spatiotemporal fragmentation problem in cross-video event tracking, enabling rapid location of target object appearances in different time periods in an offline environment. The interactive feature extraction mechanism reduces computational load while improving retrieval accuracy. The time series output method intuitively presents the target's movement trajectory, providing a complete spatiotemporal data chain for cross-period event analysis. The time series output mechanism converts discrete retrieval results into structured behavior records, improving the completeness and interpretability of abnormal event analysis.

[0073] Finally, it should be noted that the above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as above in the form of a preferred embodiment, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modifications to equivalent embodiments of the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A general scene retrieval and analysis method based on multimodal feature fusion, characterized by: The method includes a video analysis step and an application service step; wherein, The video analysis step includes preprocessing the offline video to generate structured video description information; the preprocessing includes: Extract frames from the video to obtain multiple frames of images; Performing image understanding on the multiple frames of image to generate a frame-level understanding description and a corresponding timestamp for each frame of image; Based on the frame-level understanding description and timestamp of each frame image, summarizing processing is performed to generate a video summary description of the entire video content; Extracting semantic labels from the multiple frames of image to obtain an initial label set, and processing the initial label set using a normalization cleaning algorithm to eliminate redundancy and form multi-dimensional standardized video labels covering scene types, components, and environmental parameters; The application service step includes receiving a user input request and performing at least one of the following operations based on the video summary description and the multi-dimensional standardized video tags: Cross-modal video retrieval: parses the semantic content of user queries, matches the query semantics with the video summary description and multi-dimensional standardized video tags based on the Euclidean distance algorithm, and outputs the associated video clips and timestamps; Dynamic knowledge-enhanced question answering: Retrieve video description information related to the user's question through RAG technology, associate it with normative information in the external knowledge base, and generate enhanced answers that integrate internal and external information; After executing the application service step, an optional interactive enhanced analysis step is also included. The interactive enhanced analysis step responds to the circle instruction initiated by the user in the result of the application service step, and calls the visual question and answer model to perform image understanding and question and answer on the specified target area, and returns the question and answer results to the user.

2. The general scene retrieval and analysis method based on multimodal feature fusion according to claim 1, characterized in that: The cross-modal video retrieval specifically includes: Analyze the spatiotemporal parameters and semantic content of user queries based on user prompt words; Filter candidate frames in the feature library based on the time range and location tags in the query; Calculate the Euclidean distance between the query semantic vector and the candidate frame feature vector; Output a list of video clips based on the similarity threshold, and annotate them with confidence and timestamp.

3. The general scene retrieval and analysis method based on multimodal feature fusion according to claim 1, characterized in that: The dynamic knowledge enhanced question answering includes: Use RAG technology to associate video analysis results with external knowledge bases; Retrieve the regulatory provisions and historical records related to the query event in the knowledge base; Fusion of video analysis results with external norms to generate enhanced answers.

4. The general scene retrieval and analysis method based on multimodal feature fusion according to claim 1, characterized in that: The interactive enhanced analysis further includes: Retrieve all clips where the target appears in the entire video library based on feature similarity; Output the tracking results across videos in time series.

5. A general scene retrieval and analysis system based on multimodal feature fusion, characterized in that: The system comprises: Video analysis module, used to preprocess offline videos to generate structured video description information; An application service module is configured to receive user input requests and perform operations based on the structured information generated by the video analysis module. The application service module includes: A cross-modal video retrieval submodule is used to parse the semantic content of user queries, match the query semantics with the video summary description and multi-dimensional standardized video tags based on the Euclidean distance algorithm, and output the associated video clips and timestamps; The dynamic knowledge enhancement submodule is used to retrieve video description information related to the user's question through RAG technology, and associate it with the normative information in the external knowledge base to generate an enhanced answer that integrates internal and external information; The interactive enhanced analysis submodule is used to respond to the circle instruction initiated by the user, call the visual question-answering model to perform image understanding and question-answering on the target area, and return the question-answering results to the user.

6. The general scene retrieval and analysis system based on multimodal feature fusion according to claim 5, characterized in that: The video analysis module includes: A frame extraction unit, used to extract frames from the video to obtain multiple frames of images; An image understanding unit, configured to understand the plurality of frames of image, generate a frame-level understanding description and a corresponding timestamp for each frame of image, and perform summary processing to generate a video summary description; The label processing unit is used to extract semantic labels from the multiple frames of images and process the initial label set using a normalized cleaning algorithm to form multi-dimensional standardized video labels covering scene types, components and environmental parameters.

7. The general scene retrieval and analysis system based on multimodal feature fusion according to claim 6, characterized in that: The image understanding unit includes: The metadata parsing subunit is used to parse and extract the timestamp corresponding to each frame of the video stream; A feature extraction subunit, for extracting a depth feature vector of a frame image using a CLIP model; The semantic generation subunit is used to generate the frame-level understanding description and the video summary description by using a visual language model.

8. The general scene retrieval and analysis system based on multimodal feature fusion according to claim 5, characterized in that: The dynamic knowledge enhancement submodule performs the following when generating answers: Linked retrieval and query of relevant specifications and event records.

9. The general scene retrieval and analysis system based on multimodal feature fusion according to claim 5, characterized in that: The interactive enhanced analysis submodule includes: Target delineation tool, responding to user annotations to trigger enhanced analysis; An intelligent response unit, configured to call a visual question-answering model to perform image understanding and question-answering on the target area, and return the question-answering results to the user; The warning push unit automatically pushes warnings to the management terminal when abnormal target behavior is detected.

10. The general scene retrieval and analysis system based on multimodal feature fusion according to claim 9, characterized in that: The enhanced analysis also includes: The associated segment sorting unit is used to retrieve associated segments in the entire video library based on target features and output all video segments where the target appears in time sequence.

Images