Video file management system based on two-dimensional analysis and latitude and longitude closed loop verification

Abstract

The application discloses a kind of video file management systems based on two-dimensional analysis and latitude and longitude closed loop verification, system includes equipment resource acquisition module, generates global unique equipment identifier;Video frame acquisition and pretreatment module, generates associated video frame identifier, forms double unique association key;Large model two-dimensional analysis and verification module, executes OSD analysis and scene semantic analysis, and closed loop matching verification is carried out in combination with equipment latitude and longitude;Multi-source data fusion and large model dynamic adaptation module, equipment information, analysis result and verification result are fused into standardized knowledge data;Knowledge base construction and operation maintenance module, constructs and maintains large model knowledge base;Question and answer interaction module supports oral natural language question and generates answer, realizes video resource whole-process automation, standardization management through double key association, two-dimensional analysis and latitude and longitude verification, improves analysis accuracy and query efficiency, reduces use threshold, and has perfect operation and maintenance control mechanism.

Description

Technical Field

[0001] This invention relates to the field of network data management technology, and in particular to a video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification. Background Technology

[0002] With the widespread adoption of video surveillance and media resource archiving, video resources are experiencing explosive growth. The maturity of multimodal large models provides a technological foundation for the automated parsing of video content. Meanwhile, the development of technologies such as vector databases, distributed storage, and NLP (Natural Language Processing) also supports the structured management and rapid retrieval of video resources.

[0003] The industry's current demand for video resources has evolved from "simple storage" to "efficient parsing, fast querying, and data reuse." However, traditional video management systems only achieve the acquisition and storage of video streams, lacking in-depth analysis and structured processing of video content. This fails to meet enterprises' needs for rapid extraction and cross-scenario reuse of video resource information. The application of large models in the video field is also in its early stages, mostly involving single-stage parsing applications, without forming a standardized solution for the entire process. Summary of the Invention

[0004] The purpose of this invention is to provide a video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification to solve the problems existing in the prior art.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution: A video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification includes: The device resource acquisition module is configured to acquire device information of video acquisition devices and generate a globally unique device identifier (device_id) for each device. The device information includes at least the device's latitude and longitude. The video frame acquisition and preprocessing module is configured to acquire video frames from the video acquisition device, generate a globally unique video frame identifier (frame_id) for each video frame, and perform image preprocessing on the video frame, wherein the frame_id is associated and bound with the device_id to form a double unique association key; The large model dual-dimensional analysis and verification module is configured to call the multimodal large model to perform dual-dimensional analysis of OSD analysis and scene semantic analysis on the preprocessed video frames to obtain OSD analysis information and scene semantic analysis information; and perform latitude and longitude closed-loop verification to match the device latitude and longitude with the geographic landmark information in the scene semantic analysis information to generate a matching verification result. The multi-source data fusion and large model dynamic adaptation module is configured to automatically fuse and standardize the device information, OSD parsing information, scene semantic parsing information and matching verification results based on the dual unique association key, and generate knowledge data in a unified format. The knowledge base construction and operation module is configured to import the knowledge data into a large model knowledge base, and to perform index construction and operation and maintenance management of the knowledge base; The question-and-answer interaction module is configured to receive natural language questions from users, perform searches based on the knowledge base, and generate and output conversational answers using a large model.

[0006] By adopting the above technical solutions and addressing the shortcomings of existing technologies, the core design objective of this invention is to construct a fully automated and standardized video resource analysis system based on a large model, encompassing the entire process of acquisition, parsing, data storage, and question answering. This system achieves full-process modularization and standardization by decomposing the system into six independent modules, defining the core responsibilities and data flow rules of each module, unifying the data format, enumeration values, and accuracy requirements throughout the entire process, reducing module coupling, and facilitating development, expansion, and reuse. This enables precise data correlation across all stages. For example, a dual unique association key is designed to run through the entire process, ensuring accurate matching between device information, video frames, parsed data, and knowledge entries, thus preventing data misalignment at the source. Furthermore, it can improve the accuracy and effectiveness of video content parsing. By designing a large-scale model dual-dimensional parsing (OSD + scene) process, along with multiple rounds of verification, field standardization, latitude and longitude matching verification, and overall parsing quality verification, an accuracy statistics and control mechanism is established to ensure that the parsing results are effective and accurate. It can also achieve automated fusion and format adaptation of multi-source data, pull effective data from multiple sources based on dual unique association keys, complete automated fusion, deduplication, and standardization processing, and perform exclusive format conversion for open source / commercial large models, providing knowledge bases with non-redundant and directly importable knowledge data; Enhance the flexibility of large model adaptation, standardize the large model calling logic and Prompt design, and support open source (Qwen-VL / LLaVA-VL) and commercial (Tongyi Qianwen VL / Wenxin Yige VL) multimodal large models. Enterprises can flexibly choose according to their computing power, cost and accuracy requirements, taking into account both customization and rapid deployment. Lowering the barrier to video resource search, enabling conversational / natural language question-and-answer interaction, supporting speech-to-text, fuzzy matching, and multi-turn question-and-answer, allowing non-professionals to quickly obtain video resource information and improve information acquisition efficiency; Build a comprehensive operation and maintenance management system, establish a full-process log traceability, anomaly alarm, data backup, and accuracy statistics mechanism, realize incremental updates, accuracy optimization, and access control of the knowledge base, and ensure stable system operation and secure and traceable data.

[0007] In a further embodiment, the device identifier device_id is generated in the format of: device type-region-number, and the video frame identifier frame_id is generated in the format of: device-id-acquisition date. The dual unique association key runs through the entire process of the device resource acquisition module, video frame acquisition and preprocessing module, large model dual-dimensional parsing and verification module, multi-source data fusion and large model dynamic adaptation module, knowledge base construction and operation and maintenance module, and question-and-answer interaction module, and is used to realize the precise one-to-one binding of device information, video frames, parsed data and knowledge entries.

[0008] In a further embodiment, the large model two-dimensional parsing and verification module is specifically used for: The first standardized prompt guides the multimodal large model to extract and structure the time, device number, and alarm information from the OSD region image of the video frame, thereby generating the OSD parsing information. The second standardized prompt guides the multimodal large model to perform semantic understanding of the entire frame image, outputting a scene description including scene type, core objects, geographical landmarks, and environmental state, and generating the scene semantic parsing information; Based on the GCJ02 coordinate system, the latitude and longitude of the device are converted into a fourth-level geographic description, and semantic similarity matching is performed between the description and the geographic landmark information. Based on the matching degree threshold, the matching verification results are generated as matched, unmatched, or pending manual review.

[0009] In a further embodiment, the large model dual-dimensional parsing and verification module is further configured to perform multiple rounds of parsing and verification on the OSD parsing information and / or the scene semantic parsing information, including secondary completion parsing of missing or erroneous core fields, and field standardization correction of the parsing results. The standardization correction includes at least unifying the device number format, aligning the alarm type enumeration values, and unifying the time format.

[0010] In a further embodiment, the multi-source data fusion and large model dynamic adaptation module is specifically used for: Using the frame_id and device_id as joint query conditions, retrieve the corresponding device information, video frame path, OSD parsing information, scene semantic parsing information, and matching verification results; Following the principle of one frame, one purpose, the retrieved multi-source data is mapped, deduplicated, filled with null values, and formatted uniformly to generate standardized JSON format knowledge entries. Depending on the type of the downstream large model, the JSON format knowledge entries are converted into plain text or vectorized format to adapt to the knowledge base import requirements of open source or commercial large models.

[0011] In a further embodiment, the question-and-answer interaction module is specifically used for: The natural language query is segmented and semantically analyzed to extract query keywords; Based on the query keywords, a dual search of keyword precision search and semantic similarity search is performed in the knowledge base, and search results with a matching degree higher than a preset threshold are returned first. The third standardized prompt guides the large model to generate concise, conversational response text from the search results; And while outputting the response text, provide a preview entry for video frames or device information associated with the response.

[0012] In a further embodiment, an operation and maintenance management mechanism is also included, configured as follows: Use Elasticsearch to store the full-process logs of the device resource acquisition module, video frame acquisition and preprocessing module, large model two-dimensional parsing and verification module, multi-source data fusion and large model dynamic adaptation module, knowledge base construction and operation and maintenance module, and question and answer interaction module. The system statistically analyzes and monitors the accuracy rates of OSD parsing, scene semantic parsing, and latitude and longitude matching, and triggers an anomaly alarm when the accuracy rate is lower than a preset threshold. The knowledge base is subject to incremental updates, periodic cleanups, full backups, and access control management.

[0013] In a further embodiment, the image preprocessing performed on the video frame by the video frame acquisition and preprocessing module includes: locating and cropping the OSD region image using an edge detection algorithm, image denoising, image enhancement, uniform conversion to RGB format, and adjustment to a large model adaptation size; and attaching the device_id, frame_id, and device latitude and longitude as metadata to the preprocessed image.

[0014] In a further embodiment, the parsing result output by the large model dual-dimensional parsing and verification module includes a parsing quality control field, which includes at least a parsing status, an array of anomaly types, a data validity identifier, and a statistical label; wherein, the data validity identifier is used to indicate whether the parsing result is entered as valid data into the multi-source data fusion and large model dynamic adaptation module.

[0015] In a further embodiment, the multimodal large model is any one or a combination of the following models: open-source multimodal large models Qwen-VL and LLaVA-VL, or commercial multimodal large models Tongyi Qianwen VL and Wenxin Yige VL; the system dynamically selects and calls the corresponding large model to perform the two-dimensional analysis and the question-and-answer interaction according to a preset computing power cost strategy or accuracy requirement strategy.

[0016] In summary, the present invention has the following beneficial effects: 1. Through the collaborative settings of the device resource acquisition module, video frame acquisition and preprocessing module, large model dual-dimensional analysis and verification module, and dual unique association keys, it is possible to achieve accurate binding of device information, video frames, analysis data and knowledge entries throughout the entire process, thereby avoiding data misalignment from the source and significantly improving the reliability and consistency of data association; 2. By combining OSD parsing and scene semantic parsing with latitude and longitude closed-loop verification and multi-round verification correction in the large model dual-dimensional parsing and verification module, the video content can be deeply parsed in a structured and semantic way, and closed-loop verification can be performed through geographical location information, which greatly improves the accuracy and effectiveness of the parsing results. 3. Through the setup of multi-source data fusion and large model dynamic adaptation modules and knowledge base construction and maintenance modules, it can automatically complete the fusion, deduplication, and standardization of multi-source heterogeneous data, and perform format conversion for different large models, thereby achieving efficient construction and low-cost maintenance of the knowledge base; 4. The natural language processing, dual search, and standardized prompts in the question-and-answer interaction module enable low-threshold queries with conversational language, fuzzy matching, and multi-round question-and-answer sessions, and generate easy-to-understand answers, greatly improving the efficiency of non-professionals in obtaining video resource information. 5. Through the settings of full-process log storage, accuracy statistics, anomaly alarms, incremental updates and access control in the operation and maintenance management mechanism, the stable operation of the video file management system, data security and traceability, and continuous optimization of the knowledge base can be systematically guaranteed. Attached Figure Description

[0017] Figure 1 This is a system architecture block diagram of the present invention. Detailed Implementation

[0018] The present invention will be further described in detail below with reference to the accompanying drawings.

[0019] Example 1: like Figure 1As shown, the video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification has the following overall architecture: device resource acquisition module, video frame acquisition and preprocessing module, large model dual-dimensional analysis and verification module, multi-source data fusion and large model dynamic adaptation module, knowledge base construction and operation and maintenance module, and question-and-answer interaction module. Each module is connected in sequence and operates in a closed loop, with dual unique association keys (device_id / frame_id) running through the entire process.

[0020] Specific implementation process: This invention uses a dual-primary-key association system consisting of device_id (unique device identifier, format: device type_region_number) and frame_id (unique video frame identifier, format: device_id_acquisition date). All module data is bound one-to-one through the dual primary keys, ensuring no misalignment between device information, video frames, parsing results, and knowledge entries. The device_id is strongly bound to the device's latitude and longitude (GCJ02 coordinate system), and the frame_id is automatically associated with the video frame's storage path. The aim is to build an intelligent video resource system based on a large model. The system is divided into six major modules, sequentially connected and operating in a closed loop. Each module has clear responsibilities and standardized data interaction. The dual unique association keys (`device_id` / `frame_id`) run throughout the entire process, achieving accurate cross-module data matching. The architecture is as follows: Equipment resource acquisition module → Video frame acquisition and preprocessing module → Large model dual-dimensional analysis and verification module → Multi-source data dual primary key fusion and large model dynamic adaptation module → Knowledge base construction and maintenance module → Role-adaptive context-based question-and-answer interaction module.

[0021] The specific implementation process includes six steps, and the functions of these six steps are as follows: Step 1: Equipment Resource Information Collection and Standardization (Equipment Resource Acquisition Module): The scope and method of collection in step 1: Collect basic device information for all generated video resources to ensure the uniqueness and completeness of the information, providing core support for subsequent data association throughout the entire process. The specific collection method is as follows: Automatic data collection: The system automatically retrieves core device information via device SDK, SNMP protocol, and API interface (such as device management platform interface), and synchronizes it once every day at midnight to ensure real-time information. Manual data entry: For devices without interfaces or that cannot automatically collect data, a manual data entry page is provided, which supports batch import and the system automatically verifies the format after data entry. Anomaly Validation: After data collection / entry, the system automatically verifies the integrity (required fields must be non-empty) and uniqueness (device IDs must be unique), marks abnormal data, and reminds maintenance personnel to correct it.

[0022] Step 1 also requires standardization and uniformity of format (core fields): JSON format is used as the standardized output format. All fields have consistent naming, data types, and value ranges to avoid ambiguity. `device_id` is a globally unique core key. The fields are as follows (required fields are marked with *, optional fields can be expanded according to the scenario): json {"device_info":{"device_id":"*string", / / Unique device identifier, format: device type_region_number (e.g., CAMERA_SHANGHAI_001), length 1-32 characters "device_name":"*string", / / Device name, Chinese characters + letters + numbers, length 1-64 characters (e.g., "Shanghai Xuhui District Surveillance Camera 001") "device_type":"*string", / / Device type, enumeration values: CAMERA (camera), NVR (Non-Virtual Video Recorder), DVR (Digital Video Recorder), OTHER (other). "region":"*string", / / Device deployment region, format: province-city-district-specific location (e.g., "Shanghai-Xuhui District-Xujiahui Street-1st Floor of XX Building") "longitude":"*string", / / Device longitude, format: decimal degrees (6 decimal places), e.g., "121.473701", required (for latitude and longitude matching verification). "latitude":"*string", / / Device latitude, format: decimal degrees (6 decimal places), e.g., "31.230416", required (for latitude and longitude matching verification). "video_source":"*string", / / Video resource path / URL, format: protocol: / / IP:port / path (e.g., "rtsp: / / 192.168.1.100:554 / stream1") "video_resolution":"string", / / Video resolution, enumeration values: 1080P, 720P, 4K, D1, CIF, optional "manufacturer":"string", / / Equipment manufacturer, such as "Hikvision" or "Dahua", optional "install_time":"string", / / Device installation time, format: YYYY-MM-DDHH:MM:SS (e.g., "2024-01-01 10:00:00"), optional "maintainer":"string", / / Maintenance personnel, format: name + contact information (e.g., "Zhang San 13800138000"), optional "status":"*string", / / Device status, enumeration values: ONLINE (online), OFFLINE (offline), FAULT (faulty) "collect_time":"*string" / / Information collection time, format: YYYY-MM-DDHH:MM:SS (e.g., "2024-10-01 00:30:00")}} After standardizing the format, further format validation and standardization processing are required. Field validation: Use a JSONSchema validation tool to validate the data type, value range, and format (such as time format, device ID format) of each field. Fields that do not meet the requirements are marked as abnormal and a correction prompt is returned. Deduplication: Using `device_id` as the unique key, if duplicate device information is collected, the latest `collect_time` record is retained, and the old data is overwritten; Unified Conversion: Non-standard fields will be converted uniformly (e.g., resolution "1920*1080" will be converted to "1080P", and the region name "Shanghai Xuhui" will be completed to "Shanghai-Xuhui District-None-None"). Storage Backup: Standardized device information is stored in a relational database (such as MySQL) and backed up to distributed storage (such as MinIO) for easy retrieval and traceability later; a new latitude and longitude field is added and bound to `device_id` to ensure that the latitude and longitude of each device is unique and traceable, providing a foundation for subsequent large-scale model matching and verification.

[0023] Step 2, Video Frame Acquisition and Preprocessing (Video Frame Acquisition and Preprocessing Module): The video frame acquisition rules for step 2 are as follows: Data collection time: Data is collected at a fixed time every day (08:00 AM), when the video image is relatively stable and the OSD information is complete, avoiding blurry or obstructed images; Acquisition method: Using a video stream parsing tool (such as FFmpeg), call the standardized `video_source` path in step 1 to extract single frames in JPG format with the same resolution as the original video. The naming rule is: `device_id_acquisition date.jpg` (e.g., CAMERA_SHANGHAI_001_20241001.jpg). The `frame_id` is automatically generated (which is consistent with the filename and is a unique identifier for the video frame). Anomaly Handling: If the device is offline (status is OFFLINE) or the video stream cannot be retrieved, mark the acquisition as failed, record the reason for the failure (such as "device offline" or "video stream interrupted"), retry the next day, and remind the maintenance personnel after 3 consecutive failures; Storage Management: Acquired video frames are stored in distributed storage with the path format: ` / video_frame / device_type / region / acquisition date / `. They are doubly associated with `device_id` and `frame_id` for easy binding with device information and parsed data later.

[0024] The image preprocessing method in step 2 is (to improve resolution): Standardization preprocessing is performed on the acquired video frames to ensure the accuracy of large model parsing. The steps are as follows: OSD region localization: Using edge detection algorithms (such as the Canny algorithm), the OSD region in the image (usually located in the four corners of the image) is located, and the OSD region is cropped out and used separately as the parsing input to reduce background interference; Standardize the format: Convert both the preprocessed full-frame image and the cropped OSD area image to RGB format, and adjust the size to fit the large model (e.g., 640*480) to ensure that the large model can be parsed normally; Metadata mounting: Mount basic associated metadata (`device_id`, `frame_id`, acquisition date, device latitude and longitude, storage path) to the preprocessed image. After the parsing results are generated, they can be directly bound to avoid data misalignment.

[0025] Step 2 requires the module to output the following: Standardized JPG video frames (full frame + OSD cropped area) for distributed storage. The video frame's unique identifier `frame_id` forms a unique association key with `device_id`; Preprocessed RGB format image (adapted for large model parsing); Video frame storage path, acquisition anomaly logs, and device-related metadata.

[0026] Step 3: Large Model Two-Dimensional Analysis and Verification (Large Model Two-Dimensional Analysis and Verification Module): The two-dimensional analysis process in step 3 is (OSD + scene): Local deployment supports large models for multimodal parsing (image + text). The overall parsing process follows the workflow of "single-frame image dual-dimensional parsing → multi-round verification and completion → field standardization correction → matching with associated device information → overall quality verification". It performs OSD + scene dual-dimensional collaborative parsing: ① OSD parsing focuses on the structured extraction of time, device number, and hierarchical alarm information (first-level type + second-level subtype), outputting standardized data with an accuracy of ≥0.8. If the accuracy is lower than 0.6, a second-level completion parsing is triggered; ② Scene parsing must include four elements: 'scene type + core object + geographic landmark + environmental status'. Geographic landmark extraction must be adapted to the geographic region corresponding to the device's latitude and longitude to avoid invalid information across regions; the dual-dimensional parsing results are forcibly associated through frame_id to ensure data consistency.

[0027] Specifically, this also includes structured parsing of OSD content (focusing on precise extraction of fixed fields): Using cropped OSD region images as the main input and whole-frame images as a supplement, the large model is guided by a standardized prompt to extract superimposed text from the image and convert it into structured data. The core process is "text extraction → type classification → format standardization → multi-round verification".

[0028] First, standardize the Prompt design (immanual, unambiguous, adaptable to all multimodal models): Please parse all visible text in the current OSD region image, strictly following the requirements to extract and output the results in JSON format, without any extraneous descriptions: Text types are categorized into four types: time, device number, alarm information, and others, which are extracted separately. All times should be converted to YYYY-MM-DDHH:MM:SS format. If only the hour and minute are present, fill in the complete time of the [Collection Date]. If no time is available, enter "None". The equipment number must be aligned with the format [Equipment Type_Region_Number]. If they do not match, the format should be corrected. If there is no number, fill in "None". Alarm information should include type (select only NO_ALARM / MOTION / FAULT / OTHER), level (select only INFO / WARNING / CRITICAL / None), and detailed description. If there are no alarms, fill in "None" for all fields. Extract other OSD text (manufacturer, parameters, etc.) directly; fill in "none" if none are found. Output parsing accuracy (0-1 floating-point number, rounded to 2 decimal places). Unrecognized content is marked as "unknown".

[0029] And multiple rounds of parsing and verification are required: First parsing: The large model outputs an initial version of the JSON result, and the module performs an initial check on the core fields (time, device number, alarm type); Secondary completion analysis: If the core fields are missing or incorrect, a supplementary prompt is generated (such as "Please re-check the four corners of the entire frame image and extract the time information"). The large model is then called again in conjunction with the entire frame image to focus on the precise extraction of missing information. As a fallback: core fields that cannot be extracted after secondary parsing are marked as "none" according to the rules to ensure that there are no empty values.

[0030] Furthermore, the fields need to be standardized and corrected. Device ID Correction: Match the parsed device ID with the standardized `device_id` from step 1, correct for deviations such as capitalization and abbreviation, and generate `device_id_osd` (which is completely consistent with `device_id`). Forced alignment of enumeration values: If a custom value appears in the alarm type or level, it will be automatically corrected to "OTHER" / "none"; Standardized format: Time, numerical, and other fields are validated according to system specifications, and parsing accuracy is maintained to two decimal places.

[0031] In step 3, the semantic parsing of the scene is defined as (scene type + core object + geographic landmark + environmental state): Using the preprocessed whole-frame image as input, the large model is guided by a scene-specific standardized prompt to complete the semantic description of the image and extract key features, providing a location basis for latitude and longitude matching and verification. The core is "scene classification → content description → feature extraction → multi-round optimization".

[0032] First, perform a standardized Prompt design (including enumeration value guidance and limiting description rules): Please parse the scene of the current full-frame video, and output the results in JSON format strictly according to the following requirements, without any extraneous descriptions. The scene description should be limited to 100 characters: Scene type can only be selected from the following enumerated values: INDOOR / OUTDOOR / OFFICE / WAREHOUSE / ROAD / SHOP / OTHER; Scene descriptions should include [scene type + core object + geographical landmark + environmental conditions], and should be concise. Extract core objects separately, separated by commas; fill in "none" if none exist. Environment status is selected from enumerated values ​​only: BRIGHT / DARK / SUNNY / RAINY / CLOUDY / BLOCKED / OTHER; Extract location-related information (landmarks, roads, building names, etc.) from the image; if none is found, fill in "none". Output parsing accuracy (0-1 floating-point number, rounded to 2 decimal places). If the scene cannot be recognized, mark the scene type as "OTHER" and the description as "Unknown scene".

[0033] Furthermore, multiple rounds of analysis and optimization are required: First analysis: The large model outputs an initial version of the results, and the module judges the quality of the description (whether it is blurry, whether the location information is extracted, and whether the core objects are specific). Secondary refinement analysis: If the description is not up to standard, a refinement prompt is generated (such as "Please describe the type, location and name of the core object in the image and the name of the visible landmark") to guide the large model to make a more accurate description. The methods for field standardization and feature extraction in scene semantic parsing are as follows: Forced alignment of enumeration values: Custom values ​​for scene type and environment state are automatically corrected to "OTHER"; Concise description: Scene descriptions exceeding 100 words are automatically extracted to extract the core content while ensuring fluent sentences; Location information refinement: Extract key location features (such as "XX Building in Xuhui District") from the scene description and store them separately in the `location_related_info` field to prepare for latitude and longitude matching; Standardized format: Parsing accuracy is rounded to two decimal places, and core objects are formatted as "object1, object2".

[0034] The latitude and longitude matching verification method (core verification step) in step 3 is as follows: The latitude and longitude closed-loop verification uses the GCJ02 coordinate system and performs a three-step verification: Call the geographic information API to convert the standardized latitude and longitude of the device in step 1 into a four-level geographic description of 'province-city-district-road'; Extract geographic landmark information from scene analysis and perform semantic similarity matching with the fourth-level geographic description. A similarity of ≥0.9 is judged as MATCH, ≤0.7 is judged as MISMATCH, and 0.7-0.9 triggers manual review. Mark `manual_review_result` (PASSED / UNREVIEWED / CORRECTED). UNKNOWN results do not require review and are marked as UNREVIEWED.

[0035] Step 3 requires standardized output format matching. The standardized output format (JSON) is as follows: The module's final output is the parsing result, which must include three required fields: two-dimensional accuracy, latitude and longitude matching result, and manual review mark. It distinguishes between required (*) and optional fields. All enumeration values, formats, and precision are strictly standardized and can be directly retrieved by the knowledge data integration module without secondary processing.

[0036] json {"frame_parse_check_result":{"basic_info":{"frame_id":"*string", / / Unique identifier for video frame, device_id_acquisition date} "device_id":"*string", / / Unique device identifier, exactly the same as in step 1 "collect_date":"*string", / / Collection date, YYYY-MM-DD "video_frame_path":"*string", / / Full path to distributed storage of video frames "parse_time":"*string", / / Parsing completion time, YYYY-MM-DDHH:MM:SS "parse_device":"string" / / The server / device number that performs the parsing, optional}, "osd_parse_info":{"osd_time":"string", / / OSD display time, YYYY-MM-DDHH:MM:SS, fill in "None" if none exists.} "device_id_osd":"*string", / / Standardized device ID, identical to device_id; fill in "None" if none exists. "alarm_info":{"alarm_type":"*string", / / enumeration value: NO_ALARM / MOTION / FAULT / OTHER "alarm_level":"string", / / Enumeration value: INFO / WARNING / CRITICAL / None "alarm_desc":"string" / / Alarm description, fill in "none" if none is provided. "other_osd":"string", / / Other OSD text; fill in "none" if none exists. "osd_parse_accuracy":"*float", / / OSD parsing accuracy, 0-1, rounded to 2 decimal places. "osd_parse_round":"*int" / / Parsing round, 1 = first parsing, 2 = second parsing round. "scene_parse_info":{"scene_type":"*string", / / enumeration value: INDOOR / OUTDOOR / OFFICE / WAREHOUSE / ROAD / SHOP / OTHER "scene_desc":"*string", / / Scene description, ≤100 characters, or "Unknown Scene" if none. "core_object":"string", / / Core object, comma-separated, fill in "none" if none exists. "environment":"*string", / / enumeration value: BRIGHT / DARK / SUNNY / RAINY / CLOUDY / BLOCKED / OTHER "location_related_info":"string", / / Location-related information; fill in "None" if none is available. "scene_parse_accuracy":"*float", / / Scene parsing accuracy, 0-1, rounded to 2 decimal places "scene_parse_round":"*int" / / parse round, 1=first round, 2=second round of refinement}, "lat_lng_match_check":{"device_longitude":"*string", / / device longitude, rounded to 6 decimal places.} "device_latitude":"*string", / / Device latitude, rounded to 6 decimal places "lat_lng_match":"*string", / / enumeration value: MATCH / MISMATCH / UNKNOWN "lat_lng_match_desc":"*string", / / Detailed matching description; fill in "None" if none is provided. "manual_review_result":"*string", / / enumeration value: PASSED / UNREVIEWED / CORRECTED "manual_review_time":"string", / / Review time, YYYY-MM-DDHH:MM:SS, fill in "None" if none is specified. "manual_reviewer":"string" / / Reviewer; fill in "None" if none is available. "parse_quality_control":{"parse_status":"*string", / / Enumeration values: NORMAL (normal) / ABNORMAL (abnormal) / REVIEW (pending review)} "abnormal_type":"array", / / Array of abnormal types, otherwise [], enumeration: FRAME_ID_DUPLICATE / FIELD_MISS / ACCURACY_LOW / LAT_LNG_MISMATCH / OTHER "abnormal_desc":"string", / / Abnormal description; fill in "None" if none is provided. "data_valid":"*boolean", / / true = valid data, false = invalid data "statistic_tag":"*boolean" / / true=Include in latitude and longitude matching accuracy statistics, false=Not included}}}.

[0037] After all the above steps in step 3 are completed, the final step is to store and log the data. Core storage: Standardized parsing results are stored in a MySQL database and linked to the device information table via a foreign key using `device_id`, facilitating table join queries by the integration module; Log storage: Parse the entire process log (Prompt content, call rounds, verification results, and exception information) and store it in Elasticsearch, archived by `frame_id`, for easy operation and maintenance traceability; Abnormal storage: Parse abnormal / data pending review and mark it separately, push it to the list of manual reviewers, and update the results after review.

[0038] Step 4: Knowledge Data Integration and Format Conversion (Multi-Source Data Dual Primary Key Fusion and Large Model Dynamic Adaptation Module): Using dual unique association keys (`device_id` / `frame_id`) as the core anchor points, it automatically pulls standardized and effective data from multiple sources. Following the logic of "source data alignment → dual key precise matching → field standardization → structured fusion → format conversion", it completes data fusion and adaptability format conversion, providing knowledge data that is free of redundancy, unbiased, and can be directly imported for knowledge base construction.

[0039] The data retrieval rules and implementation methods in step 4 are as follows: The data retrieval process is characterized by "timed triggering + precise filtering + automated interface calls," which retrieves only valid data and prevents invalid data from entering the fusion process. No manual intervention is required throughout the entire process.

[0040] Triggering rules: Triggered at 02:00 every day (to avoid peak times with the parsing module and ensure full data storage), pulling all valid data from the previous day (calendar day); Filtering rules: Only retrieve parsed data with `data_valid=true`, device information with `status=ONLINE`, and accessible video frame storage path information; Pull interface: Through the preset MySQL structured query interface, the table is joined to pull and parse data and device information, and the video frame path is obtained through the distributed storage interface. If the pull fails, it will trigger 3 retries (with an interval of 5 minutes). If it still fails, an alarm will be pushed. Temporary storage: Fetched data from multiple sources is stored in a Redis temporary cache to avoid duplicate queries and to verify data integrity (by counting the number of `frame_id` / `device_id`).

[0041] The logic for multi-source data fusion in step 4 is as follows: The fusion follows a six-step logic: "source data pre-alignment → double-key precise matching → field extraction and mapping → full field standardization → structured integration → secondary validity verification". The core is "one frame, one entry", that is, one `frame_id` corresponds to one fused knowledge entry, ensuring data granularity is consistent.

[0042] Source data pre-alignment is used to eliminate fundamental differences between multi-source data, preparing for accurate matching: Time-based alignment: Only retain device information, parsed data, and video frame information from the same acquisition date; Validity alignment: Only retain valid records of various data types (parsed data `data_valid=true`, complete device information fields, and accessible video frame paths); Format dimension alignment: Unified null value identifier (fill in "None" if there is no data), time format (YYYY-MM-DDHH:MM:SS), numerical precision (parsing accuracy to 2 decimal places, latitude and longitude to 6 decimal places).

[0043] Double-key precise matching: Enables one-to-one binding of multi-source data, preventing mismatches and omissions at the source. Level 1 Match (device_id): Using `device_id` in the parsed data as the query condition, match the complete device information with the same `device_id` from the device information pool. If no match is found, fill in "none" in the device field and mark `device_match_flag=false` (only for logging, does not affect fusion). Second-level matching (frame_id): Using the `frame_id` in the parsed data as the core, extract the complete storage path of the pre-stored video frame. If the path is inaccessible, mark `frame_path_valid=false`. Matching results: Each valid `frame_id` forms a precise binding combination of "1 piece of parsed data + 1 piece of device information + 1 video frame path".

[0044] Full field standardization: Following the principles of "retaining core, eliminating redundancy, and categorizing and grouping," core fields are extracted from the bound data and mapped to fixed levels of knowledge entries to avoid field confusion. Fields with the same name are standardized based on the parsing module's standardized value. The grouped fields undergo final standardization to ensure no formatting errors or semantic ambiguity, making them ready for direct format conversion. Unified null value rules: For optional fields with no data, fill in "None" (string) or [] (array); do not use null or empty strings. Enumeration value validation: All enumeration fields are forced to align with system specifications, and custom values ​​are automatically corrected to "OTHER"; The format should be consistent: time fields should be YYYY-MM-DDHH:MM:SS, floating-point numbers should be retained to 2 decimal places, and latitude and longitude should be retained to 6 decimal places; Text cleaning: Remove special characters and extra spaces from text fields; keep scene descriptions within 100 characters. Uniqueness Guarantee: entry_id / frame_id / device_id are unique throughout the entire process, with no duplicates.

[0045] Structured integration and secondary validation: Structured integration: Standardized fields are organized into unified JSON format knowledge entries at a fixed hierarchy, with entry_id and frame_id being completely identical, achieving a strong association with parsed data and video frames; Secondary validity validation: Only the core related fields (device_id / frame_id / entry_id) are verified to be unique and without missing values, and `data_valid` is true, to avoid the loss of core fields during the fusion process; Result classification: Entries that pass the validation are marked as "Fusion Valid" and stored in the data source pool to be converted; Entries that fail the validation are marked as "Fusion Abnormal," stored separately, and pushed for manual review.

[0046] The data format conversion (adapting to large models) in step 4 is as follows: For the standardized JSON knowledge entries after integration, a specific format conversion is performed according to the major model type to ensure that it is compatible with the knowledge base import requirements. The two solutions can be executed independently and can be selected as needed.

[0047] The first method is text format conversion (suitable for large open-source models such as Qwen / LLaMA). This converts JSON entries into a plain text format of "key-value pairs + natural language descriptions," adapting to the open-source model's knowledge base reading logic. The steps are: Valid entry filtering: Remove entries with fusion anomalies, keeping only those with `data_valid=true`; Field normalization: Extract core fields, remove redundant maintenance fields, fill in "none" for empty values, and convert enumerated values ​​to Chinese descriptions (e.g., NO_ALARM → no alarms); Text concatenation: Concatenate text into natural language text according to fixed rules, formatted as "field name:field value;". Example: Item ID: CAMERA_SHANGHAI_001_20241001; Device ID: CAMERA_SHANGHAI_001; Device Name: Shanghai Xuhui District Surveillance Camera 001; Device Type: Camera; Deployment Area: Shanghai - Xuhui District - Xujiahui Street - XX Building 1st Floor; Longitude: 121.473701; Latitude: 31.230416; Device Status: Online; Acquisition Date: 2024-10-01; Video Frame Path: / video_frame / CAMERA / Shanghai / 20241001 / CAMERA_SHANGHAI_001_20241001.jpg; OSD Time: 2024-10-01 00:10:00; Alarm Type: No Alarm; Scene Type: Outdoor Road; Scene Description: Outdoor road scene, well-lit, with a white car in the center of the image, and the XX Building in Xuhui District visible in the background; Latitude and Longitude Matching Result: Matched; Parsing Accuracy: OSD 0.92, Scene 0.88; Batch Saving and Verification: Saved as a UTF-8 format TXT file "one line at a time", naming rule: `knowledge_text_date.txt`; Randomly select 10%-20% of entries for format verification to ensure no missing fields / spelling errors; Adaptation Adjustment: If the open-source model has special requirements for text (such as character limit), adjust the spelling rules and regenerate.

[0048] Step 4 also requires module output and logging: Core outputs: TXT format text files adapted for open-source models, vector data + JSONL files adapted for commercial models, and standardized JSON knowledge entries after fusion; Log recording: Pulled logs, merged logs, and format-converted logs are stored in Elasticsearch, recording data volume, time consumption, and exception information for easy traceability; Error handling: Entries that fail to convert formats are stored separately, pushed to manual review, and converted again after optimization.

[0049] Step 5, Knowledge Base Construction and Maintenance (Knowledge Base Construction and Maintenance Module): Complete the "batch / incremental import, initialization, and index optimization" of knowledge data, and implement daily updates, cleaning, backup, and accuracy optimization. Build and maintain a "searchable, highly available, and real-time updated" large-scale model-specific knowledge base to provide stable data support for the question-answering module.

[0050] Step 5, regarding knowledge base import and initialization: Knowledge base import: Initial setup: Batch import all merged and transformed knowledge data into the knowledge base; Daily incremental updates: At 03:00 every day, the incremental knowledge data added the previous day is imported to overwrite the old related entries (such as changes in device status and corrections to parsing results). Open source large model: Through the model API interface, upload TXT text files in batches, initialize the knowledge base, and set the search strategy (keyword + semantic search). Initialization verification: After the import is complete, randomly select 10%-20% of the knowledge entries to check the import integrity (no missing fields, correct vector matching, and retrieval can be successful) to ensure that the knowledge base is usable.

[0051] The retrieval optimization configuration method in step 5 is as follows: Configure a combined search strategy of "keyword search + semantic search", prioritizing the matching of precise keywords such as device_id / frame_id / collect_date, and then performing semantic completion; Cache optimization: Cache the search results of frequently asked questions to Redis for 24 hours to improve the response speed of question and answer (from hundreds of milliseconds to tens of milliseconds).

[0052] The daily operation and maintenance steps for the knowledge base in step five are as follows: Daily incremental updates: Import new data from the previous day at fixed intervals to overwrite old data (such as device status and parsing result corrections) to ensure the real-time nature of knowledge base data; Regular cleanup: Clean up expired knowledge entries (such as video frame parsing information older than 6 months; the retention period can be configured according to business needs) at the end of each month to free up storage space; Accuracy optimization: Based on question and answer feedback, correct erroneous entries in the knowledge base (such as parsing errors and association errors), and simultaneously optimize the Prompt and fusion rules of large model parsing to improve the quality of subsequent data; Backup strategy: Perform a full backup of the knowledge base weekly and store it in distributed storage (such as MinIO) to support rapid recovery from failures; Access control: Strictly set access permissions for the knowledge base, allowing only the system's Q&A module to access it, prohibiting external access, and ensuring data security.

[0053] Step 5 also requires module output: A large-scale, dedicated knowledge base for specific models has been initialized and is now searchable. Optimized search index configuration and caching strategy; Knowledge base update / cleanup / backup logs, and accuracy optimization records.

[0054] Step 6: Large-scale conversational video resource information Q&A (role-adaptive context-based Q&A interaction module): Users ask questions using conversational natural language / voice, and the large model automatically searches the knowledge base to output easy-to-understand conversational answers. It also supports the display of associated video frames and device information, realizing a closed loop of "natural language question - accurate retrieval - conversational response - viewing associated resources", lowering the threshold for video resource search and making it suitable for non-professionals.

[0055] The following is an example of the complete question-and-answer interaction process in step 6: The entire process is automated, with user-friendly prompts for abnormal scenarios. The core workflow is as follows: Question input: Users can ask questions by entering conversational text through the front-end interface (Web / Mini Program) or by voice input (integrating Baidu / Aliyun voice API to achieve speech-to-text conversion), with no restrictions on input format; Question preprocessing: Using NLP tools (jieba word segmentation) + large model semantic parsing, keywords in the question are extracted (such as device name / number, collection date, query requirements), associated with knowledge base fields (such as "camera 001" → device_id / device_name), and ambiguity is handled (such as "cameras in Shanghai" → supplementary area filtering). Knowledge base retrieval: Based on the extracted keywords, the knowledge base retrieval interface is called to perform a **dual retrieval** (first, precise keyword filtering, then semantic similarity matching), prioritizing the return of results with a matching degree ≥ 0.8; Large-scale model generates answers: The large-scale model obtains search results and generates conversational answers guided by a standardized prompt. Requirements: accurate matching of needs, concise and easy to understand (within 3 sentences), no technical jargon, and multiple entries are arranged from newest to oldest according to the collection date. Response output: Display conversational responses to users, while providing access to **related resources** (such as clicking "Daily Footage" to preview video frames, or clicking "Device Information" to view full details); Error handling: No matching result: Output a prompt (e.g., "Sorry, no relevant video resources were found. Please check if the device number or collection date is correct."); Ambiguous question: Output a guiding prompt (e.g., "Please tell me the specific device name / number and collection date, and I will help you search.").

[0056] Step 6: After introducing the large model, the large model question-answering configuration (key optimization) is as follows: Standardized Prompt design (core) guides large models to generate accurate, conversational responses, avoiding redundancy and jargon. Example: Based on the retrieved knowledge entries, please answer the user's conversational questions, strictly adhering to the following requirements: Answers must be conversational and easy to understand, avoiding technical terms such as JSON and field names, and should be spoken like a human conversation. It accurately matches user needs, answers questions directly, and avoids adding irrelevant information; Multiple matching entries are sorted from newest to oldest by collection date, and each entry is answered individually; Appropriately mention key verification results (such as "the screen information matches the device deployment location"); Keep your answers concise, no more than three sentences. If no matching entry is found, the message will be: "Sorry, no relevant video resources were found. Please check if the device number and collection date are correct."

[0057] Search and answer optimization: Fuzzy matching: Supports fuzzy matching of user queries (e.g., “Xuhui camera 001” → “Shanghai Xuhui District surveillance camera 001”, “October 1st” → “2024-10-01”), lowering the threshold for asking questions; Multi-turn question answering support: A multi-turn question answering interface is reserved. When the user asks follow-up questions (such as "What color is the car in the picture?"), the large model makes an accurate answer based on the already retrieved items without the need for re-retrieval. Linked Resources: Lightweight prompts for related resources are embedded in the answer, which can be clicked to jump to the preview, improving the interactive experience.

[0058] Question and answer example: User question: What scene was captured by camera 001 in Xuhui District, Shanghai on October 1st? Were there any alarms? The large model responded: The footage from camera 0011 in Xuhui District, Shanghai on October 1st is an outdoor road scene with ample lighting. The footage shows a white car and the surveillance camera, but there were no alarms, and the image information matches the device's deployment location. User question: Is device CAMERA_SHANGHAI_001 online today? The large model replied: Device CAMERA_SHANGHAI_001 is online today~.

[0059] In the description of this invention, unless otherwise explicitly defined, terms such as "set up," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0060] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of the claims of the present invention.

Claims

1. A video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification, characterized in that, include: The device resource acquisition module is configured to acquire device information of video acquisition devices and generate a globally unique device identifier for each device. The device information includes at least the device's latitude and longitude. The video frame acquisition and preprocessing module is configured to acquire video frames from the video acquisition device, generate a globally unique video frame identifier for each video frame, and perform image preprocessing on the video frames, wherein the video frame identifier is associated and bound with the device identifier to form a double unique association key; The large model dual-dimensional analysis and verification module is configured to call the multimodal large model to perform dual-dimensional analysis of OSD analysis and scene semantic analysis on the preprocessed video frames to obtain OSD analysis information and scene semantic analysis information; and perform latitude and longitude closed-loop verification to match the device latitude and longitude with the geographic landmark information in the scene semantic analysis information to generate a matching verification result. The multi-source data fusion and large model dynamic adaptation module is configured to automatically fuse and standardize the device information, OSD parsing information, scene semantic parsing information and matching verification results based on the dual unique association key, and generate knowledge data in a unified format. The knowledge base construction and operation module is configured to import the knowledge data into a large model knowledge base, and to perform index construction and operation and maintenance management of the knowledge base; The question-and-answer interaction module is configured to receive natural language questions from users, perform searches based on the knowledge base, and generate and output conversational answers using a large model.

2. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that, The generation format of the device identifier is: device type-region-number; The generation format of the video frame identifier is: device-id-acquisition date; the dual unique association key runs through the entire process of the device resource acquisition module, video frame acquisition and preprocessing module, large model dual-dimensional parsing and verification module, multi-source data fusion and large model dynamic adaptation module, knowledge base construction and operation and maintenance module, and question-and-answer interaction module, and is used to realize the precise one-to-one binding of device information, video frames, parsed data and knowledge entries.

3. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that, The large model two-dimensional analysis and verification module is specifically used for: The first standardized prompt guides the multimodal large model to extract and structure the time, device number, and alarm information from the OSD region image of the video frame, thereby generating the OSD parsing information. The second standardized prompt guides the multimodal large model to perform semantic understanding of the entire frame image, outputting a scene description including scene type, core objects, geographical landmarks, and environmental state, and generating the scene semantic parsing information; Based on the GCJ02 coordinate system, the latitude and longitude of the device are converted into a fourth-level geographic description, and semantic similarity matching is performed between the description and the geographic landmark information. Based on the matching degree threshold, the matching verification results are generated as matched, unmatched, or pending manual review.

4. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 3, characterized in that: The large model dual-dimensional parsing and verification module is also configured to perform multiple rounds of parsing and verification on the OSD parsing information and / or the scene semantic parsing information, including secondary completion parsing of missing or erroneous core fields, and field standardization correction of the parsing results. The standardization correction includes at least unifying the device number format, aligning the alarm type enumeration values, and unifying the time format.

5. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that, The multi-source data fusion and large model dynamic adaptation module is specifically used for: Using the video frame identifier and device identifier as joint query conditions, retrieve the corresponding device information, video frame path, OSD parsing information, scene semantic parsing information, and matching verification results; Following the principle of one frame, one purpose, the retrieved multi-source data is mapped, deduplicated, filled with null values, and formatted uniformly to generate standardized JSON format knowledge entries. Depending on the type of the downstream large model, the JSON format knowledge entries are converted into plain text or vectorized format to adapt to the knowledge base import requirements of open source or commercial large models.

6. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that, The question-and-answer interaction module is specifically used for: The natural language query is segmented and semantically analyzed to extract query keywords; Based on the query keywords, a dual search is performed in the knowledge base, combining precise keyword retrieval and semantic similarity retrieval, prioritizing the return of search results with a matching degree higher than a preset threshold; The third standardized prompt guides the large model to generate concise, conversational response text from the search results; And while outputting the response text, provide a preview entry for video frames or device information associated with the response.

7. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that, It also includes an operation and maintenance management mechanism, configured as follows: Use Elasticsearch to store the full-process logs of the device resource acquisition module, video frame acquisition and preprocessing module, large model two-dimensional parsing and verification module, multi-source data fusion and large model dynamic adaptation module, knowledge base construction and operation and maintenance module, and question and answer interaction module. The system statistically analyzes and monitors the accuracy rates of OSD parsing, scene semantic parsing, and latitude and longitude matching, and triggers an anomaly alarm when the accuracy rate is lower than a preset threshold. The knowledge base is subject to incremental updates, periodic cleanups, full backups, and access control management.

8. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that: The video frame acquisition and preprocessing module performs image preprocessing on the video frame, including: locating and cropping the OSD region image using an edge detection algorithm, image denoising, image enhancement, uniform conversion to RGB format, and adjustment to a large model adaptation size; and attaching the device identifier, video frame identifier, and device latitude and longitude as metadata to the preprocessed image.

9. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that: The parsing results output by the large model dual-dimensional parsing and verification module include parsing quality control fields, which at least include parsing status, an exception type array, data validity identifier, and statistical labels; wherein, the data validity identifier is used to indicate whether the parsing result is entered into the multi-source data fusion and large model dynamic adaptation module as valid data.

10. The video file management system based on dual-dimensional analysis and latitude-longitude closed-loop verification according to claim 1, characterized in that: The multimodal large model is any one or a combination of the following models: open-source multimodal large models Qwen-VL and LLaVA-VL, or commercial multimodal large models Tongyi Qianwen VL and Wenxin Yige VL; the system dynamically selects and calls the corresponding large model to perform the two-dimensional analysis and the question-and-answer interaction according to the preset computing power cost strategy or accuracy requirement strategy.

Images