A cross-modal natural language retrieval method and system for noisy surveillance video

Abstract

The present application relates to the technical field of video processing, in particular to a cross-modal natural language retrieval method and system for noisy monitoring video, which drives natural language description through an open vocabulary detection model, expands the searchable target from traditional fixed categories to any natural language describable target, automatically captures the target appearance attributes and behavior semantics after multi-modal model coding; At the same time, a multi-factor quality evaluation mechanism based on target tracking is introduced, the best individual frame is selected for vectorization in the whole life cycle of the target, and the redundant feature expansion is inhibited from the source; And build a dual retrieval system of metadata inverted index and semantic vector index, combine large language model to structure the analysis of colloquial query, realize the double-channel parallel execution of metadata constraint filtering and semantic vector retrieval and the intersection operation of candidate set, which significantly improves the retrieval accuracy and efficiency.

Description

Technical Field

[0001] This invention relates to the field of video processing technology, specifically to a cross-modal natural language retrieval method and system for noisy surveillance videos. Background Technology

[0002] With the continuous advancement of projects such as Safe City and Skynet, public security departments have accumulated petabytes of surveillance video data. Therefore, how to quickly and accurately retrieve targets from massive amounts of unstructured video has become a core technical challenge in the field of public safety. Traditional video image retrieval methods heavily rely on manually pre-labeled tags (such as license plate numbers, target IDs, and object categories), which have three major drawbacks: First, the labeling cost is high; manual labeling of massive amounts of data consumes enormous manpower, and the labeling granularity cannot cover abstract behavioral descriptions such as "loitering" or "transaction." Second, there is a deep semantic gap; the closed, fixed label system cannot respond to officers' colloquial, undefined query needs (such as "finding a person wearing a red shirt and carrying a black backpack riding an electric bike"). Third, the retrieval efficiency is low; frame-by-frame traversal cannot meet the second-level response time in a database of hundreds of millions of images, and continuous video frames generate massive amounts of approximately redundant features, leading to a rapid expansion of the vector database. Furthermore, existing detection systems typically only support three fixed target categories: "people / motor vehicles / non-motor vehicles," leaving a large amount of physical evidence, animals, scene objects, and urban management clues outside the retrieval scope, severely restricting the practical application value of video image data.

[0003] In recent years, multimodal pre-trained models such as CLIP and BLIP have achieved a unified mapping of the visual-linguistic semantic space through image-text comparison learning, providing a technical foundation for "text-based image search." The emergence of large language models has also provided a new path for structural decoupling of colloquial natural language queries. However, existing technical solutions still have many shortcomings: for example, general image retrieval solutions such as CN114461327A do not address temporal redundancy removal and spatiotemporal constraint retrieval for noisy surveillance videos; existing solutions in the security field mostly use fixed category tags or attribute table keyword retrieval, failing to organically integrate open vocabulary-expanded target detection, multi-factor optimal frame redundancy removal, metadata inverted index, and parallel joint constraints of semantic vector retrieval into a complete retrieval solution for public security practice. Therefore, there is an urgent need for a solution that can simultaneously solve a series of complex technical problems unique to public security practice, such as maintaining retrieval accuracy under noisy and degraded images, expanding the types of searchable targets, controlling vector library expansion, and dual-channel parsing and parallel retrieval of colloquial queries. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a cross-modal natural language retrieval method and system for noisy surveillance videos, thereby solving the technical problems of low retrieval accuracy of noisy and degraded images, vector library expansion caused by massive video frames, and disconnect between metadata constraints and semantic retrieval.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A cross-modal natural language retrieval method for noisy surveillance videos includes the following steps:

[0007] S1: Acquire video stream and generate keyframes, associate each keyframe with metadata records containing time information, spatial information and scene category information, and preprocess the keyframes;

[0008] S2: Perform object detection and scene classification on keyframes using an open vocabulary detection model: The object detection is driven by a set of target categories described in natural language, and expands the target categories outside the predetermined categories to detect and crop target image patches; The scene classification extracts scene category labels from the overall keyframe image and uses them as scene category information in the metadata record.

[0009] S3: Track the foreground target in a continuous keyframe sequence, assign a tracking identifier to each target and maintain its lifecycle, calculate the quality score for all frames with the same tracking identifier within its lifecycle according to a preset quality evaluation criterion, select the only frame with the highest score as the representative frame for output, and skip the subsequent encoding of the remaining frames.

[0010] S4: An image encoder using a multimodal pre-trained model encodes the target image patch in the representative frame to generate a semantic vector;

[0011] S5: Construct a dual indexing system: Construct a vector index for the semantic vector; construct an inverted index for each field of the metadata record, and store the values ​​of each field in a bitmap structure;

[0012] S6: Receive natural language queries and use a large language model to parse the queries into metadata constraints and semantic query vectors;

[0013] S7: The metadata constraints are used to find the first candidate set that satisfies the constraints in the inverted index, and the semantic query vector is used to retrieve the second candidate set in the vector index. The two retrieval channels are executed in parallel, and the intersection of the first candidate set and the second candidate set is taken to obtain the final retrieval result.

[0014] Furthermore, the target category set described by natural language used for target detection in S2 adopts a hierarchical design, including an animal category layer, a personal item category layer, an urban management object category layer, and a vehicle component category layer, but does not include the target's attribute description; the target's attribute information and observable posture and behavior information are automatically extracted into the semantic vector by the multimodal pre-trained model in S4; the natural language description used for scene classification is a set of public security combat-related scene types.

[0015] Furthermore, the preset quality evaluation criterion in S3 is a multi-factor quality score:

[0016]

[0017] In the formula, The ratio of the target bounding box to the image area. For clarity indicators, The frontal alignment tendency score is estimated based on the target's aspect ratio and the number of visible key points. The occlusion rate is derived from the intersection-union ratio of the target bounding box and its neighboring detection boxes; , , , These are the weighting coefficients.

[0018] Furthermore, the specific structure of the metadata inverted index in S5 is as follows: the time index divides the time information into buckets according to a preset time granularity, and each bucket corresponds to a bitmap that records all target identifiers appearing within that time period; in the device index, each device identifier corresponds to a bitmap that records all target identifiers collected by that device; in the scene index, each scene category label corresponds to a bitmap that records all target identifiers belonging to that scene; the bitmaps are compressed and stored using a roaring bitmap, which divides the integer space into multiple containers, and stores them using ordered arrays, bitmaps, or run-length encoding according to the number of elements in each container.

[0019] Furthermore, the vector index in S5 has a hierarchical structure: the hot data layer constructs an index based on the nearest neighbor graph for features from recent or high-frequency query periods and stores it in high-speed memory; the cold data layer constructs an inverted file index for historical low-frequency features and stores it on disk; the index supports incremental merging, and newly arriving feature batches are compressed and appended to the corresponding layer without the need to rebuild the entire index.

[0020] Furthermore, the specific execution method of the two retrieval channels in S7 is as follows: the first channel searches for the corresponding bitmaps for the time range, device set, and scene label extracted from the query, and obtains the first candidate set that satisfies all metadata constraints through multi-field AND operation; the second channel performs an approximate nearest neighbor search on the vector index to recall Top-K candidates as the second candidate set; the two channels are executed in parallel, and finally the intersection of the two candidate sets is obtained through bitmap AND operation; if the number of intersection results exceeds the threshold, the candidates in the intersection are reordered by calling the image-text matching module based on cross-modal cross-attention.

[0021] Furthermore, it also includes an incremental fine-tuning step: collecting retrieval feedback to form positive sample image-text pairs and negative sample image-text pairs, accumulating them to a trigger threshold, and then injecting a trainable low-rank matrix into the attention projection matrix of the multimodal pre-trained model through a low-rank adapter to perform fine-tuning; the updated adapter weights are then hot-loaded into the online service.

[0022] A cross-modal natural language retrieval system for noisy surveillance videos includes:

[0023] The video access and preprocessing module is used to acquire video streams and generate keyframes, associate each keyframe with metadata records containing time information, spatial information and scene category information, and preprocess the keyframes.

[0024] The object detection and scene classification module is used to drive an open vocabulary detection model to perform object detection on keyframes using a set of object categories described in natural language, detect and crop extended objects outside the predetermined categories into target image patches; and is used to extract scene category labels from the overall image of the keyframe as scene category information in the metadata record.

[0025] The target tracking and redundancy removal module is used to track foreground targets in a continuous keyframe sequence and maintain their lifecycle. It calculates the quality score for all frames of the same target within the lifecycle according to a preset quality evaluation criterion and selects the unique frame with the highest score as the representative frame for output.

[0026] The feature encoding module, based on a multimodal pre-trained model, is used to encode the target image patch in the representative frame and generate a semantic vector;

[0027] The index building module is used to build a vector index for the semantic vector and to build an inverted index stored in a bitmap structure for each field of the metadata record.

[0028] The query parsing module, based on a large language model, is used to receive natural language queries and parse the queries into metadata constraints and semantic query vectors.

[0029] The dual-channel parallel retrieval module is used to constrain the metadata to find the first candidate set in the inverted index and retrieve the semantic query vector to obtain the second candidate set in the vector index. The two channels are executed in parallel and the intersection of the two candidate sets is taken to obtain the final retrieval result.

[0030] The results output and feedback module is used to return the search results to the user interface and feed the feedback data back to the incremental fine-tuning pipeline.

[0031] Furthermore, the system adopts a three-layer architecture: the user interface layer includes a session management module, which supports session creation, persistent storage, and context management for multi-turn dialogues; the functional service layer includes a data access module, a model service module, and a common component module; and the basic resource layer realizes unified management and elastic scheduling of underlying computing resources.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] This invention utilizes an open vocabulary detection model driven by natural language description to expand the searchable targets from traditional fixed categories to any target that can be described by natural language. After encoding by a multimodal model, it automatically captures the target's appearance attributes and behavioral semantics. Simultaneously, it introduces a multi-factor quality evaluation mechanism based on target tracking, selecting the single highest-quality individual frame within the target's complete lifecycle for vectorization, thus suppressing redundant feature expansion at the source. Furthermore, it constructs a dual retrieval system consisting of a metadata inverted index and a semantic vector index, combined with a large language model to perform structured parsing of colloquial queries. This enables parallel execution of metadata constraint filtering and semantic vector retrieval through dual channels, as well as candidate set intersection operations, significantly improving retrieval accuracy and efficiency. The system also supports low-rank adaptive incremental fine-tuning based on retrieval feedback, allowing continuous adaptation to local long-tail scenarios without full retraining, demonstrating good engineering practicality. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the steps of an embodiment of the cross-modal natural language retrieval method for noisy surveillance videos according to the present invention.

[0035] Figure 2 This is a schematic diagram of the dual-channel query parsing of an embodiment of the cross-modal natural language retrieval method for noisy surveillance videos according to the present invention.

[0036] Figure 3 This is a schematic diagram of the overall system architecture of a cross-modal natural language retrieval system for noisy surveillance videos according to a second embodiment of the present invention. Detailed Implementation

[0037] The present invention will be further described in detail below through specific embodiments:

[0038] The specific implementation process is as follows:

[0039] Example 1

[0040] Example 1 is attached. Figure 1 As shown, a cross-modal natural language retrieval method for noisy surveillance videos includes the following steps:

[0041] S1: Acquire video stream and generate keyframes, associate each keyframe with metadata records containing time information, spatial information and scene category information, and preprocess the keyframes;

[0042] S2: Perform object detection and scene classification on keyframes using an open vocabulary detection model: The object detection is driven by a set of target categories described in natural language, and expands the target categories outside the predetermined categories to detect and crop target image patches; The scene classification extracts scene category labels from the overall keyframe image and uses them as scene category information in the metadata record.

[0043] S3: Track the foreground target in a continuous keyframe sequence, assign a tracking identifier to each target and maintain its lifecycle, calculate the quality score for all frames with the same tracking identifier within its lifecycle according to a preset quality evaluation criterion, select the only frame with the highest score as the representative frame for output, and skip the subsequent encoding of the remaining frames.

[0044] S4: An image encoder using a multimodal pre-trained model encodes the target image patch in the representative frame to generate a semantic vector;

[0045] S5: Construct a dual indexing system: Construct a vector index for the semantic vector; construct an inverted index for each field of the metadata record, and store the values ​​of each field in a bitmap structure;

[0046] S6: Receive natural language queries and use a large language model to parse the queries into metadata constraints and semantic query vectors;

[0047] S7: The metadata constraints are used to find the first candidate set that satisfies the constraints in the inverted index, and the semantic query vector is used to retrieve the second candidate set in the vector index. The two retrieval channels are executed in parallel, and the intersection of the first candidate set and the second candidate set is taken to obtain the final retrieval result.

[0048] This solution provides a cross-modal natural language retrieval method for noisy surveillance videos. The entire retrieval process starts with the access of video data, and proceeds sequentially through target detection and scene analysis, keyframe optimization, feature encoding, and index construction. Finally, it responds to the police officer's natural language query and returns the retrieval results.

[0049] Specifically, the first step in the entire retrieval process is to acquire the video stream and generate keyframes. The system connects to multi-source video or image streams from public security front-end cameras, alarm centers, and lower-level view libraries. The video stream subscribes to keyframe JPEG streams and associated metadata pushed by each camera through a Kafka message queue. After the system segments the video stream into keyframes, it associates each keyframe with a metadata record. This metadata record includes time information (such as timestamp), spatial information (such as device_id and geo_location), and scene category information (scene_type). The scene category information is automatically generated by the subsequent scene classification function, while the remaining fields are obtained from the video stream's associated metadata.

[0050] Next, keyframes undergo preprocessing, including noise reduction, format normalization, and quality control, filtering out invalid frames with quality below preset thresholds. For example, by setting minimum brightness thresholds and Laplacian variance sharpness thresholds, approximately 2.6 million valid frames are retained after quality control filtering out approximately 9 million new frames added daily.

[0051] Next, two independent functions are performed on the keyframes using an open-vocabulary object detection big model (such as GroundingDINO or its equivalent).

[0052] The first function is extended object detection—driven by a set of object categories described in natural language, it detects and crops extended objects beyond the predetermined categories (i.e., the traditional "people / motor vehicles / non-motor vehicles") into object image patches. As mentioned earlier, this object category set adopts a hierarchical design, including an animal category layer (such as dogs, cats, birds, etc.), a personal item category layer (such as backpacks, suitcases, etc.), a city management object category layer (such as trash cans, fire extinguishers, cardboard boxes, etc.), and a vehicle component category layer (such as electric scooters, shared bicycles, wheelchairs, etc.). This Prompt set only lists the object categories to be detected and does not include the object's attribute descriptions—the object's attribute information and observable pose and behavior information are automatically extracted into the semantic vector by the subsequent multimodal pre-trained model.

[0053] In specific processing, the input consists of keyframe images and an extended target prompt set, and the output is an expanded target set, including the bounding box, category label, and confidence score of each target. Target bounding boxes with a confidence score higher than a threshold (e.g., 0.32) are cropped from the scene image into target image patches and stored in the encoding queue. For target image patches that are too small or lack sharpness due to motion blur, a lightweight super-resolution network (such as an equivalent scheme of ESRGAN) is used for resolution enhancement and deblurring repair. Patches whose sharpness is still lower than a preset threshold after repair are filtered out.

[0054] The second function is scene classification—extracting scene category labels from the overall keyframe image and storing them as scene category information in the metadata record. The natural language description used for scene classification is a set of scene types relevant to public security operations, such as "road," "intersection," "pedestrian walkway," "parking lot," "school gate," "residential area," "shopping mall entrance," "bus stop," "subway entrance," "hospital entrance," "park / green space," "riverbank," "overpass," "underpass," and "construction site." Specifically, visual features are extracted from the entire keyframe image, and cross-modal similarity is calculated between these features and the textual features of each scene's prompt. The scene category with the highest similarity is taken as the scene label. In actual testing, the Top-1 accuracy for scene classification was approximately 89.5%.

[0055] In videos, the same target often appears repeatedly across multiple consecutive frames. Encoding and storing each frame in a database would generate a large amount of redundant features, leading to a rapid expansion of the vector library. Therefore, this method tracks foreground targets in a continuous keyframe sequence, assigning a tracking identifier to each target and maintaining its complete lifecycle—from the target entering the field of view to leaving it. Specifically, equivalent target tracking algorithms such as ByteTrack or DeepSORT can be used, with each camera independently maintaining an active track list.

[0056] For all frames within the lifecycle of the same tracking identifier, a multi-factor quality score is calculated for each frame. This score comprehensively considers four dimensions: the ratio of the target bounding box to the image area (as a size factor, the larger the area ratio, the more prominent the target is in the image, and the richer the pixel information), the sharpness index (calculated by the variance of the second derivative of the Laplacian of the target image patch; the larger the variance, the sharper the image), the frontal alignment tendency score (estimated based on the target's aspect ratio and the number of visible keypoints; the closer to frontal alignment, the higher the score), and the occlusion rate (derived from the intersection-union ratio of the target bounding box and its neighboring detection boxes; the more severe the occlusion, the lower the score). The four dimensions are comprehensively calculated using weighting coefficients α=0.35, β=0.30, γ=0.20, and δ=0.15, with the specific formula as follows:

[0057]

[0058] In the formula, The ratio of the target bounding box to the image area. For clarity indicators, The frontal alignment tendency score is estimated based on the target's aspect ratio and the number of visible key points. The occlusion rate is derived from the intersection-union ratio (IoU) of the target bounding box and its neighboring detection boxes. In this embodiment, assuming a tracking identifier appears 47 times within its lifecycle, the frame with the highest comprehensive score is selected as the representative frame. Only the target image patch of this representative frame is sent for subsequent feature encoding; the remaining redundant frames are skipped. Actual testing shows that this strategy reduces the daily feature input from 9 million to approximately 2.6 million, a reduction of about 71% to 73%. Furthermore, because it retains the largest and clearest frame among all tracked targets, the retrieval accuracy is improved rather than decreased.

[0059] After feature encoding is completed, an efficient index needs to be built to support subsequent fast retrieval. This method constructs a dual indexing system—a vector index and a metadata inverted index.

[0060] For vector indexing, product quantization compression is implemented on semantic vectors—the original high-dimensional vector space is decomposed into multiple low-dimensional subspaces, each of which is independently quantized and encoded, encoding the original vectors into compact codes, significantly reducing storage requirements and distance calculation overhead. The vector index adopts a hierarchical structure: the hot data layer constructs a nearest neighbor graph-based index (such as HNSW) for features from recent or high-frequency query periods, residing in high-speed memory (such as GPU memory) to achieve millisecond-level nearest neighbor search; the cold data layer constructs an inverted file index (such as IVFADC) for historical low-frequency features and stores it on disk, balancing storage costs and acceptable latency. The index supports incremental merging; newly arriving feature batches are compressed using the same quantization encoding parameters and appended to the corresponding layer without requiring a full index reconstruction.

[0061] Regarding the metadata inverted index, an inverted index is constructed for each field of the metadata record associated with each target, and the values ​​of each field are stored in a bitmap structure. Specifically: the time index bins timestamps according to a preset time granularity (e.g., hours), and each bin corresponds to a bitmap recording all target identifiers appearing within that time period; in the device index, each device identifier corresponds to a bitmap recording all target identifiers collected by that device; in the scene index, each scene category label corresponds to a bitmap recording all target identifiers belonging to that scene. The bitmaps are compressed using a roar bitmap – dividing the 32-bit integer space into... Each container uses an ordered array for sparse containers, a bitmap for dense containers, and run-length encoding for containers with many consecutive values. During retrieval, an AND intersection operation is performed on the roaring bitmaps of multiple metadata fields to locate the target identifier set that satisfies all metadata constraints in sub-millisecond time.

[0062] As attached Figure 2As shown, when police officers need to retrieve a target, they can input conversational query text through the user interface, such as "A person wearing a red shirt and carrying a black backpack riding an electric bike near intersection A around 7:00 AM on October 12, 2023". The system calls a large language model (such as DeepSeek or its equivalent optimized Chinese LLM) for structured parsing and outputs dual-channel information.

[0063] In terms of the metadata channel, the large language model extracts structured constraints from the query, including the time range (calculated from the colloquial phrase "around 7 a.m." to a precise timestamp interval, such as 7:00-8:00 on October 12, 2023), the device set (the spatial range "near intersection A" is mapped to the corresponding device_id set through GIS device location association), and scene tags. In terms of the semantic channel, the large language model rewrites the retrieval intent portion of the query ("a person wearing a red shirt and carrying a black backpack riding an electric bike") into a canonical semantic description, which is then encoded into a query vector by a text encoder.

[0064] For example, if the input query is "a yellow husky that appeared near the school yesterday afternoon", the large language model outputs structured JSON. The metadata channel extracts the time range (e.g., 2025-05-27 13:00:00 to 19:00:00), the device set (corresponding locations near the school), and the scene label ("school"). The semantic channel outputs the standardized semantic description "yellow husky". The large language model needs to have Chinese semantic resolution capabilities to handle fuzzy time expressions, fuzzy spatial descriptions, and colloquial behavioral descriptions in police officers' colloquial queries.

[0065] After parsing, the two types of information are sent to two independent execution channels, which are executed in parallel without blocking each other.

[0066] Based on the parsed metadata constraints, the metadata channel performs multi-field AND queries in the metadata inverted index. Specifically, it searches for the corresponding roar bitmaps for the time range, device set, and scene tag, takes the OR union of the roar bitmaps of all hour buckets covered by the time range, takes the OR union of the roar bitmaps of all devices in the device set, and does the same for scene tags. Finally, it takes the AND intersection of the three to obtain the target identifier set that satisfies all metadata constraints. The AND operation of the Roaring Bitmap is accelerated using SIMD instructions, and the execution time on a set of hundreds of millions of identifiers is typically less than 5ms.

[0067] The semantic channel feeds the generated query vector into the FAISS engine, performs an approximate nearest neighbor search on the hierarchical vector index, recalls the Top-500 candidates, and obtains a set of semantically matched target identifiers. .

[0068] After the two channels have been executed in parallel, the intersection of the two candidate sets is obtained by performing an AND operation on the roaring bitmap. Only targets that simultaneously satisfy metadata constraints and semantic similarity are retained. If the number of intersection results is still large (e.g., more than 20), the candidate in the intersection is processed by the image-text matching module based on cross-modal cross-attention to calculate fine matching scores and re-rank them, and the top 20 are selected as the final output.

[0069] The technical advantages of this parallel retrieval method are as follows: the metadata channel significantly narrows the candidate set from hundreds of millions to a subset that meets the spatiotemporal constraints, the semantic channel recalls semantically similar candidates from the full vector library, and finally takes the intersection through efficient roaring bitmap AND operation, avoiding the computational overhead wasted by a large number of candidates that are destined to be filtered in the traditional serial method of "full recall first and then filtering".

[0070] The final result is returned to the user interface in the form of keyframe screenshots, associated video segment time intervals, and semantic summaries. The semantic summaries are automatically generated by a large language model based on extended labels of the hit keyframes, such as "The target appeared at 07:12:33, at camera CH-038 eastbound at intersection A, wearing a red shirt, carrying a black backpack, and riding an electric bike."

[0071] This method also includes an optional incremental fine-tuning step. The system collects police officers' confirmation, rejection feedback, and ranking adjustments of the returned search results, forming positive and negative image-text pairs. After accumulating to a trigger threshold (e.g., 500 pairs of positive and negative samples), a trainable low-rank matrix is ​​injected into the attention projection matrix of the multimodal pre-trained model via a low-rank adapter (LoRA) to perform mini-batch fine-tuning. Specifically, the LoRA rank is 8 (to prevent overfitting on small datasets), and the injection position is the Wq / Wv of the last-order attention of the visual encoder and the equivalent position of the text encoder. The updated adapter weights are hot-loaded to the online service without requiring full retraining or long service interruptions. This mechanism enables the system to automatically accumulate training data through the supervision signals implicit in the daily search behavior of police officers, achieving continuous evolution of the model's capabilities.

[0072] Example 2

[0073] Example 2 is attached. Figure 3 As shown, this is a cross-modal natural language retrieval system for noisy surveillance videos. The system adopts a three-layer architecture, consisting of a user interface layer, a functional service layer, and a basic resource layer from top to bottom. Each layer communicates through a standard interface.

[0074] The user interface layer serves as the entry point for officers to interact with the system. It includes a session management module that supports session creation, persistent storage, and storage and organization of context messages for multi-turn dialogues, enabling the system to remember officers' query history during continuous investigations.

[0075] The functional service layer is the core capability layer, which includes the data access module (real-time access, preprocessing and structured storage of video / image streams), the data and model service module (full-link AI model services such as text / image feature extraction, multimodal large models, vector retrieval and comparison, and language large models), and the common component module (middleware such as Kafka message queue, PostgreSQL relational database, MPP massively parallel processing database, file storage and vector database).

[0076] The basic resource layer is based on a cloud resource management platform, which enables unified management, elastic scheduling, and monitoring and alarming of underlying physical resources (CPU, GPU, memory, disk, network), and supports hybrid deployment of CPU and GPU.

[0077] Specifically, the system in this embodiment includes the following functional modules:

[0078] The video access and preprocessing module acquires video streams and generates keyframes. Each keyframe is associated with metadata records containing time, spatial, and scene category information, and preprocesses the keyframes. This module connects to multi-source video or image streams from public security front-end cameras, alarm centers, and lower-level view libraries. It subscribes to keyframe JPEG streams and associated metadata pushed by each camera via a Kafka message queue. After the video stream is segmented into keyframes, metadata records (timestamp, device ID, GPS coordinates, etc.) are associated, and noise reduction, format standardization, and quality control filtering are performed. For example, minimum brightness thresholds and Laplacian variance sharpness thresholds are set to filter out invalid frames with quality below the preset thresholds. Of the approximately 9 million new frames added daily, about 2.6 million are retained after quality control filtering.

[0079] The object detection and scene classification module drives an open-vocabulary detection model to perform object detection on keyframes using a set of object categories described in natural language. It detects and crops extended objects outside the predetermined categories into object image patches. It also extracts scene category labels from the entire keyframe image as scene category information in the metadata record. This module calls an open-vocabulary object detection model (such as GroundingDINO) to drive keyframe detection with a hierarchical natural language Prompt set covering animal categories, personal belongings categories, urban management object categories, and vehicle component categories. Detected extended objects (with confidence scores above a threshold such as 0.32) are cropped into object image patches; simultaneously, scene category labels (such as "school," "intersection," "parking lot," etc.) are extracted from the entire keyframe image and written into the metadata record. For object image patches that are too small or lack sufficient clarity, a lightweight super-resolution network is used for repair.

[0080] The target tracking and redundancy removal module tracks and maintains the lifecycle of foreground targets in a continuous keyframe sequence. For the same target within its lifecycle, a quality score is calculated according to a preset quality evaluation criterion, and the single frame with the highest score is selected as the representative frame for output. This module independently maintains an active track list for each camera and assigns a tracking label to each detected foreground target using algorithms such as ByteTrack or DeepSORT. The quality evaluation criterion is a multi-factor quality score, comprehensively considering four dimensions: the ratio of the target bounding box to the image area, the sharpness index (Laplacian variance), the frontal alignment tendency score, and the occlusion rate, with weighting coefficients of 0.35, 0.30, 0.20, and 0.15, respectively. Only when the tracking label's lifecycle ends is the frame with the highest score selected as the representative frame for output.

[0081] The feature encoding module, based on a multimodal pre-trained model (such as the BLIP series), encodes target image patches in representative frames to generate semantic vectors. The image encoder, based on the ViT visual backbone and the Q-Former query-based Transformer, outputs visual feature vectors. These semantic vectors automatically capture the target's appearance attributes, observable pose and behavior information, and visual semantics. Specifically, the base model includes a ViT-L / 14 visual encoder and a Q-Former information bottleneck module, with an image input resolution of 384×384.

[0082] The index building module is used to construct vector indexes for semantic vectors and inverted indexes for each field of metadata records, stored in a bitmap structure. For vector indexes, a hierarchical vector index is constructed after product quantization compression of semantic vectors: the hot data layer uses a nearest neighbor graph-based index (such as HNSW) resident in GPU memory for recent or high-frequency query data, while the cold data layer uses an inverted file index (such as IVFADC) stored on disk for historical low-frequency data. For metadata inverted indexes, roaring bitmap inverted indexes are constructed for fields such as time, device, and scene: the time index is bucketed according to a preset time granularity, the device index is bucketed according to device identifier, and the scene index is bucketed according to scene category label. The roaring bitmap divides the integer space into multiple containers, using ordered array storage, bitmap storage, or run-length encoding depending on the number of elements in each container. The index supports incremental merging without requiring a full rebuild.

[0083] The query parsing module, based on a large language model (such as DeepSeek), receives natural language queries and parses them into metadata constraints and semantic query vectors. This module receives colloquial query text input by police officers (e.g., "A person wearing a red shirt and carrying a black backpack riding an electric bike near intersection A at around 7:00 AM on October 12, 2023"), calls the large language model for structured parsing, and outputs structured constraints (time range, device set, scene label) for the metadata channel and a standardized semantic description for the semantic channel. The large language model needs to have Chinese semantic resolution capabilities to handle fuzzy temporal expressions, fuzzy spatial descriptions, and colloquial behavioral descriptions.

[0084] The dual-channel parallel retrieval module is used to search for the first candidate set in the inverted index based on metadata constraints, and to retrieve the second candidate set by searching the vector index based on the semantic query vector. The two channels execute in parallel and take the intersection of the two candidate sets to obtain the final retrieval result. The first channel performs a multi-field AND query in the Roaring Bitmap inverted index based on metadata constraints. The second channel performs an approximate nearest neighbor search on the hierarchical vector index to recall the Top-500 candidates. After the two channels are executed in parallel, their intersection is obtained through a roaring bitmap AND operation. If the number of intersection results exceeds 20, the image-text matching module based on cross-modal cross-attention is called for reordering.

[0085] The results output and feedback module returns search results to the user interface in the form of keyframe screenshots, associated video clip time intervals, and semantic summaries, and feeds feedback data back to the LoRA incremental fine-tuning pipeline. The semantic summary is automatically generated by the large language model based on expanded tags of the hit keyframes. Feedback data (officer confirmations, rejections, and ranking adjustments) forms positive and negative image-text pairs, which are then fed back to the LoRA incremental fine-tuning pipeline.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A cross-modal natural language retrieval method for noisy surveillance videos, characterized in that, Including the following steps: S1: Acquire video stream and generate keyframes, associate each keyframe with metadata records containing time information, spatial information and scene category information, and preprocess the keyframes; S2: Perform object detection and scene classification on keyframes using an open vocabulary detection model: The object detection is driven by a set of target categories described in natural language, and expands the target categories outside the predetermined categories to detect and crop them into target image patches; The scene classification extracts scene category labels from the overall keyframe image and uses them as scene category information in the metadata record. S3: Track the foreground target in a continuous keyframe sequence, assign a tracking identifier to each target and maintain its lifecycle, calculate the quality score for all frames with the same tracking identifier within its lifecycle according to a preset quality evaluation criterion, select the only frame with the highest score as the representative frame for output, and skip the subsequent encoding of the remaining frames. S4: An image encoder using a multimodal pre-trained model encodes the target image patch in the representative frame to generate a semantic vector; S5: Construct a dual indexing system: Construct a vector index for the semantic vector; construct an inverted index for each field of the metadata record, and store the values ​​of each field in a bitmap structure; S6: Receive natural language queries and use a large language model to parse the queries into metadata constraints and semantic query vectors; S7: The metadata constraints are used to find the first candidate set that satisfies the constraints in the inverted index, and the semantic query vector is used to retrieve the second candidate set in the vector index. The two retrieval channels are executed in parallel, and the intersection of the first candidate set and the second candidate set is taken to obtain the final retrieval result.

2. The cross-modal natural language retrieval method for noisy surveillance videos according to claim 1, characterized in that: The target category set described by natural language used in S2 for target detection adopts a hierarchical design, including animal category layer, personal item category layer, urban management object category layer and vehicle component category layer, but does not include target attribute description; the target attribute information and observable posture and behavior information are automatically extracted into semantic vectors by the multimodal pre-trained model in S4; the natural language description used for scene classification is a set of public security combat-related scene types.

3. The cross-modal natural language retrieval method for noisy surveillance videos according to claim 1, characterized in that: The preset quality evaluation criterion in S3 is a multi-factor quality score: In the formula, The ratio of the target bounding box to the image area. For clarity indicators, The frontal alignment tendency score is estimated based on the target's aspect ratio and the number of visible key points. The occlusion rate is derived from the intersection-union ratio of the target bounding box and its neighboring detection boxes; , , , These are the weighting coefficients.

4. The cross-modal natural language retrieval method for noisy surveillance videos according to claim 1, characterized in that: The specific structure of the inverted index of metadata in S5 is as follows: the time index divides time information into buckets according to a preset time granularity, and each bucket corresponds to a bitmap that records all target identifiers appearing within that time period; in the device index, each device identifier corresponds to a bitmap that records all target identifiers collected by that device; in the scene index, each scene category label corresponds to a bitmap that records all target identifiers belonging to that scene; the bitmap is compressed and stored using a roaring bitmap, which divides the integer space into multiple containers, and stores them using ordered arrays, bitmaps, or run-length encoding according to the number of elements in each container.

5. The cross-modal natural language retrieval method for noisy surveillance videos according to claim 1, characterized in that: The vector index in S5 has a hierarchical structure: the hot data layer constructs an index based on the nearest neighbor graph for features from recent or high-frequency query periods and stores it in high-speed memory; the cold data layer constructs an inverted file index for historical low-frequency features and stores it on disk; the index supports incremental merging, and newly arrived feature batches are compressed and appended to the corresponding layer without the need to rebuild the entire index.

6. The cross-modal natural language retrieval method for noisy surveillance videos according to claim 1, characterized in that: The specific execution method of the two retrieval channels in S7 is as follows: the first channel searches for the corresponding bitmaps for the time range, device set, and scene label extracted from the query, and obtains the first candidate set that satisfies all metadata constraints through multi-field AND operation; the second channel performs an approximate nearest neighbor search on the vector index to recall Top-K candidates as the second candidate set; the two channels are executed in parallel, and finally the intersection of the two candidate sets is obtained through bitmap AND operation; if the number of intersection results exceeds the threshold, the candidates in the intersection are reordered by calling the image-text matching module based on cross-modal cross-attention.

7. A cross-modal natural language retrieval method for noisy surveillance videos according to any one of claims 1 to 6, characterized in that: It also includes an incremental fine-tuning step: collecting retrieval feedback to form positive sample image-text pairs and negative sample image-text pairs, accumulating them to a trigger threshold, and then injecting a trainable low-rank matrix into the attention projection matrix of the multimodal pre-trained model through a low-rank adapter to perform fine-tuning; The updated adapter weights are hot-loaded to the online service.

8. A cross-modal natural language retrieval system for noisy surveillance videos, characterized in that, include: The video access and preprocessing module is used to acquire video streams and generate keyframes, associate each keyframe with metadata records containing time information, spatial information and scene category information, and preprocess the keyframes. The object detection and scene classification module is used to drive the open vocabulary detection model to perform object detection on keyframes using a set of object categories described in natural language, and to detect and crop extended objects outside the predetermined categories into object image patches. And to extract scene category labels from the overall image of the keyframe, as scene category information in the metadata record; The target tracking and redundancy removal module is used to track foreground targets in a continuous keyframe sequence and maintain their lifecycle. It calculates the quality score for all frames of the same target within the lifecycle according to a preset quality evaluation criterion and selects the unique frame with the highest score as the representative frame for output. The feature encoding module, based on a multimodal pre-trained model, is used to encode the target image patch in the representative frame and generate a semantic vector; The index building module is used to build a vector index for the semantic vector and to build an inverted index stored in a bitmap structure for each field of the metadata record. The query parsing module, based on a large language model, is used to receive natural language queries and parse the queries into metadata constraints and semantic query vectors. The dual-channel parallel retrieval module is used to constrain the metadata to find the first candidate set in the inverted index and retrieve the semantic query vector to obtain the second candidate set in the vector index. The two channels are executed in parallel and the intersection of the two candidate sets is taken to obtain the final retrieval result. The results output and feedback module is used to return the search results to the user interface and feed the feedback data back to the incremental fine-tuning pipeline.

9. A cross-modal natural language retrieval system for noisy surveillance video according to claim 8, characterized in that, The system adopts a three-layer architecture: the user interface layer includes a session management module, which supports session creation, persistent storage, and context management for multi-turn dialogues; the functional service layer includes a data access module, a model service module, and a common component module; and the basic resource layer realizes unified management and elastic scheduling of underlying computing resources.

Images