Multi-target accurate retrieval method and system in security video

Abstract

The invention provides a multi-target accurate retrieval method and system in a security video, and relates to the field of interdisciplinary application, and the method is characterized in that the method comprises the following steps: carrying out the standardized preprocessing of an original security video, and deploying an improved YOLOv8 model on a preprocessed key frame to carry out the high-precision multi-target detection. The method has the advantages that the appearance, motion and semantic complementary features of the target are synchronously extracted by improving the YOLOv8 model, cross-frame association and trajectory modeling are realized by adopting an enhanced DeepSORT tracker, and a compact feature index database and a three-level progressive retrieval mechanism are constructed in combination with a locality sensitive hashing algorithm; according to the method, the multi-target retrieval precision and efficiency in massive videos are remarkably improved, the target relevance and the complex scene semantic understanding ability are enhanced, meanwhile, the calculation overhead is greatly reduced through rapid approximate matching and a hierarchical retrieval strategy, and the requirements of security application for high accuracy, rapid response and real-time intelligent research and judgment are met.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary applications, and in particular to a method and system for accurate multi-target retrieval in security videos. Background Technology

[0002] With the acceleration of urbanization and the increasing demand for public safety, security video surveillance systems have become an important part of urban security. Although current security video systems are relatively advanced in terms of data collection and storage, there are still many problems in quickly and accurately retrieving specific targets from massive amounts of video data.

[0003] Currently, with the acceleration of urbanization and the increase in public safety demands, although security video surveillance systems are becoming increasingly sophisticated in terms of data collection and storage, achieving fast and accurate multi-target retrieval in massive amounts of video still faces many challenges: existing technologies generally suffer from low retrieval accuracy, poor efficiency, weak multi-target correlation, lack of semantic understanding capabilities, and insufficient real-time performance in complex environments, making it difficult to meet the demands of high accuracy, high response speed, and intelligent judgment in actual security scenarios. Therefore, there is an urgent need for a new video analysis method that integrates appearance, motion, and semantic features and has efficient indexing and multi-level retrieval mechanisms. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for accurate multi-target retrieval in security videos, which solves the technical problems of low retrieval accuracy, poor efficiency, weak multi-target correlation, lack of semantic understanding, and insufficient real-time performance in existing security video surveillance systems.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0006] A method for accurate multi-target retrieval in security videos, characterized by including standardized preprocessing of the original security video, including frame rate adjustment and resolution standardization, and video segmentation combined with scene change detection technology, and on this basis, using a key frame extraction strategy to select representative frame sequences;

[0007] On the preprocessed keyframes, an improved YOLOv8 model is deployed to perform high-precision multi-object detection and obtain detection results. The YOLOv8 model not only outputs target bounding boxes, but also simultaneously extracts three types of complementary features: appearance features, motion features, and semantic features.

[0008] The detection results are input into the improved DeepSORT tracker, which uses a fusion strategy of appearance features and motion features to achieve cross-frame target association, perform cross-frame tracking on the detected targets, and construct the motion feature trajectory of the targets.

[0009] The static appearance is obtained by fusing the detection results and motion feature trajectories. A unified high-dimensional target representation vector is generated by combining appearance features, motion features, and semantic features or motion trajectory information. The vector is then approximated by the nearest neighbor mapping algorithm using the locality-sensitive hashing algorithm to construct a compact feature index library that supports fast similarity queries.

[0010] A three-level progressive retrieval mechanism is implemented based on the established feature index library, including coarse retrieval, fine retrieval, and related retrieval;

[0011] The output search results are quantified with confidence, deduplicated, sorted, and visualized to generate a structured and interpretable multi-objective search report.

[0012] An improved sensitive hashing algorithm is used for fast similarity retrieval of high-dimensional target representation vectors that integrate appearance, motion, and semantic information, thereby constructing a compact feature index library. Each historical target (such as pedestrians and vehicles) is represented as a high-dimensional vector integrating appearance, motion, and semantic information, and then mapped to multiple hash buckets for storage using a sensitive hashing algorithm function. When a user initiates a search request such as "find the person wearing red clothes who walks to the left," the system transforms the query conditions into a corresponding vector and calculates the hash value using the same sensitive hashing algorithm function. Subsequently, it only needs to perform precise similarity matching within the target hash bucket and its neighboring buckets, thereby significantly narrowing the search scope and greatly reducing computational overhead.

[0013] In the three-level progressive retrieval mechanism, the coarse detection method uses the locality-sensitive hashing algorithm to perform a fast approximate nearest neighbor search on the high-dimensional target representation vector, efficiently filtering out a candidate result set that is initially similar to the query target from a large-scale index, significantly reducing the subsequent computational complexity;

[0014] Fine-grained retrieval builds upon coarse-grained retrieval results by performing high-precision similarity calculations through more refined feature comparisons (such as weighted fusion of appearance, motion, and semantic features), eliminating false matches and improving retrieval accuracy.

[0015] Furthermore, the association retrieval combines the target's cross-frame motion trajectory, spatiotemporal context relationship, and event logic to mine behavioral associations or co-occurrence patterns between targets, realizing context-enhanced retrieval based on scene semantics, thereby supporting in-depth backtracking and intelligent analysis of complex security events.

[0016] Optionally, the improved YOLOv8 model introduces a multi-branch feature fusion module on the basis of the original YOLOv8 architecture, which is used to simultaneously extract the appearance features, motion features and semantic features of the target. The appearance features obtain high-resolution target texture information by enhancing the spatial detail preservation ability in the backbone network. The motion features capture the target's dynamic attributes by embedding a lightweight optical flow estimation subnetwork or an inter-frame difference guidance mechanism. The semantic features generate an embedding vector with high-level semantic consistency by using a projection head aligned with a pre-trained vision-language model.

[0017] Optionally, the improved DeepSORT tracker utilizes appearance features to construct an enhanced deep appearance embedding model and dynamically optimizes the target appearance template through an online update strategy. Semantic features are introduced as high-level constraints to perform semantic consistency correction on the matching cost matrix during trajectory association, effectively suppressing ID switching issues caused by occlusion, deformation, or lighting changes.

[0018] Optionally, the appearance features are extracted based on the ResNet-50 deep neural network, including visual information such as the pedestrian's clothing, hairstyle, and body posture, as well as the target's spatial trajectory, temporal trajectory, and behavioral trajectory, wherein the behavioral trajectory is obtained by analyzing the target's speed changes, movement direction, and dwelling patterns.

[0019] ResNet-50 is a deep convolutional neural network based on residual learning, proposed by Microsoft Research. It belongs to the typical 50-layer structure of the ResNet (Residual Network) series. Its core innovation lies in the introduction of "residual blocks," which directly add the input to the output of the convolutional layer through skip connections. This effectively alleviates the gradient vanishing and performance degradation problems during the training of deep networks, enabling the network to be trained stably to deeper layers.

[0020] Optionally, the spatiotemporal index structure employs a two-dimensional time-space index to achieve rapid target localization and filtering of relevant video segments. In the multi-level retrieval, coarse retrieval uses the spatiotemporal index for rapid filtering, fine retrieval performs precise matching based on feature similarity, and association retrieval utilizes a target association model to identify different manifestations of the same target.

[0021] A multi-target precision retrieval system in security videos, characterized in that it includes: a data acquisition module, used to perform standardized preprocessing on the original security videos, including frame rate adjustment and standardization, resolution standardization, and video segmentation combined with scene change detection technology, and on this basis, a key frame extraction strategy is adopted to select representative frame sequences, and the data acquisition module supports multiple video format inputs and real-time stream processing;

[0022] The intelligent analysis module is used to deploy an improved YOLOv8 model on preprocessed keyframes for high-precision multi-target detection. The intelligent analysis module includes a target detection unit, a feature extraction unit, and a target tracking unit.

[0023] The index management module is used to perform an approximate nearest neighbor mapping on the vector using the locality-sensitive hashing algorithm, and to build a compact feature index library that supports fast similarity queries. The index management module builds and maintains the feature index and the spatiotemporal index.

[0024] The retrieval service module is used to perform a three-level progressive retrieval mechanism based on the index structure established in the above steps, including coarse retrieval, fine retrieval and related retrieval. The retrieval service module provides multi-level retrieval algorithms and result ranking.

[0025] The system management module is used to generate structured and interpretable multi-target retrieval reports, supporting security incident retrospective and intelligent analysis. The system management module provides a user interface and permission management functions.

[0026] Optionally, the target detection unit employs an improved YOLOv8 deep learning model, capable of detecting various target types, including pedestrians, vehicles, etc.

[0027] Optionally, the feature extraction unit can extract three types of features: appearance features, motion features, and semantic features, and fuse them into a unified target representation vector.

[0028] Optionally, the index management module uses an improved locality-sensitive hashing algorithm to construct a feature index, which supports dynamic updates and maintenance.

[0029] Optionally, the retrieval service module supports multiple retrieval methods, including feature-based retrieval, spatiotemporal-based retrieval, and behavior-based retrieval.

[0030] The beneficial effects of this invention are as follows: by introducing an improved YOLOv8 model to simultaneously extract three complementary features of the target: appearance, motion, and semantics, and by using an improved DeepSORT tracker to achieve cross-frame target association and trajectory modeling, combined with a compact feature index library built based on the locality-sensitive hashing algorithm and a three-level progressive retrieval mechanism, the accuracy and efficiency of multi-target retrieval in massive video data can be significantly improved, effectively enhancing multi-target association and semantic understanding of complex scenes. At the same time, the computational overhead is greatly reduced through fast approximate matching and hierarchical retrieval strategies, thereby meeting the real-time requirements of high accuracy, high response speed, and intelligent judgment in security applications. Attached Figure Description

[0031] Figure 1 This is an overall flowchart of a multi-target accurate retrieval method in security video according to the present invention.

[0032] Figure 2 This is an architecture diagram of a multi-target precise retrieval system in security video according to the present invention.

[0033] Figure 3 The diagram shows the multi-branch feature fusion network structure of YOLOv8, which is a system improvement of the multi-target accurate retrieval method in security videos according to the present invention. Detailed Implementation

[0034] To make the content of this invention easier to understand, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Identical components are represented by the same reference numerals. It should be noted that the terms "front," "rear," "left," "right," "up," and "down" used in the following description refer to directions in the accompanying drawings, while the terms "inner" and "outer" refer to directions toward or away from the geometric center of a specific component, respectively.

[0035] A method for accurate multi-target retrieval in security videos, characterized by the following steps:

[0036] The original security video is preprocessed in a standardized manner, including frame rate adjustment and resolution standardization, and video segmentation is performed in combination with scene change detection technology. Based on this, a key frame extraction strategy is adopted to select representative frame sequences.

[0037] On the preprocessed keyframes, an improved YOLOv8 model is deployed for high-precision multi-object detection. This model not only outputs the target bounding boxes, but also extracts three types of complementary features simultaneously: appearance features, motion features, and semantic features.

[0038] The frame-by-frame detection results from the above steps are input into the improved DeepSORT tracker. By using the fusion strategy of appearance similarity and motion dynamics model, cross-frame target association is achieved, cross-frame tracking is performed on the detected targets, and the motion trajectory model of the targets is constructed.

[0039] By integrating the static appearance, dynamic motion, and high-level semantic information obtained from the above two steps, a unified high-dimensional target representation vector is generated through feature weighting or attention mechanism. An improved locality-sensitive hashing algorithm is used to perform an approximate nearest neighbor mapping on this vector, and a compact feature index library that supports fast similarity queries is constructed.

[0040] The index structure established based on the above steps is used for a three-level progressive retrieval mechanism, including coarse retrieval, fine retrieval, and related retrieval.

[0041] The system quantifies the confidence level of the output search results, performs deduplication, sorting, and visualization integration, and generates structured and interpretable multi-target search reports, supporting security incident retrospective and intelligent analysis.

[0042] Sensitive hashing (SHA) is used for fast similarity retrieval of high-dimensional target representation vectors that integrate appearance, motion, and semantic information, thereby constructing a compact feature index. Each historical target (such as a pedestrian or vehicle) is represented as a high-dimensional vector integrating appearance, motion, and semantic information, and then mapped to multiple hash buckets for storage using the sensitive hashing algorithm function. When a user initiates a search request such as "find the person wearing red who walks to the left," the system transforms the query conditions into a corresponding vector and calculates the hash value using the same sensitive hashing algorithm function. Subsequently, it only needs to perform precise similarity matching within the target hash bucket and its neighboring buckets, thus significantly narrowing the search scope and greatly reducing computational overhead.

[0043] In the three-level progressive retrieval mechanism, the coarse detection method uses the locality-sensitive hashing algorithm to perform a fast approximate nearest neighbor search on the high-dimensional target representation vector, efficiently filtering out a candidate result set that is initially similar to the query target from a large-scale index, significantly reducing the subsequent computational complexity;

[0044] Fine-grained retrieval builds upon coarse-grained retrieval results by performing high-precision similarity calculations through more refined feature comparisons (such as weighted fusion of appearance, motion, and semantic features), eliminating false matches and improving retrieval accuracy.

[0045] Furthermore, the association retrieval combines the target's cross-frame motion trajectory, spatiotemporal context relationship, and event logic to mine behavioral associations or co-occurrence patterns between targets, realizing context-enhanced retrieval based on scene semantics, thereby supporting in-depth backtracking and intelligent analysis of complex security events.

[0046] Optionally, the improved YOLOv8 model introduces a multi-branch feature fusion module on the basis of the original YOLOv8 architecture, which is used to simultaneously extract the appearance features, motion features and semantic features of the target. The appearance features obtain high-resolution target texture information by enhancing the spatial detail preservation ability in the backbone network. The motion features capture the target's dynamic attributes by embedding a lightweight optical flow estimation subnetwork or an inter-frame difference guidance mechanism. The semantic features generate an embedding vector with high-level semantic consistency by using a projection head aligned with a pre-trained vision-language model.

[0047] Optionally, the improved DeepSORT tracker utilizes appearance features to construct an enhanced deep appearance embedding model and dynamically optimizes the target appearance template through an online update strategy. Semantic features are introduced as high-level constraints to perform semantic consistency correction on the matching cost matrix during trajectory association, effectively suppressing ID switching issues caused by occlusion, deformation, or lighting changes.

[0048] Optionally, the appearance features are extracted based on the ResNet-50 deep neural network, including visual information such as the pedestrian's clothing, hairstyle, and body posture, as well as the target's spatial trajectory, temporal trajectory, and behavioral trajectory, wherein the behavioral trajectory is obtained by analyzing the target's speed changes, movement direction, and dwelling patterns.

[0049] ResNet-50 is a deep convolutional neural network based on residual learning, proposed by Microsoft Research. It belongs to the typical 50-layer structure of the ResNet (Residual Network) series. Its core innovation lies in the introduction of "residual blocks," which directly add the input to the output of the convolutional layer through skip connections. This effectively alleviates the gradient vanishing and performance degradation problems during the training of deep networks, enabling the network to be trained stably to deeper layers.

[0050] Optionally, the spatiotemporal index structure employs a two-dimensional time-space index to achieve rapid target localization and filtering of relevant video segments. In the multi-level retrieval, coarse retrieval uses the spatiotemporal index for rapid filtering, fine retrieval performs precise matching based on feature similarity, and association retrieval utilizes a target association model to identify different manifestations of the same target.

[0051] A multi-target precision retrieval system in security videos, characterized in that it includes: a data acquisition module, used to perform standardized preprocessing on the original security videos, including frame rate adjustment and standardization, resolution standardization, and video segmentation combined with scene change detection technology, and on this basis, a key frame extraction strategy is adopted to select representative frame sequences, and the data acquisition module supports multiple video format inputs and real-time stream processing;

[0052] The intelligent analysis module is used to deploy an improved YOLOv8 model on preprocessed keyframes for high-precision multi-target detection. The intelligent analysis module includes a target detection unit, a feature extraction unit, and a target tracking unit.

[0053] The index management module is used to perform an approximate nearest neighbor mapping on the vector using the locality-sensitive hashing algorithm, and to build a compact feature index library that supports fast similarity queries. The index management module builds and maintains feature indexes and spatiotemporal indexes.

[0054] The retrieval service module is used to perform a three-level progressive retrieval mechanism based on the index structure established in the above steps, including coarse retrieval, fine retrieval and related retrieval. The retrieval service module provides multi-level retrieval algorithms and result ranking.

[0055] The system management module is used to generate structured and interpretable multi-target retrieval reports, supporting security incident retrospective and intelligent analysis. The system management module provides a user interface and permission management functions.

[0056] Optionally, the target detection unit employs an improved YOLOv8 deep learning model, capable of detecting various target types, including pedestrians, vehicles, etc.

[0057] Optionally, the feature extraction unit can extract three types of features: appearance features, motion features, and semantic features, and fuse them into a unified target representation vector.

[0058] Optionally, the index management module uses an improved locality-sensitive hashing algorithm to construct a feature index, which supports dynamic updates and maintenance.

[0059] Optionally, the retrieval service module supports multiple retrieval methods, including feature-based retrieval, spatiotemporal-based retrieval, and behavior-based retrieval.

[0060] Example: Suppose a smart security system is installed in a city center area for 24 / 7 monitoring to ensure public safety. The area has recently experienced continuous heavy rain, resulting in severe street flooding and poor nighttime lighting, increasing the difficulty of monitoring. Furthermore, pedestrian and vehicle obstructions may occur due to trees, billboards, and other factors. The solution is as follows: For severe weather conditions, high dynamic range (HDR) technology and night vision enhancement technology are used to adjust and unify the frame rate and standardize the resolution of the original video. Scene change detection technology is used for video segmentation, and the keyframe extraction strategy focuses on selecting representative frame sequences that reflect the impact of extreme weather and changes in lighting conditions.

[0061] Considering the challenges of extreme environments, we further optimized the YOLOv8 model, particularly enhancing its robustness to low light and reflection interference in appearance feature extraction. The motion feature extraction module was specifically designed to adapt to the effects of fast-moving objects (such as leaves blown by the wind) and water flow. Semantic feature extraction combines a pre-trained vision-language model to understand and distinguish different types of occlusions. To address the occlusion problem, the DeepSORT tracker utilizes a fusion strategy of appearance similarity and motion dynamics models while introducing a high-level constraint mechanism based on semantic features. This helps reduce ID switching caused by temporary disappearance or overlap of targets.

[0062] The target representation vector, generated by fusing the appearance, motion, and semantic information obtained in the above steps, is used to construct a compact feature index library through an improved locality-sensitive hashing algorithm. This enables efficient target querying even in complex contexts. The coarse search stage uses locality-sensitive hashing to quickly filter out potentially relevant result sets; the fine search stage eliminates mismatches by meticulously comparing appearance, motion, and semantic features; and the association search delves into the spatiotemporal relationships and behavioral patterns between targets, supporting in-depth backtracking of complex events.

[0063] Ultimately, the system will provide structured and interpretable search reports based on user needs (such as finding people walking in heavy rain with umbrellas during a specific time period), including the target's movement trajectory and behavior analysis, to help security personnel quickly locate and assess the situation.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for accurate multi-target retrieval in security videos, characterized in that, include: The original security video is preprocessed in a standardized manner, including frame rate adjustment and resolution standardization, and video segmentation is performed in combination with scene change detection technology. Based on this, a key frame extraction strategy is adopted to select representative frame sequences. On the preprocessed keyframes, an improved YOLOv8 model is deployed to perform high-precision multi-object detection and obtain detection results. The YOLOv8 model not only outputs target bounding boxes, but also simultaneously extracts three types of complementary features: appearance features, motion features, and semantic features. The detection results are input into the improved DeepSORT tracker, which uses a fusion strategy of appearance features and motion features to achieve cross-frame target association, perform cross-frame tracking on the detected targets, and construct the motion feature trajectory of the targets. The static appearance is obtained by fusing the detection results and motion feature trajectories. A unified high-dimensional target representation vector is generated by combining appearance features, motion features, and semantic features or motion trajectory information. The vector is then approximated by the nearest neighbor mapping algorithm using the locality-sensitive hashing algorithm to construct a compact feature index library that supports fast similarity queries. A three-level progressive retrieval mechanism is implemented based on the established feature index library, including coarse retrieval, fine retrieval, and related retrieval; The output search results are quantified with confidence, deduplicated, sorted, and visualized to generate a structured and interpretable multi-objective search report.

2. The method for accurate multi-target retrieval in security videos according to claim 1, characterized in that, The improved YOLOv8 model introduces a multi-branch feature fusion module on the basis of the original YOLOv8 architecture, which is used to simultaneously extract the appearance features, motion features and semantic features of the target. The appearance features obtain high-resolution target texture information by enhancing the spatial detail preservation ability in the backbone network. The motion features capture the target's dynamic attributes by embedding a lightweight optical flow estimation subnetwork or an inter-frame difference guidance mechanism. The semantic features generate an embedding vector with high-level semantic consistency by using a projection head aligned with a pre-trained vision-language model.

3. The method for accurate multi-target retrieval in security videos according to claim 1, characterized in that, The improved DeepSORT tracker utilizes appearance features to construct an enhanced deep appearance embedding model and dynamically optimizes the target appearance template through an online update strategy. Semantic features are introduced as high-level constraints to perform semantic consistency correction on the matching cost matrix during trajectory association, effectively suppressing ID switching problems caused by occlusion, deformation, or illumination changes.

4. The method for accurate multi-target retrieval in security videos according to claim 2, characterized in that, The appearance features are extracted based on the ResNet-50 deep neural network, including visual information such as the pedestrian's clothing, hairstyle, and body posture, as well as the target's spatial trajectory, temporal trajectory, and behavioral trajectory. The behavioral trajectory is obtained by analyzing the target's speed changes, movement direction, and dwelling patterns.

5. The method for accurate multi-target retrieval in security videos according to claim 4, characterized in that, The spatiotemporal index structure employs a two-dimensional time-space index to achieve rapid target localization and filtering of relevant video segments. In the multi-level retrieval, coarse retrieval uses the spatiotemporal index for rapid filtering, fine retrieval performs precise matching based on feature similarity, and association retrieval utilizes a target association model to identify different manifestations of the same target.

6. A multi-target precise retrieval system in security videos, characterized in that, include: The data acquisition module is used to perform standardized preprocessing on the original security video, including frame rate adjustment and resolution standardization, and video segmentation combined with scene change detection technology. Based on this, a key frame extraction strategy is used to select representative frame sequences. The data acquisition module supports multiple video format inputs and real-time stream processing. The intelligent analysis module is used to deploy an improved YOLOv8 model on preprocessed keyframes for high-precision multi-target detection. The intelligent analysis module includes a target detection unit, a feature extraction unit, and a target tracking unit. The index management module is used to perform an approximate nearest neighbor mapping on the vector using the locality-sensitive hashing algorithm, and to build a compact feature index library that supports fast similarity queries. The index management module builds and maintains feature indexes and spatiotemporal indexes. The retrieval service module is used to perform a three-level progressive retrieval mechanism based on the index structure established in the above steps, including coarse retrieval, fine retrieval and related retrieval. The retrieval service module provides multi-level retrieval algorithms and result ranking. The system management module is used to generate structured and interpretable multi-target retrieval reports, supporting security incident retrospective and intelligent analysis. The system management module provides a user interface and permission management functions.

7. A multi-target precise retrieval system in security video according to claim 6, characterized in that, The target detection unit uses an improved YOLOv8 deep learning model, which can detect various target types, including pedestrians and vehicles.

8. A multi-target precise retrieval system in security video according to claim 7, characterized in that, The feature extraction unit can extract three types of features: appearance features, motion features, and semantic features, and integrate them into a unified target representation vector.

9. A multi-target precise retrieval system in security video according to claim 6, characterized in that, The index management module uses an improved locality-sensitive hashing algorithm to construct feature indexes, which support dynamic updates and maintenance.

10. A multi-target precise retrieval system in security video according to claim 6, characterized in that, The search service module supports multiple search methods, including search based on feature conditions, search based on spatiotemporal conditions, and search based on behavioral conditions.

Images