Image search method and system based on self-supervised visual features

Abstract

This application provides an image search method and system based on self-supervised visual features, relating to the fields of computer vision and information retrieval technology. The method includes: generating a dataset to be entered into the database based on multi-source input data through unified database processing; the multi-source input data includes static images and video data, with the video data undergoing multi-strategy cascade processing to generate a sequence of frames to be entered into the database; extracting a global semantic feature vector set from the dataset to be entered into the database using the DINOv3 self-supervised visual model; constructing and batch storing the feature vectors and corresponding structured index information using a vector database to generate a searchable feature index library; extracting query feature vectors from the query image and generating preliminary search results by cosine similarity matching with the searchable feature index library; and generating final search results based on the preliminary search results through similarity threshold filtering, deduplication of video results, and quantity restrictions.

Description

Technical Field

[0001] This application relates to the fields of computer vision and information retrieval technology, and more particularly to an image search method and system based on self-supervised visual features, which is applicable to scenarios such as video tracing, film and television material management, public opinion monitoring, and image library retrieval. Background Technology

[0002] With the rapid development of computer vision and multimedia retrieval technologies, image search technology has been increasingly widely used in numerous scenarios such as public safety, media content management, public opinion monitoring, film and television production, and smart photo albums. While existing image search technologies have made some progress, they still face several technical bottlenecks. First, traditional systems are often designed independently for static images or videos, lacking a unified processing capability for multi-source heterogeneous data. Static image retrieval systems only support image library retrieval, and video retrieval systems typically only process consecutive frames. Even systems that claim compatibility with both often suffer from retrieval result deviations and management difficulties due to the lack of a unified feature extraction and indexing mechanism. Second, at the visual feature representation level, early systems relied heavily on handcrafted features such as SIFT and HOG, which are extremely sensitive to changes in lighting and scene complexity, making it difficult to obtain stable semantic representations. While deep features based on CNNs have improved representation capabilities to some extent, they heavily rely on large-scale labeled data and lack effective modeling of high-level semantic information such as global image layout and scene structure. This results in poor performance when facing cross-scene and cross-modal retrieval tasks, failing to meet the actual needs of high-precision retrieval.

[0003] For video data processing, existing frame extraction strategies are generally simple and crude, severely impacting retrieval quality. Fixed-interval frame extraction easily generates a large number of redundant frames, increasing storage and computational burden; single keyframe extraction strategies rely excessively on inter-frame difference thresholds, easily missing key content in rapidly changing scenes and generating redundancy in slowly changing scenes; simultaneously, traditional frame extraction methods lack the ability to perceive image content, failing to prioritize the extraction of frames containing key information such as faces and specific targets, resulting in unstable frame extraction results and significantly limiting the coverage and accuracy of video retrieval. Furthermore, in large-scale retrieval scenarios, when the scale of image and video frames reaches hundreds of millions, retrieval architectures based on traditional relational databases or inverted indexes struggle to meet real-time requirements. The lack of efficient vector index structures leads to high retrieval latency and poor system scalability; moreover, the lack of effective control strategies for retrieval results means that the same video may return a large number of similar frames, causing redundant result stacking and severely impacting the user experience. In conclusion, how to achieve high-precision, high-efficiency, and controllable visual content retrieval while being compatible with multi-source data input has become a pressing technical challenge in this field. Summary of the Invention

[0004] The purpose of this application is to provide an image search method and system based on self-supervised visual features, to solve the technical problems in existing technologies such as the inability to uniformly process multi-source data, the simple and crude video frame extraction strategies leading to redundancy or omissions, insufficient feature semantic expression, low efficiency of large-scale vector retrieval, and uncontrollable redundancy of retrieval results. This application generates a dataset to be entered into the database by uniformly storing multi-source input data, extracts global semantic feature vectors using the DINOv3 self-supervised visual model and constructs a searchable feature index library, generates preliminary retrieval results based on cosine similarity matching, and then outputs the final retrieval results after similarity threshold filtering, deduplication of results from the same video, and quantity limitation. This enables unified feature space management and high-precision, high-efficiency retrieval of static image and video data.

[0005] To achieve the above objectives, the image search method based on self-supervised visual features provided in this application specifically includes: generating a set of data to be entered into the database through unified database processing based on multi-source input data; extracting a set of global semantic feature vectors from the set of data to be entered into the database using the DINOv3 self-supervised visual model; indexing and batch storing the set of global semantic feature vectors and corresponding structured index information through a vector database to generate a searchable feature index library; generating a query feature vector based on the query image through global semantic feature extraction; generating preliminary search results by matching the query feature vectors with the searchable feature index library using cosine similarity; and generating final search results by filtering based on similarity thresholds, deduplicating results from the same video, and limiting the number of results based on the preliminary search results.

[0006] In the above-mentioned image search method based on self-supervised visual features, optionally, generating a set of data to be entered into the database through unified database processing based on multi-source input data includes: generating images to be entered into the database from static images in the multi-source input data through standardized preprocessing; generating a sequence of frames to be entered into the database from video data in the multi-source input data through multi-strategy cascade processing; and generating the set of data to be entered into the database by combining the images to be entered into the database and the sequence of frames to be entered into the database.

[0007] In the aforementioned image search method based on self-supervised visual features, optionally, generating a sequence of frames to be added to the database through multi-strategy cascaded processing of video data includes: generating a basic frame sequence by extracting frames at fixed intervals from the input video; generating a supplementary frame sequence by dynamic keyframe extraction and scene switching detection based on the inter-frame gaps of the basic frame sequence; merging the basic frame sequence and the supplementary frame sequence to generate an initial candidate frame set; generating a candidate frame set with database priority based on the initial candidate frame set by object detection and priority marking; and generating the sequence of frames to be added to the database based on the candidate frame set with database priority by feature-level non-maximum deduplication, adaptive size scaling, and structured time index marking.

[0008] In the aforementioned image search method based on self-supervised visual features, optionally, generating a supplementary frame sequence based on the inter-frame gaps of the base frame sequence through dynamic keyframe extraction and scene transition detection includes: generating an effective frame sequence by eliminating invalid transition frames from adjacent frames through grayscale variance verification; generating initial dynamic frames from the effective frame sequence through absolute pixel difference calculation and adaptive dynamic frame difference threshold determination; generating dynamic keyframes from the initial dynamic frames through structural similarity calculation and dynamic trigger threshold verification based on image two-dimensional information entropy; and generating dynamic keyframes from adjacent frames through DINOv3 self-supervised visual model... The system extracts global semantic features and calculates feature cosine similarity. When the cosine similarity is lower than a preset scene switching threshold, it is marked as a scene switching frame. Based on adjacent frames of the ultra-high-speed motion scene, ultra-fast dynamic keyframes are generated through dense optical flow field calculation and global camera movement suppression verification. The supplementary frame sequence is generated by merging the dynamic keyframes, the scene switching frames, and the ultra-fast dynamic keyframes. The invalid transition frames include pure black frames, no signal frames, solid color frames, and / or blurred frames. The adaptive dynamic frame difference threshold is dynamically generated based on historical frame difference statistics and is adjusted for low-light attenuation as the average brightness of the frame increases.

[0009] In the aforementioned image search method based on self-supervised visual features, optionally, the dynamic trigger threshold verification based on the image's two-dimensional information entropy to generate dynamic keyframes includes: calculating the two-dimensional information entropy representing the texture complexity of the current frame based on the gray values ​​of each pixel in the current frame and the average gray values ​​of its neighborhood; generating a safety threshold and an extreme threshold based on the statistical distribution of historical frame entropy values ​​within a sliding window; comparing the two-dimensional information entropy of the current frame with the safety threshold and the extreme threshold, generating a degradation factor through nonlinear mapping, and dynamically adjusting the structural similarity judgment threshold based on the product of the baseline threshold and the degradation factor; when the structural similarity corresponding to the initially screened dynamic frame is lower than the dynamically adjusted judgment threshold, the current frame is marked as a dynamic keyframe.

[0010] In the aforementioned image search method based on self-supervised visual features, optionally, generating high-speed dynamic keyframes based on adjacent frames of a high-speed motion scene through dense optical flow field calculation and global camera movement suppression verification includes: generating motion vectors for each pixel based on dense optical flow field calculation of two adjacent frames; performing global camera movement determination based on the directional distribution of motion vectors for each pixel; stopping the high-speed dynamic keyframe extraction determination based on dense optical flow method when global camera movement is determined; generating local extreme motion amplitudes through extreme value statistics based on the motion amplitudes of all pixels in the frame when the motion amplitudes of all pixels in the frame exceed both the relative background threshold and the absolute motion threshold; marking the current frame as a high-speed dynamic keyframe when the local extreme motion amplitudes exceed both the relative background threshold and the absolute motion threshold.

[0011] In the aforementioned image search method based on self-supervised visual features, optionally, generating a candidate frame set with entry priority based on the initial candidate frame set through target detection and priority labeling includes: generating candidate frames containing key targets by detecting them using a lightweight target detection model based on the initial candidate frame set; generating candidate frames that meet preset conditions by verifying the detection confidence and target proportion of the candidate frames containing key targets; labeling the candidate frames that meet the preset conditions as target-weighted frames and assigning them the highest entry priority; labeling the remaining frames with corresponding priorities according to the source frame extraction strategy; and generating a candidate frame set with entry priority based on the labels; wherein the corresponding priorities, from high to low, are: scene switching frames, dynamic key frames, and fixed interval frames.

[0012] In the above-mentioned image search method based on self-supervised visual features, optionally, generating a deduplicated frame set by feature-level non-maximum deduplication based on the candidate frame set with the input priority includes: calculating the cosine similarity between each pair of candidate frames based on the DINOv3 feature vector of each candidate frame in the candidate frame set with the input priority; dividing visual duplicate frame groups according to the cosine similarity and a preset deduplication threshold; selecting the highest priority single frame from each visual duplicate frame group as the retained frame according to the input priority; and generating the deduplicated frame set based on the combination of all retained frames; wherein the input priority from high to low is target weighted frame, scene switching frame, dynamic key frame, and fixed interval frame.

[0013] In the aforementioned image search method based on self-supervised visual features, optionally, the global semantic feature vector set and corresponding structured index information are indexed and batch-stored through a vector database to generate a searchable feature index library. This includes: marking the data source and data type of each vector in the global semantic feature vector set to generate feature data with type identifiers, where the data types include static images and video frames; configuring a unique video identifier, relative timestamp, and original frame number for the feature vectors corresponding to video frames to generate feature data with a composite structured index; performing L2 normalization on all feature data to generate a standardized feature vector set; constructing an approximate nearest neighbor index based on the standardized feature vector set to generate an indexed feature set; and writing the indexed feature set into the vector database through batch processing to generate the searchable feature index library.

[0014] In the aforementioned image search method based on self-supervised visual features, optionally, generating preliminary search results by matching the query feature vector with the searchable feature index library using cosine similarity includes: inputting the normalized global semantic feature vector of the query into the vector database search interface, performing fast retrieval through an approximate nearest neighbor index, and recalling a preset number of candidate results; calculating the cosine similarity between the query feature vector and the feature vector of each candidate result, and generating the preliminary search results by arranging them in descending order of cosine similarity values.

[0015] In the aforementioned image search method based on self-supervised visual features, optionally, generating the final search results based on the preliminary search results through similarity threshold filtering, deduplication of video results, and quantity limits includes: generating a filtered result set based on the preliminary search results through a preset similarity threshold; generating multiple video group results by grouping them using unique video identifiers based on the structured index information in the filtered result set; sorting each video group result in descending order of cosine similarity, retaining a preset maximum number of frames to generate a simplified single-video result; and arranging all simplified single-video results in descending order of global similarity to generate the final search result list.

[0016] In the above-mentioned image search method based on self-supervised visual features, optionally, when the query input is a video frame, the method includes: filtering invalid matching results by using a source nearest neighbor time window based on the timestamp information of the simplified single video result and the query image, wherein the source nearest neighbor time window filtering is used to remove matching results that belong to the same video as the query image and whose timestamp difference is less than a preset time window; and arranging the filtered simplified single video results in descending order of global similarity to generate a final search result list.

[0017] This application also provides an image search system based on self-supervised visual features. The system includes: a multi-source data entry module, a feature extraction and storage module, and a retrieval and matching module. The multi-source data entry module is used to generate a set of data to be entered into the database based on multi-source input data through unified entry processing. The feature extraction and storage module is used to extract and generate a set of global semantic feature vectors based on the set of data to be entered into the database using the DINOv3 self-supervised visual model. The module also uses a vector database to index and batch store the set of global semantic feature vectors and corresponding structured index information to generate a searchable feature index library. The retrieval and matching module is used to generate a query feature vector based on the query image through global semantic feature extraction, generate preliminary retrieval results based on the query feature vectors and the searchable feature index library through cosine similarity matching, and generate final retrieval results based on the preliminary retrieval results through similarity threshold filtering, deduplication of video results, and quantity limitation.

[0018] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0019] This application also provides a computer-readable storage medium storing a computer program that performs the above-described methods.

[0020] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0021] The beneficial technical effects of this application are as follows: By processing multi-source input data in a unified database, static images and video data are represented and managed uniformly in the same feature space, significantly simplifying the system architecture and reducing maintenance costs; by using the DINOv3 self-supervised visual model to extract global semantic feature vectors, highly robust semantic representations across scenes and perspectives can be learned without relying on labeled data, overcoming the problem of insufficient feature expression capabilities of traditional CNNs and significantly improving retrieval accuracy and generalization ability; through multi-strategy cascaded processing of fixed-interval frame extraction, dynamic keyframe extraction and scene switching detection, target weighted sorting, and feature-level non-maximum deduplication, the system ensures the accuracy of video keyframes. The comprehensive content coverage significantly reduces redundant frames, effectively lowering storage and computational burdens. A searchable feature index library is built based on a vector database, combining near-nearest neighbor indexing and cosine similarity matching to achieve millisecond-level efficient retrieval across a large-scale feature library. Simultaneously, similarity threshold filtering, deduplication of video results, and quantity limits effectively prevent redundant stacking of search results, improving result compactness and business usability. The entire solution is highly modular and scalable, supporting incremental data import and dynamic index updates. It can be widely applied to various scenarios such as video tracing, film and television material management, public opinion monitoring, and image library retrieval, demonstrating significant practical value and promising prospects for wider adoption. Attached Figure Description

[0022] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. In the drawings: Figure 1 This is a schematic diagram of the overall process of an image search method based on self-supervised visual features provided in an embodiment of this application; Figure 2 This is a schematic diagram of a unified data entry process for multiple sources provided in an embodiment of this application; Figure 3 This is a schematic diagram of the process for generating a sequence of frames to be stored in the database through multi-strategy cascading processing of video data, as provided in an embodiment of this application. Figure 4This is a schematic diagram of the process for dynamic keyframe extraction and scene switching detection to generate supplementary frame sequences according to an embodiment of this application; Figure 5 This is a schematic diagram of the process for generating dynamic keyframes based on dynamic trigger threshold verification of image two-dimensional information entropy according to an embodiment of this application. Figure 6 This is a schematic diagram of the process for generating ultra-fast dynamic keyframes based on dense optical flow field and global camera movement suppression verification, provided in an embodiment of this application. Figure 7 This is a schematic diagram illustrating the process of generating a candidate frame set with priority in the database for target detection and priority marking according to an embodiment of this application; Figure 8 This is a schematic diagram of the process for generating a deduplicated frame set by deduplicating feature-level non-maximum values ​​according to an embodiment of this application. Figure 9 A schematic diagram illustrating the process of constructing and batch storing a global semantic feature vector index according to an embodiment of this application; Figure 10 This is a schematic diagram illustrating the process of generating preliminary search results through query feature vector retrieval matching according to an embodiment of this application. Figure 11 This is a schematic diagram illustrating the process of generating final search results by similarity threshold filtering, deduplication of same video results, and quantity limitation, as provided in an embodiment of this application. Figure 12 This is a schematic diagram of the architecture of an image search system based on self-supervised visual features provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0024] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0025] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0026] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed description. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0027] To facilitate understanding of the technical solutions of this application, several terms used in the following embodiments are explained uniformly. In this application, "image to be imported" and "frame to be imported" are collectively referred to as "data to be imported," both of which are components of the data to be imported, the only difference being that the former originates from static images and the latter from video data. "Candidate frame set" and "initial candidate frame set" have the same meaning, both referring to the set of frames to be processed obtained by merging the basic frame sequence and the supplementary frame sequence. "Structured index information" refers to the index data associated with the feature vector used for retrieval and positioning; "composite structured index" is a specific form of structured index information, specifically referring to a combined index structure containing a unique video identifier, a relative timestamp, and the original frame number. "Target weighted frame" and "high priority frame" have the same meaning, both referring to candidate frames that meet preset conditions after target detection and verification and are assigned the highest import priority. "Deduplicated frame set" has the same meaning as the candidate frame set after deduplication. "Query image" generally refers to the visual input submitted by the user for retrieval, which can be either a static image or a video frame. "Global camera movement state" and "camera global motion state" have the same meaning, both referring to the overall image displacement state caused by the camera's own movement (rather than the movement of objects within the frame). "Stop the rapid dynamic keyframe extraction judgment based on dense optical flow" and "Freeze the frame extraction instruction based on optical flow" have the same meaning, both referring to pausing the processing logic of triggering rapid dynamic keyframe marking using optical flow in the global camera movement state.

[0028] Please refer to Figure 1 As shown in one embodiment of this application, the image search method based on self-supervised visual features specifically includes: S101 generates a dataset to be imported into the database based on multi-source input data through unified data import processing; the multi-source input data includes static images and video data. For static images, images to be imported into the database are generated through standardized preprocessing (including size scaling, pixel value normalization, etc.); for video data, a sequence of frames to be imported into the database is generated through multi-strategy cascaded processing (including fixed-interval frame extraction, dynamic keyframe and scene transition detection, target weighted sorting, and feature-level non-maximum deduplication, etc.).

[0029] S102 uses the DINOv3 self-supervised visual model to extract and generate a set of global semantic feature vectors based on the dataset to be imported. DINOv3 is a self-supervised visual feature extraction model based on the VisionTransformer architecture, which can learn a global semantic representation with high robustness across scenes and viewpoints without relying on labeled data. For each frame of image (including static images and video frames) in the dataset to be imported, it is uniformly scaled to the input size adapted to the DINOv3 model (e.g., 224×224 pixels), and after pixel value normalization, it is input into the pre-trained DINOv3 visual Transformer encoder, which outputs a fixed-dimensional global semantic feature vector. Static images and video frames share the same feature space, thus achieving a unified representation of multi-source data.

[0030] S103 uses a vector database to construct and batch store the global semantic feature vector set and corresponding structured index information to generate a searchable feature index library. For each feature vector, a globally unique data ID is configured, and the data type (static image or video frame) is marked. For feature vectors from which video frames originate, a composite structured index containing a unique video identifier, a relative timestamp, and the original frame number is also added. After all feature vectors are L2 normalized, an approximate nearest neighbor index (such as HNSW or IVF-PQ type) is constructed in a vector database (such as Milvus or FAISS) and written in batch mode to generate a searchable feature index library.

[0031] S104 generates a query feature vector based on the query image through global semantic feature extraction. The query feature vector is then matched with the searchable feature index using cosine similarity to generate preliminary search results. After the user uploads the query image, standardized preprocessing (size scaling, pixel value normalization) consistent with the database entry stage is performed first. Then, the global semantic feature vector of the query image is extracted using the same DINOv3 model and L2 normalized. The normalized query feature vector is input into the retrieval interface of the vector database. A fast retrieval is performed based on a pre-built approximate nearest neighbor index, recalling a preset number of candidate results. The cosine similarity between the query feature vector and the feature vector of each candidate result is calculated, and the preliminary search results are generated by sorting them in descending order of cosine similarity value.

[0032] S105 generates the final search results based on the preliminary search results through similarity threshold filtering, deduplication of video results, and quantity limits. First, the preliminary search results are filtered using a preset similarity threshold (e.g., 0.7 to 0.85), retaining matches with a cosine similarity greater than or equal to the threshold and discarding low-quality matches. Then, the filtered results are grouped according to the unique video identifier in the structured index. Within each video group, only frames with the highest similarity (e.g., 3 to 8 frames) are retained, achieving deduplication and quantity limits for video results. Finally, all retained grouped results are sorted in descending order of global similarity, and the final search result list is output, including the matched image or video frames, the original video file path (if it is a video), and timestamp or frame number information, achieving content-level positioning and visual result display.

[0033] Please refer to Figure 2 As shown in one embodiment of this application, generating a set of data to be imported into the database through unified import processing based on multi-source input data includes: S201 generates images to be added to the database by standardizing the static images in the multi-source input data. For the input static images, they are first scaled to the input size adapted to the DINOv3 model (e.g., 224×224 pixels) through bilinear interpolation. Then, the pixel values ​​are standardized (e.g., the pixel values ​​are normalized to the [0,1] interval or standardized according to the mean and standard deviation of the ImageNet dataset) to generate images to be added to the database that meet the model input requirements.

[0034] S202 generates a sequence of frames to be added to the database based on video data from multiple input sources through multi-strategy cascaded processing. For the input video data, a collaborative logic of "global coarse screening—local fine extraction—semantic weighting—feature deduplication" is used for processing. Specifically, firstly, a video timeline skeleton is constructed using fixed-interval frame extraction as the main strategy. Then, within the fixed intervals, high-dynamic-range changes and shot transitions are captured through dynamic keyframe extraction and scene switching detection. Next, candidate frames are semantically weighted and their inclusion priority is set through object detection. Finally, redundancy is eliminated through feature-level non-maximum deduplication, and the sequence of frames to be added to the database is generated after adaptive scaling and structured time indexing. The detailed steps of this multi-strategy cascaded processing will be further explained in subsequent embodiments.

[0035] S203 generates the data set to be imported by combining the image to be imported with the frame sequence to be imported. The image to be imported generated in S201 and the frame sequence to be imported generated in S202 are merged to form a unified data set to be imported for use in subsequent feature extraction steps.

[0036] Please refer to Figure 3As shown in one embodiment of this application, generating a sequence of frames to be added to the database based on video data through multi-strategy cascading processing includes: S301 generates a basic frame sequence by extracting frames from the input video at fixed intervals. As the backbone strategy, the system extracts the basic frame sequence from the input video at a preset fixed frequency (e.g., 1 frame per second, or 1 FPS). This strategy ensures the overall uniformity of the video's temporal dimension, constructs the video's temporal skeleton, and avoids the loss of time segments due to algorithmic omissions.

[0037] S302 generates a supplementary frame sequence based on the inter-frame gaps of the basic frame sequence through dynamic keyframe extraction and scene transition detection. The basic frame sequence and the supplementary frame sequence are then merged to generate an initial candidate frame set. During the fixed-interval frame extraction intervals, the system simultaneously initiates dynamic keyframe extraction and scene transition detection algorithms as a secondary supplementary strategy to capture moments of high dynamic changes and camera transitions. Dynamic keyframe extraction is based on motion and structural changes: First, OpenCV is used to calculate the image grayscale variance of adjacent frames, quickly eliminating invalid transition frames (such as pure black frames, no-signal frames, solid-color frames, or extremely blurry frames) with grayscale variances below a very small threshold, avoiding invalid features from being included in the database. Then, the absolute pixel difference between adjacent valid frames is calculated. If the average pixel difference exceeds the adaptive dynamic frame difference threshold, the structural similarity (SSIM) between the two frames is further calculated. When the SSIM is lower than the dynamic trigger threshold based on the image's two-dimensional information entropy, it is marked as a dynamic keyframe. Scene transition detection reuses DINOv3's feature extraction capabilities: it extracts DINOv3 feature vectors from adjacent frames at the thumbnail scale, calculates cosine similarity, and when the cosine similarity is lower than a preset scene transition threshold (e.g., 0.3 to 0.6), it is determined to be a scene transition boundary and marked as a scene transition frame. The dynamic keyframes, scene transition frames, and base frame sequences are merged to generate an initial candidate frame set.

[0038] S303 generates a candidate frame set with entry priority based on the initial candidate frame set through object detection and priority marking; for each candidate frame in the initial candidate frame set, object detection is performed using a lightweight object detection model (such as YOLOv8). If the frame contains a preset key object (such as a face, a specific vehicle model, a specific product entity, etc.), and the detection confidence is greater than a preset confidence threshold (such as 0.5 to 0.7), and the area of ​​the target bounding box occupies a proportion of the entire image greater than a preset area proportion threshold (dynamically set according to the target type, for example, 3% to 8% of the entire image area for a face, and 10% to 20% for a vehicle), then the frame is marked as a target-weighted frame and given the highest entry priority; the remaining frames are marked with corresponding priorities according to their source frame extraction strategy (scene switching frames are second, dynamic key frames are third, and fixed interval frames are the lowest), generating a candidate frame set with entry priority.

[0039] S304 generates the sequence of frames to be added to the database based on the candidate frame set with the priority for inclusion in the database through feature-level non-maximum deduplication, adaptive size scaling, and structured time indexing. To prevent duplicate redundancy of the same visual image caused by parallel extraction of multiple strategies, the system performs feature-level non-maximum suppression before final inclusion in the database: comparing the cosine similarity between the DINOv3 feature vectors of all frames in the candidate frame buffer pool, if the cosine similarity of multiple consecutive frames is greater than a preset deduplication threshold (e.g., 0.92 to 0.97), it is determined to be a visual duplicate frame, and the single frame with the highest priority is retained according to the inclusion priority (priority order: target weighted frame > scene switching frame > dynamic key frame > fixed interval frame), and the remaining duplicate frames are deleted. After deduplication, the retained frames are uniformly scaled to the input size adapted to the DINOv3 model (e.g., 224×224 pixels) through bilinear interpolation, and a composite structured index containing a globally unique video ID, relative timestamp, and original frame number is generated for each frame. This index is serialized together with the feature vector and written to the vector database to generate the sequence of frames to be added to the database.

[0040] It should be noted that the above frame extraction strategies do not operate in isolation, but rather are integrated throughout the entire processing flow through a collaborative logic of "global coarse screening - local fine extraction - semantic weighting - feature deduplication". Fixed-interval frame extraction serves as the global coarse screening strategy to construct the timeline skeleton, dynamic keyframe extraction and scene switching detection serve as the local fine extraction strategy to capture highly dynamic changes and camera transitions, target weighted sorting serves as the semantic weighting strategy to improve the priority of key target frames, and feature-level non-maximum deduplication serves as the final redundancy elimination strategy to ensure the compactness of the frames entering the database.

[0041] Please refer to Figure 4 As shown, in one embodiment of this application, generating a supplementary frame sequence based on the inter-frame gap of the base frame sequence through dynamic keyframe extraction and scene switching detection includes: S401 generates a valid frame sequence by eliminating invalid transition frames based on adjacent frames through grayscale variance verification, and generates an initial dynamic frame based on the valid frame sequence by calculating the absolute pixel difference and determining the adaptive dynamic frame difference threshold. First, OpenCV is used to calculate the grayscale variance of adjacent frames, quickly removing invalid transition frames (including pure black frames, no signal frames, solid color frames, and / or extremely blurry frames) with grayscale variances below a very small threshold, thus obtaining a valid frame sequence. Then, the absolute pixel difference between adjacent valid frames is calculated; if the average pixel difference exceeds a dynamic threshold... Then, the structural similarity (SSIM) between the two frames is further calculated. When When a significant content mutation occurs, the system identifies it as a dynamic keyframe. For extremely high-speed motion scenes, the system uses optical flow to calculate the motion vector field and extracts dynamic keyframes at high speed. To adapt to different lighting conditions and video quality, the [specific features / features] in this application [are described]. Instead of a fixed constant, it employs an adaptive dynamic calculation model based on a context-sliding window. In practice, the system maintains a historical frame difference mean buffer of length N (e.g., N=30 frames). The calculation formula is as follows: ; in, This represents the average inter-frame difference over the past N frames. Its standard deviation, This is the sensitivity adjustment factor (values ​​range from 1.2 to 2.0). Additionally, for low-light (nighttime) video, the system will automatically adjust the sensitivity based on the average brightness of the entire frame. Apply a decay penalty of 0.5 to 0.8 to ensure that subtle dynamic changes can still be captured sensitively in low-light environments.

[0042] S402 generates dynamic keyframes based on the structural similarity calculation of the initial screening dynamic frames, combined with dynamic trigger threshold verification based on image two-dimensional information entropy. For the initial dynamic frames generated by S401, the structural similarity (SSIM) between adjacent frames is further calculated. SSIM calculation uses a Gaussian weighted sliding window instead of a traditional mean window to eliminate block artifacts and align with the human visual system's greater sensitivity to the central region of the field of vision. Specifically, SSIM is an index that measures the similarity between two images. Compared to pure pixel difference (frame difference method), it better reflects the human visual system's perception of structural information. Its calculation integrates three dimensions: brightness, contrast, and structure. Given local windows in two adjacent frames... and The formula for calculating SSIM is:

[0043] in, , It is a local mean. , For local variance, For covariance. , To maintain a stable constant in the calculation. In real-world video frame extraction scenarios, we often encounter completely black fading frames at the beginning and end of the video or pure gray frames with no signal in the monitoring screen due to power outages. In these situations, the average value of local pixels... and variance All of these values ​​may approach 0. This application introduces an adaptive stabilization term based on the 8-bit color depth dynamic range in the SSIM similarity calculation module. and This not only avoids system crashes caused by division by zero anomalies, but also effectively suppresses misjudgments caused by noise being amplified into false high-contrast structures in low-light environments, ensuring the high availability of the global coarse screening and local fine extraction algorithms under extreme image quality conditions. The SSIM value of the entire image is the mean of all local windows (MSSIM), with a value range from -1 to 1. The closer to 1, the more similar the images are.

[0044] In calculating structural similarity, this application uses a Gaussian weighted sliding window instead of a traditional mean window, which not only effectively eliminates block artifacts but also better aligns with the human visual system's greater sensitivity to the central region of the field of vision. SSIM employs a weighted sliding window (typically Gaussian weighted).

[0045] Set a size as A sliding window. Within this window, the system presets a two-dimensional Gaussian kernel function with a standard deviation that is always set to... This Gaussian kernel generates a set of normalized spatial weights. (where N=121), and satisfies Pixels closer to the center of the window have a higher weight; pixels further out have a lower weight. Let the image patch within the window be... (Partial view of the previous frame) and (Partial view of the next frame), the pixel value sequences are as follows: and .

[0046] Brightness is calculated using a locally weighted average: ; ; Contrast ratio is calculated using local weighted variance: ; ; The structure is calculated using locally weighted covariance: ; threshold The typical empirical value for similarity tolerance threshold is set between 0.75 and 0.85. However, in complex high-frequency video scenes (such as flickering water ripples, violently swaying leaves, and other scenes accompanied by high ambient noise), minute non-semantic physical movements can cause uncontrollable sudden drops in SSIM between consecutive frames. If a static high threshold is maintained, such background noise is easily misjudged as critical dynamic changes, leading to frame overload; conversely, if the global setting is too low, critical actions may be missed in smooth scenes. To address these issues, this application introduces a similarity tolerance threshold downgrade compensation model based on image two-dimensional information entropy. The system does not raise the judgment threshold, but rather actively lowers the similarity lower limit in high-noise scenes, giving the algorithm higher noise tolerance. Details will be provided in subsequent embodiments.

[0047] S403 extracts global semantic features and calculates feature cosine similarity based on adjacent frames using the DINOv3 self-supervised visual model. When the cosine similarity is lower than a preset scene switching threshold, it is marked as a scene switching frame. Unlike traditional color histogram methods, this step deeply reuses the pre-existing feature extraction capabilities—the system extracts DINOv3 feature vectors from adjacent frames at the thumbnail scale and calculates cosine similarity. When the cosine similarity is lower than a preset scene transition threshold (ranging from 0.3 to 0.6), it indicates a break in the depth visual semantics, which is determined as a scene transition boundary, and the current frame is marked as a scene transition frame. This method effectively overcomes the misjudgment caused by gradual changes in illumination in traditional algorithms and accurately locates scene transition frames.

[0048] S404 generates ultra-fast dynamic keyframes based on adjacent frames of ultra-fast motion scenes through dense optical flow field calculation and global camera movement suppression verification. For extremely high-speed motion scenarios (such as sports events and high-speed vehicle travel), traditional frame difference methods and SSIM algorithms are prone to failure due to excessive target displacement. To address this, this application introduces a dense optical flow method to construct a pixel-level dense motion vector field between adjacent frames.

[0049] The system first calculates the global motion vector magnitude of the entire map. The calculation formula is: ; in, , These are the physical pixel width and height of the input video frame, respectively. This is the spatial coordinate index of the pixel point in the two-dimensional image plane of the video frame. The coordinates are calculated by the optical flow algorithm. The subpixel-level physical displacement vector of a pixel in the horizontal direction (X-axis). This is the sub-pixel level physical displacement vector of the pixel in the vertical direction (Y-axis). Represents a single pixel The true absolute motion distance (Euclidean norm) between the current two adjacent frames.

[0050] Simply relying on the global amplitude mean It is highly susceptible to severe interference from global camera movements (such as panning shots and handheld device shaking). Therefore, this application designs the triggering condition for rapid frame dropping as a dual-track separation verification mechanism consisting of global background motion suppression and local abnormal high-energy triggering: 1. Background motion suppression (suppressing camera movement): The system calculates the orientation histogram of the optical flow field. A global orientation consistency ratio parameter is set. (The system typically uses a value of 70% to 80%). When it exceeds... The proportion of pixels and their motion vectors When pointing to the same angular range, the system determines that the camera is in a global movement state. At this time, the system will forcibly freeze the frame extraction command based on optical flow to avoid generating a large number of redundant garbage frames.

[0051] 2. Localized abnormal high kinetic energy triggering (capturing the extreme transient state of a real target): After excluding global motion, the system extracts the top 10% of pixels with the largest motion amplitude values ​​from all pixels in the entire image, and calculates the arithmetic mean of this set, which is defined as the local extreme motion amplitude. This parameter accurately represents the kinetic energy of the most rapidly moving foreground target in the image.

[0052] 3. Final threshold determination criteria: The final joint logic expression for the system to trigger ultra-fast dynamic keyframe extraction is: ; in: (Absolute Motion Threshold): Used to filter out regular low-speed motion and determine whether a target has reached the standard for extremely fast crossing. To ensure scale invariance across resolutions, this value is not a fixed constant, but is dynamically calculated based on the image diagonal length and the video frame rate (FPS). ,in This is a relative displacement scaling factor (typically taken as 0.03). If the video is at 60 FPS, the time interval between adjacent frames is extremely short, and the pixel displacement caused by physical motion is relatively small. It should be automatically adjusted downwards; if it's 15 FPS, the time interval is long and the displacement span is large. It should be automatically adjusted upwards. The relative background suppression coefficient (RSC) measures the prominence of foreground motion relative to the global background environment; this system parameter is typically between 3.0 and 5.0. Only when the most intense local motion is detected... The amplitude exceeds the global average fluctuation. of and greater than the absolute motion threshold Only then is it considered a valid keyframe.

[0053] S405 generates the supplementary frame sequence by merging the dynamic keyframe, the scene switching frame, and the ultra-fast dynamic keyframe; the dynamic keyframe generated in S402, the scene switching frame generated in S403, and the ultra-fast dynamic keyframe generated in S404 are merged to form the supplementary frame sequence, which is then merged with the base frame sequence to generate the initial candidate frame set.

[0054] The invalid transition frames include pure black frames, no signal frames, solid color frames, and / or blurred frames; the adaptive dynamic frame difference threshold is dynamically generated based on historical frame difference statistics and is adjusted for low-light attenuation according to the average brightness of the frame.

[0055] Please refer to Figure 5 As shown, in one embodiment of this application, generating dynamic keyframes based on dynamic trigger threshold verification of image two-dimensional information entropy includes: S501 calculates the two-dimensional information entropy representing the texture complexity of the current frame based on the gray value of each pixel and the average gray value of its neighborhood. For each pixel in the current frame, obtain its own grayscale value. At the same time, calculate its Mean gray value of all pixels in the neighborhood This forms a feature pair. Statistically analyze each feature pair in the entire image. Frequency of occurrence Let the total number of pixels in the image be... Then the joint probability density of this state is: ; Calculating the entropy of two-dimensional information: ; The two-dimensional information entropy is used to characterize the texture complexity of the current frame—the higher the entropy value, the richer the texture and the more complex the details; the lower the entropy value, the smoother the image and the more it tends to be a solid color or simple texture.

[0056] S502 generates safety and extreme value thresholds based on the statistical distribution of historical frame entropy values ​​within the sliding window. To adapt to the inherent texture differences of different video sources (such as minimalist animation and high-noise surveillance), the system abandons hard-coded constants and uses statistical methods to extrapolate the "normal entropy baseline" of the current video in real time. The system calculates the mean of the entropy values ​​of all historical frames within the sliding window. with standard deviation Dynamically generate the current time node. safety threshold With extreme value threshold : ; ; in, , The sensitivity coefficient is set to [value]. , , To ensure absolute safety, a lower threshold is set to prevent the algorithm from becoming overly sensitive due to excessively low window mean in minimalist scenarios such as pure black. The value is adaptively mapped and configured according to the business scenario: for digitally synthesized videos containing a large number of large areas of solid color blocks (such as animations and courseware screen recordings), the lower limit threshold is 2.0~2.5; for naturally shot videos and security monitoring, the lower limit threshold is 3.5~4.5; for low-light environments, it can be dynamically compensated to 5.0~6.0.

[0057] S503 compares the two-dimensional information entropy of the current frame with the security threshold and the extreme value threshold, generates a degradation factor through nonlinear mapping, and dynamically adjusts the structural similarity determination threshold according to the product of the benchmark threshold and the degradation factor. Specifically, the information entropy of the current frame can be... Substitute into the nonlinear mapping function ; The final dynamic trigger threshold is calculated as follows: ; in, The baseline threshold is 0.85. The maximum downgrade penalty weight (e.g., 0.05).

[0058] S504 When the structural similarity corresponding to the initial screening dynamic frame is lower than the dynamically adjusted judgment threshold, the current frame is marked as a dynamic keyframe. Specifically, when the structural similarity SSIM between adjacent frames corresponding to the initial screening dynamic frame generated in S401 is lower than the adaptive triggering threshold T_ssim(t) calculated in S503, the frame is marked as a dynamic keyframe. This dynamic threshold mechanism enables the system to actively lower the similarity lower limit in complex high-frequency video scenes (such as flickering water ripples, violently shaking leaves, etc., accompanied by high environmental noise), avoiding misjudging background noise as key dynamic mutations; at the same time, it maintains a higher threshold in smooth scenes to avoid missing key actions.

[0059] Please refer to Figure 6 As shown, in one embodiment of this application, generating ultra-fast dynamic keyframes based on adjacent frames of an ultra-fast motion scene through dense optical flow field calculation and global camera movement suppression verification includes: S601 generates motion vectors for each pixel by calculating a dense optical flow field based on two adjacent frames. The system uses a dense optical flow algorithm (such as the Farnebäck algorithm or a dense version of the Lucas-Kanade algorithm) to construct a pixel-level dense motion vector field between two adjacent frames, and obtains the displacement vector u(x,y) in the horizontal direction and the displacement vector v(x,y) in the vertical direction for each pixel.

[0060] The S602 performs global camera movement determination based on the directional distribution of motion vectors of each pixel. When a global camera movement is determined, the rapid dynamic keyframe extraction determination based on dense optical flow is stopped. The system calculates the orientation histogram of the optical flow field and statistically analyzes the angular distribution of motion vectors of each pixel. A global orientation consistency ratio parameter P_dir is set (typically set to 70% to 80% in the system). When the motion vectors of more than a certain percentage of pixels point to the same angular range, the system determines that the camera is currently in a global movement state (such as panning, zooming, handheld device shaking, etc.). At this time, the system will forcibly freeze the rapid dynamic keyframe triggering command based on optical flow to avoid generating a large number of redundant garbage frames.

[0061] In a non-global motion state, S603 generates local extreme motion amplitude values ​​by extreme value statistics based on the motion amplitude values ​​of all pixels in the frame. After excluding global motion, the system extracts the top 10% of pixels with the largest motion amplitude values ​​from all pixels in the entire image, calculates the arithmetic mean of this set, and defines it as the local extreme motion amplitude V_local. This parameter accurately represents the kinetic energy of the most rapidly moving foreground target in the image, effectively eliminating interference from low-speed motion in the background area.

[0062] S604 When the local extreme value motion amplitude simultaneously exceeds both the relative background threshold and the absolute motion threshold, the current frame is marked as an ultra-fast dynamic keyframe. The specific implementation logic has been described in the aforementioned S404, and will not be detailed here.

[0063] Please refer to Figure 7 As shown, in one embodiment of this application, generating a candidate frame set with library priority based on the initial candidate frame set through target detection and priority marking includes: S701 generates candidate frames containing key targets based on the initial candidate frame set using a lightweight object detection model; for each frame in the initial candidate frame set, object detection is performed using a lightweight object detection model (such as YOLOv8). Key targets are dynamically defined by the system configuration dictionary and may include business-related target categories such as faces, specific vehicle models, and specific product entities.

[0064] S702 generates candidate frames that meet preset conditions based on the candidate frames containing key targets by detecting confidence and target proportion verification; For candidate frames that detect key targets, a double check is performed: whether the confidence level is greater than a preset confidence threshold. (e.g., 0.6); the proportion of the target bounding box area to the total map area. (If extremely small background targets are filtered out), then the frame is given the highest priority for inclusion in the database. The system sets dynamic gradient thresholds; for example, for faces, The threshold is set to 0.05 (5% of the total map area); for vehicles, it is set to 0.15. If a target is detected but its area is smaller than this threshold, it will not be given the highest priority for database entry and will be processed as a regular dynamic frame. The system will attach a metadata tag for the target category for joint filtering during subsequent vector retrieval. Only when both the confidence level and area ratio meet the preset conditions will the frame be considered a candidate frame that meets the preset conditions.

[0065] S703 marks candidate frames that meet the preset conditions as target weighted frames and assigns them the highest priority for database entry. The remaining frames are marked with their corresponding priorities according to the source frame extraction strategy. Candidate frames that meet the dual verification conditions in S702 are marked as target weighted frames and assigned the highest priority for database entry. The remaining frames are marked with their corresponding priorities according to their source frame extraction strategy, and the corresponding priorities from high to low are: scene switching frames, dynamic keyframes, and fixed interval frames.

[0066] S704 generates a set of candidate frames with priority in the library based on the markings; combines all candidate frames marked with priority in S703 to generate a set of candidate frames with priority in the library for use in the subsequent feature-level non-maximum deduplication step.

[0067] The corresponding priorities, from highest to lowest, are: scene switching frames, dynamic keyframes, and fixed interval frames.

[0068] Please refer to Figure 8 As shown, in one embodiment of this application, generating a deduplicated frame set by deduplicating candidate frames based on the priority set of the input library through feature-level non-maximum deduplication includes: S801 calculates the cosine similarity between each candidate frame based on the DINOv3 feature vector of each candidate frame in the candidate frame set with the input priority; for each frame in the candidate frame set, its DINOv3 feature vector has been extracted in step S102. The system calculates the cosine similarity between candidate frames to measure the degree of similarity of visual content.

[0069] S802 divides visually repeating frame groups according to the cosine similarity and a preset deduplication threshold; when the cosine similarity of multiple consecutive frames is greater than the preset deduplication threshold (ranging from 0.92 to 0.97), these consecutive frames are determined to be visually repeating frames and are divided into the same visually repeating frame group. It should be noted that "consecutive frames" in this application refer to a sequence of frames that are adjacent or close to each other on the video timeline. Since the candidate frames generated by multi-strategy frame extraction may not be strictly continuous on the timeline (for example, there may be several original frames between fixed interval frames and dynamic keyframes), "continuous" here refers to adjacent frames after being sorted by timestamp in the candidate frame buffer pool.

[0070] S803 selects the highest-priority single frame from each visual duplicate frame group as the retained frame according to the database entry priority. For each visual duplicate frame group, the highest-priority single frame is selected as the retained frame according to the database entry priority (from high to low: target-weighted frame > scene transition frame > dynamic keyframe > fixed interval frame), and other duplicate frames in the group are deleted. If there are multiple frames with the same priority in the same group, the frame with the highest mean cosine similarity between the DINOv3 feature and other frames in the group (i.e., the most "central" representative frame) is further selected as the retained frame.

[0071] S804 generates the deduplicated frame set based on all retained frame combinations; merges the single frames retained in all visually duplicated frame groups to form the deduplicated frame set, effectively eliminating visual redundancy caused by multi-strategy parallel frame extraction.

[0072] The data entry priorities, from highest to lowest, are: target weighted frames, scene switching frames, dynamic keyframes, and fixed interval frames.

[0073] Please refer to Figure 9 As shown, in one embodiment of this application, the global semantic feature vector set and corresponding structured index information are indexed and batch-stored through a vector database to generate a searchable feature index library, including: S901 generates feature data with type identifiers based on the data source of each vector in the global semantic feature vector set, whereby the data types include static images and video frames. For each piece of data in the data set to be entered into the database, the corresponding data type identifier is marked according to its source (static image or video frame) so as to distinguish between image sources and video sources in the search results.

[0074] S902 configures the feature vector corresponding to the video frame with a unique video identifier, relative timestamp, and original frame number to generate feature data with a composite structured index; for feature vectors derived from video frames, it configures a composite structured index containing a globally unique video ID (VideoID), relative timestamp (Timestamp), and original frame number (FrameIndex). This composite structured index enables the retrieval results to accurately locate the specific time and frame position in the video, achieving content-level positioning.

[0075] S903 performs L2 normalization on all feature data to generate a standardized set of feature vectors; it also performs L2 normalization on all global semantic feature vectors to ensure that each feature vector has the same magnitude, thereby improving the stability of cosine similarity calculation and retrieval accuracy.

[0076] S904 constructs an approximate nearest neighbor index based on the standardized feature vector set to generate an indexed feature set; in a vector database (such as Milvus or FAISS), an approximate nearest neighbor (ANN) index structure is constructed based on the standardized feature vector set, including but not limited to HNSW (hierarchical navigable small-world graph) index structure and / or IVF-PQ (inverted file and product quantization) index structure, to support efficient retrieval under large-scale vectors.

[0077] S905 writes the indexed feature set into the vector database via batch processing, generating the searchable feature index library. The indexed feature vector set is then written into the vector database in batch processing to generate the searchable feature index library. This index library supports incremental data import and dynamic index updates, adapting to large-scale data expansion needs.

[0078] Please refer to Figure 10 As shown, in one embodiment of this application, generating preliminary search results based on the query feature vector and the searchable feature index library through cosine similarity matching includes: S1001 inputs the normalized global semantic feature vector of the query into the vector database retrieval interface, performs fast retrieval using an approximate nearest neighbor index, and retrieves a preset number of candidate results. After the user uploads the query image, the system performs standardized preprocessing consistent with the database entry stage, extracts the global semantic feature vector of the query image using the DINOv3 model, and performs L2 normalization. The normalized query feature vector is then input into the vector database retrieval interface, and fast approximate nearest neighbor retrieval is performed based on a pre-built approximate nearest neighbor index (HNSW or IVF-PQ), retrieving a preset number (e.g., Top-1000) of candidate results.

[0079] S1002 calculates the cosine similarity between the query feature vector and the feature vector of each candidate result, and sorts them in descending order of cosine similarity value to generate the preliminary search results. For each recalled candidate result, the cosine similarity between the query feature vector and the feature vector of that candidate result is calculated, and all candidate results are sorted in descending order of cosine similarity value to generate preliminary search results.

[0080] Please refer to Figure 11 As shown, in one embodiment of this application, generating the final search results based on the preliminary search results through similarity threshold filtering, deduplication of same video results, and quantity restrictions includes: S1101 generates a filtered result set based on the preliminary search results using a preset similarity threshold. Then, based on the structured index information in the filtered result set, multiple video grouping results are generated by grouping by video unique identifiers. First, the preliminary search results are filtered using a preset similarity threshold (ranging from 0.7 to 0.85), retaining only matching results with a cosine similarity greater than or equal to the threshold, eliminating low-quality matches to improve result accuracy. Next, based on the structured index information of each matching result in the filtered result set, grouping is performed by video unique identifier (VideoID)—all static images form their own category (without VideoID), and matching results from the same video source are grouped into the same video group, generating multiple video grouping results.

[0081] S1102 sorts the results of each video group in descending order of cosine similarity, retaining a preset maximum number of frames to generate a simplified single-video result; for each video group, it sorts the matching results within it in descending order of cosine similarity, retaining only a preset maximum number of matching result frames (e.g., 3 to 8 frames), and discarding the rest. This step effectively avoids the problem of redundant stacking of results caused by a large number of similar frames returned from the same video.

[0082] S1103 sorts all the simplified results of individual videos in descending order of global similarity to generate the final search results list. It then merges the simplified results of all video groups (along with the static image matching results), sorts them globally in descending order of cosine similarity, generates the final search results list, and outputs it to the user.

[0083] In one embodiment of this application, when the query input is a video frame, it includes: Based on the timestamp information of the simplified single video result and the query image, invalid matching results are removed through a same-source nearest neighbor time window filter. This same-source nearest neighbor time window filter is used to remove matching results that belong to the same video as the query image and whose timestamp difference is less than a preset time window. When the query image submitted by the user originates from a video frame (i.e., the query image contains video source information and timestamp information), the system further performs same-source nearest neighbor time window filtering during the generation of the final search results: For each video group result, the difference between the timestamp of each matching result frame and the original timestamp of the query image is checked. If the difference is less than a preset time window (ranging from 1 to 5 seconds), the matching result is determined to be a "self-match" or "nearest neighbor invalid match" and is removed. This filtering mechanism effectively avoids the problem of invalid matches where the query video frame retrieves itself or adjacent frames, further improving the effectiveness of the search results and the user experience.

[0084] The filtered, simplified single-video results are sorted in descending order of global similarity to generate the final search results list. After completing the source nearest neighbor time window filtering, the simplified results of each video group are sorted in descending order of global similarity to generate the final search results list.

[0085] This application also provides an image search system based on self-supervised visual features, the system comprising: a multi-source data storage module, a feature extraction and storage module, and a retrieval and matching module; The multi-source data entry module is used to generate a set of data to be entered into the database based on multi-source input data through unified entry processing. Specifically, the multi-source data entry module receives static images and video data as input: for static images, images to be entered into the database are generated through standardized preprocessing (size scaling, pixel value normalization); for video data, a sequence of frames to be entered into the database is generated through multi-strategy cascaded processing (fixed interval frame extraction, dynamic keyframe and scene switching detection, target weighted sorting, and feature-level non-maximum deduplication); then the images to be entered into the database and the sequence of frames to be entered into the database are combined to generate the set of data to be entered into the database.

[0086] The feature extraction and storage module is used to extract and generate a global semantic feature vector set based on the dataset to be imported using the DINOv3 self-supervised visual model; and to build and batch store the global semantic feature vector set and corresponding structured index information through a vector database to generate a searchable feature index library. Specifically, the feature extraction and storage module uniformly scales each frame image in the dataset to be imported to the input size of the DINOv3 model, inputs it into the pre-trained DINOv3 visual Transformer encoder after pixel value standardization, outputs a fixed-dimensional global semantic feature vector, and performs L2 normalization on all feature vectors. Then, it configures a globally unique data ID for each feature vector and marks the data type (static image or video frame), attaches a composite structured index (unique video identifier, relative timestamp, and original frame number) to the feature vector from the video frame, builds an approximate nearest neighbor index (HNSW or IVF-PQ) in the vector database, and writes it in batch mode to generate a searchable feature index library.

[0087] The retrieval and matching module is used to generate a query feature vector based on the query image through global semantic feature extraction, and to generate preliminary retrieval results by matching the query feature vector with the searchable feature index library through cosine similarity. Finally, based on the preliminary retrieval results, it generates a final retrieval result through similarity threshold filtering, deduplication of results from the same video, and quantity limitation. Specifically, the retrieval and matching module receives the query image submitted by the user, performs standardized preprocessing consistent with the database entry stage, extracts the global semantic feature vector of the query image using the DINOv3 model and performs L2 normalization; inputs the normalized query feature vector into the retrieval interface of the vector database, performs fast retrieval based on the approximate nearest neighbor index and calculates cosine similarity to generate preliminary retrieval results; then, it filters low-quality matches through a preset similarity threshold, groups videos by unique video identifiers and limits the number of each video group (retaining frames with the highest similarity within a preset maximum number), and finally generates a final retrieval result list by sorting in descending order of global similarity. When the query image is a video frame, the retrieval and matching module also performs source nearest neighbor time window filtering, removing matching results that belong to the same video as the query image and whose timestamp difference is less than a preset time window.

[0088] Furthermore, it may also include a results display module, which is used to present the final search results generated by the search and matching module to the user in a visual manner, including outputting the matched image or video frames, the original video file path (if it is a video), and timestamp or frame number information, thereby achieving content-level positioning and visual result display. See details for further information. Figure 12As shown, in practical applications, it may include a multi-source data entry module, a feature extraction module, a vector storage module, a retrieval and matching module, and a result display module; since the specific implementation logic of each module has been described in detail in the foregoing embodiments, it will not be described in detail here.

[0089] The beneficial technical effects of this application are as follows: By processing multi-source input data in a unified database, static images and video data are represented and managed uniformly in the same feature space, significantly simplifying the system architecture and reducing maintenance costs; by using the DINOv3 self-supervised visual model to extract global semantic feature vectors, highly robust semantic representations across scenes and perspectives can be learned without relying on labeled data, overcoming the problem of insufficient feature expression capabilities of traditional CNNs and significantly improving retrieval accuracy and generalization ability; through multi-strategy cascaded processing of fixed-interval frame extraction, dynamic keyframe extraction and scene switching detection, target weighted sorting, and feature-level non-maximum deduplication, the system ensures the accuracy of video keyframes. The comprehensive content coverage significantly reduces redundant frames, effectively lowering storage and computational burdens. A searchable feature index library is built based on a vector database, combining near-nearest neighbor indexing and cosine similarity matching to achieve millisecond-level efficient retrieval across a large-scale feature library. Simultaneously, similarity threshold filtering, deduplication of video results, and quantity limits effectively prevent redundant stacking of search results, improving result compactness and business usability. The entire solution is highly modular and scalable, supporting incremental data import and dynamic index updates. It can be widely applied to various scenarios such as video tracing, film and television material management, public opinion monitoring, and image library retrieval, demonstrating significant practical value and promising prospects for wider adoption.

[0090] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described method.

[0091] This application also provides a computer-readable storage medium storing a computer program that performs the above-described methods.

[0092] This application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0093] like Figure 13 As shown, the electronic device 600 may also include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily need to include these components. Figure 13 All components shown; in addition, the electronic device 600 may also include Figure 13 For components not shown, please refer to existing technology.

[0094] like Figure 13 As shown, the central processing unit 100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operation of various components of the electronic device 600.

[0095] The memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 100 may execute the program stored in the memory 140 to perform information storage or processing, etc.

[0096] Input unit 120 provides input to central processing unit 100. Input unit 120 may be, for example, a keypad or touch input device. Power supply 170 provides power to electronic device 600. Display 160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0097] The memory 140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 140 can also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function 142 storage unit for storing application programs and function programs or processes for executing the operation of the electronic device 600 via the central processing unit 100.

[0098] The memory 140 may also include a data storage unit (data 143) for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit (driver 144) of the memory 140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0099] The communication module 110 is a transmitter / receiver that transmits and receives signals via the antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processing unit 100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0100] Based on different communication technologies, multiple communication modules 110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 110 (transmitter / receiver) is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby enabling typical telecommunications functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 130 is coupled to a central processing unit 100, enabling on-device recording via the microphone 132 and on-device playback of stored audio via the speaker 131.

[0101] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0102] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0103] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0105] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0106] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0107] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image search method based on self-supervised visual features, characterized in that, The method includes: A set of data to be imported is generated by a unified import process based on multi-source input data. Based on the dataset to be imported, a global semantic feature vector set is generated by extracting and generating it using the DINOv3 self-supervised visual model. The global semantic feature vector set and the corresponding structured index information are indexed and batch stored using a vector database to generate a searchable feature index library; Based on the query image, a query feature vector is generated by global semantic feature extraction. Based on the query feature vector and the searchable feature index, a preliminary search result is generated by cosine similarity matching. Based on the preliminary search results, the final search results are generated by filtering with similarity thresholds, deduplicating results from the same video, and limiting the number of results.

2. The image search method based on self-supervised visual features according to claim 1, characterized in that, Based on multi-source input data, a unified data entry processing method is used to generate a dataset to be entered into the database, including: Images to be entered into the database are generated from static images in multi-source input data through standardized preprocessing. A sequence of frames to be added to the database is generated by multi-strategy cascading processing based on video data from multiple input data sources. The data set to be imported is generated by combining the image to be imported with the frame sequence to be imported.

3. The image search method based on self-supervised visual features according to claim 2, characterized in that, Based on video data, a sequence of frames to be added to the database is generated through multi-strategy cascading processing, including: A basic frame sequence is generated by extracting frames at fixed intervals from the input video. Based on the inter-frame gaps of the base frame sequence, a supplementary frame sequence is generated through dynamic keyframe extraction and scene switching detection. The base frame sequence and the supplementary frame sequence are then merged to generate an initial candidate frame set. Based on the initial candidate frame set, a candidate frame set with priority for inclusion in the database is generated through target detection and priority labeling; Based on the candidate frame set with the priority to be added to the database, the sequence of frames to be added to the database is generated through feature-level non-maximum deduplication, adaptive size scaling, and structured time indexing.

4. The image search method based on self-supervised visual features according to claim 3, characterized in that, Based on the inter-frame gaps of the base frame sequence, a supplementary frame sequence is generated through dynamic keyframe extraction and scene transition detection, including: A valid frame sequence is generated by eliminating invalid transition frames based on adjacent frames through grayscale variance verification. A preliminary dynamic frame is generated based on the valid frame sequence by calculating the absolute pixel difference and determining the adaptive dynamic frame difference threshold. Based on the initial screening of dynamic frames, dynamic keyframes are generated by calculating structural similarity and combining dynamic trigger threshold verification based on image two-dimensional information entropy. Global semantic features are extracted from adjacent frames using the DINOv3 self-supervised visual model, and feature cosine similarity is calculated. When the cosine similarity is lower than a preset scene switching threshold, it is marked as a scene switching frame. Based on adjacent frames of an ultra-high-speed motion scene, ultra-high-speed dynamic keyframes are generated through dense optical flow field calculation and global camera movement suppression verification. The supplementary frame sequence is generated by merging the dynamic keyframe, the scene switching frame, and the ultra-fast dynamic keyframe. The invalid transition frames include pure black frames, no signal frames, solid color frames, and / or blurred frames; the adaptive dynamic frame difference threshold is dynamically generated based on historical frame difference statistics and is adjusted for low-light attenuation according to the average brightness of the frame.

5. The image search method based on self-supervised visual features according to claim 4, characterized in that, Dynamic keyframe generation based on image 2D information entropy dynamic trigger threshold verification includes: The two-dimensional information entropy, which characterizes the texture complexity of the current frame, is calculated based on the gray value of each pixel in the current frame and the average gray value of its neighborhood. The safety threshold and extreme value threshold are generated based on the statistical distribution of historical frame entropy values ​​within the sliding window. The two-dimensional information entropy of the current frame is compared with the security threshold and the extreme value threshold. A degradation factor is generated through nonlinear mapping. The structural similarity determination threshold is dynamically adjusted based on the product of the benchmark threshold and the degradation factor. When the structural similarity of the initial screening dynamic frame is lower than the dynamically adjusted judgment threshold, the current frame is marked as a dynamic key frame.

6. The image search method based on self-supervised visual features according to claim 4, characterized in that, Based on adjacent frames of an extremely high-speed motion scene, high-speed dynamic keyframes are generated through dense optical flow field calculation and global camera movement suppression verification, including: Motion vectors for each pixel are generated by calculating a dense optical flow field between two adjacent frames. Global camera movement determination is performed based on the directional distribution of motion vectors of each pixel. When global camera movement is determined, the rapid dynamic keyframe extraction determination based on dense optical flow method is stopped. In non-global motion state, local extreme motion amplitude is generated by extreme value statistics based on the motion amplitude of all pixels in the frame. When the local extreme value motion amplitude exceeds both the relative background threshold and the absolute motion threshold, the current frame is marked as an ultra-fast dynamic key frame.

7. The image search method based on self-supervised visual features according to claim 3, characterized in that, Based on the initial candidate frame set, a candidate frame set with library priority is generated through target detection and priority labeling, including: Based on the initial candidate frame set, a lightweight object detection model is used to detect and generate candidate frames containing key targets; Based on the candidate frames containing key targets, candidate frames that meet preset conditions are generated by verifying the detection confidence and target proportion. Candidate frames that meet the preset conditions are marked as target weighted frames and assigned the highest priority for inclusion in the database; the remaining frames are marked with their corresponding priorities according to the source frame extraction strategy. Generate a set of candidate frames with priority for inclusion in the library based on the tags; The corresponding priorities, from highest to lowest, are: scene switching frames, dynamic keyframes, and fixed interval frames.

8. The image search method based on self-supervised visual features according to claim 3, characterized in that, Based on the candidate frame set with the input priority, a deduplicated frame set is generated through feature-level non-maximum deduplication, including: Based on the DINOv3 feature vectors of each candidate frame in the candidate frame set with the priority of the library, calculate the cosine similarity between each pair of candidate frames. Visual duplicate frames are divided into groups based on the cosine similarity and the preset deduplication threshold. Based on the priority of data entry, the highest priority single frame is selected as the retained frame from each visual repetition frame group; The deduplicated frame set is generated based on all combinations of retained frames; The data entry priorities, from highest to lowest, are: target weighted frames, scene switching frames, dynamic keyframes, and fixed interval frames.

9. The image search method based on self-supervised visual features according to claim 1, characterized in that, The global semantic feature vector set and corresponding structured index information are indexed and batch-stored using a vector database to generate a searchable feature index library, including: Based on the data source and data type of each vector in the global semantic feature vector set, feature data with type identifiers is generated. The data types include static images and video frames. Configure the feature vector corresponding to the video frame with a unique video identifier, relative timestamp and original frame number to generate feature data with a composite structured index; Perform L2 normalization on all feature data to generate a standardized set of feature vectors; Based on the standardized feature vector set, an approximate nearest neighbor index is constructed to generate an indexed feature set; The indexed feature set is written into the vector database through batch processing to generate the searchable feature index library.

10. The image search method based on self-supervised visual features according to claim 1, characterized in that, Preliminary search results are generated by matching the query feature vector with the searchable feature index using cosine similarity, including: The normalized global semantic feature vector of the query is input into the vector database retrieval interface, and a fast retrieval is performed through the approximate nearest neighbor index to recall a preset number of candidate results. Calculate the cosine similarity between the query feature vector and the feature vector of each candidate result, and sort them in descending order of cosine similarity value to generate the preliminary search results.

11. The image search method based on self-supervised visual features according to claim 10, characterized in that, Based on the preliminary search results, the final search results are generated through similarity threshold filtering, deduplication of results from the same video, and quantity limitations, including: Based on the preliminary search results, a filtered result set is generated by filtering through a preset similarity threshold. Based on the structured index information in the filtered result set, multiple video grouping results are generated by grouping through video unique identifiers. For each video grouping result, sort the results in descending order of cosine similarity, and retain the frame with a preset maximum number of frames to generate a simplified single video result; All simplified single-video results are sorted in descending order of global similarity to generate the final search results list.

12. The image search method based on self-supervised visual features according to claim 11, characterized in that, When the query input is a video frame, it includes: Based on the simplified results of a single video and the timestamp information of the query image, invalid matching results are filtered out by the same source nearest neighbor time window. The same source nearest neighbor time window filter is used to remove matching results that belong to the same video as the query image and whose timestamp difference is less than the preset time window. The filtered, simplified single-video results are sorted in descending order of global similarity to generate the final search results list.

13. An image search system based on self-supervised visual features, characterized in that, The system includes: a multi-source data storage module, a feature extraction and storage module, and a retrieval and matching module; The multi-source data entry module is used to generate a set of data to be entered into the database based on multi-source input data through unified entry processing. The feature extraction and storage module is used to extract and generate a global semantic feature vector set based on the data set to be entered into the database using the DINOv3 self-supervised visual model; and to build and batch store the global semantic feature vector set and corresponding structured index information through a vector database to generate a searchable feature index library. The retrieval and matching module is used to generate a query feature vector by extracting global semantic features from the query image, generate preliminary retrieval results by matching the query feature vector with the searchable feature index library through cosine similarity, and generate final retrieval results by filtering with similarity thresholds, deduplicating results from the same video, and limiting the number of results based on the preliminary retrieval results.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 12.

16. A computer program product, characterized in that, It includes a computer program / instructions stored on a computer-readable storage medium, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

Images