An Image Deduplication Method and System Based on Dual Encoder Feature Extraction and Two-Stage Retrieval

Abstract

This disclosure relates to the field of image processing technology, specifically an image deduplication method and system based on dual-encoder feature extraction and two-stage retrieval. First, a dual-encoder architecture is constructed based on a visual-language pre-trained model and a self-supervised visual model to extract high-level semantics and entity structural morphological features from the image, respectively. Second, a two-stage retrieval strategy is designed: first, CLIP features are used for rapid coarse screening, and then DINOv2 features are used for precise reordering, achieving efficient and accurate global similarity image retrieval. In the fine-ranking stage, a manifold distance metric module based on thermodynamic Markov transitions is introduced to replace traditional cosine similarity calculation, and a list-wise reordering module based on long context sequence modeling is introduced to improve the system's ability to detect complex image manipulation behaviors. This method offers high image deduplication accuracy, is fully automated by computer without manual intervention, and can be used for detecting image reuse, forgery, and tampering in scientific research image integrity review.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image deduplication system and method for dual encoder feature extraction and two-stage retrieval, which is applicable to the detection of academic image forgery, review of scientific research integrity, and automatic identification and location of various types of image tampering. Background Technology

[0002] Research integrity is the cornerstone of academic research. However, in recent years, academic misconduct, especially the forgery, alteration, and reuse of images in papers, has become increasingly prominent. Data released by the National Natural Science Foundation of China indicates that image misconduct has become one of the most challenging issues in research integrity review, urgently requiring effective technical means for detection and prevention.

[0003] Academic image forgery mainly includes two common types: image splicing and copy-move. Image splicing refers to combining parts of different images into a single image, while copy-move involves copying and pasting a region of an image to another location within the same image. With the increasing prevalence of image editing tools and the continuous upgrading of manipulation techniques, these two types of forgery have become more covert and difficult to detect. Traditional image forgery detection methods mainly rely on manual review or simple image comparison. However, when faced with tampered images that have undergone complex operations such as rotation, scaling, brightness adjustment, and partial cropping, manual review is not only inefficient but also struggles to guarantee accuracy. Statistics show that identifying image manipulation in academic image review still largely relies on human experience and judgment, a situation that severely restricts the efficiency and reliability of large-scale academic review work.

[0004] To address the aforementioned issues, academia and industry have successively proposed various image deduplication and forgery detection technologies. Early image deduplication methods primarily employed techniques such as perceptual hashing and feature point matching (e.g., Scale Invariant Feature Transform (SIFT)), capable of detecting simple, nearly identical images. Among these, the copy-and-move forensics method based on SIFT achieved good results in tampering region matching and transformation recovery. Subsequently, related research systematically evaluated mainstream copy-and-move forgery detection methods and proposed more efficient dense field detection strategies to improve adaptability to complex tampering. However, these methods still have significant limitations when handling complex image transformations, especially for images that have undergone rotation, scaling, brightness adjustments, etc., where their detection robustness remains insufficient. In the field of academic image forgery detection, the introduction of deep learning methods has significantly improved detection performance. In recent years, researchers have begun exploring the use of models such as Convolutional Neural Networks (CNN), Generative Adversarial Networks (GAN), and Transformers to detect and locate tampered regions in images. Among these, methods such as ManTra-Net have demonstrated strong performance in various types of image manipulation detection and localization tasks. However, most existing research focuses on detecting single types of forgery, lacking a unified detection framework for complex tampered images involving both splicing and copy-and-move. Achieving joint detection and precise localization of multiple forgery types within the same model remains a critical challenge in the field of image forensics.

[0005] In recent years, foundational visual models, such as the Contrastive Language-Image Pre-training (CLIP) and the Self-Distillation with No Labels version 2 (DINOv2), have demonstrated exceptional capabilities in image representation learning. CLIP, trained through large-scale image-text pair contrastive learning, is able to extract high-level semantic features from images and performs excellently in multimodal tasks. Complementing this, DINOv2, as a foundational model centered on vision, excels in extracting image structure and morphological features through self-supervised learning, capturing more robust visual detail. This suggests that integrating the representational advantages of different foundational visual models holds promise for providing a new technical path for building high-performance image plagiarism and forgery detection systems.

[0006] In the fields of image retrieval and similarity detection, the development of graph neural network (GNN) technology has also provided new ideas for image plagiarism detection systems. GNNs have demonstrated unique advantages in handling non-Euclidean data structures and modeling complex relationships between images. Some researchers have proposed models based on graph attention networks (GAT), which capture similarity relationships between samples through nodes and edges in the graph structure, enabling the model to construct context-aware latent representations. These technological advancements have laid the foundation for building efficient and accurate image plagiarism detection systems. However, how to effectively integrate these advanced technologies and optimize them for the specific needs of academic image plagiarism detection remains an important direction for further exploration in this field. Summary of the Invention

[0007] The purpose of this invention is to propose an image deduplication method and system based on dual encoder feature extraction and two-stage retrieval, so as to solve the problems of low efficiency in scientific research image deduplication and difficulty in detecting complex transformations and local reuse behavior in the existing technology.

[0008] Firstly, this disclosure proposes an image deduplication method based on dual encoder feature extraction and two-stage retrieval. The steps include: extracting high-level semantic features from a query image and its corresponding geometrically transformed enhanced image using a first encoder; retrieving the top N1 candidate images with the highest similarity from an image database based on the query feature vector obtained by fusing the semantic features, thus forming a candidate image list; performing structured feature sorting on the candidate images in the candidate image list to obtain a rearranged candidate image list; wherein, the structured feature sorting is performed by extracting structured features from the query image and the candidate images to be sorted using a second encoder, concatenating the structured features of the query image and the candidate images to be sorted to obtain a ranking score for the candidate images to be sorted; and performing overlapping sliding on the current candidate image list. For each candidate image within a window, structured information is used for ranking. Then, a weighted average of the ranking scores of each candidate image across all windows is calculated to obtain the current ranking score of each candidate image. The top RN candidate images with the highest ranking scores are selected. For each current candidate image, the LPMT manifold distance between the query image and the candidate image in the high-level semantic feature space is calculated, as well as the LPMT manifold distance between the query image and the candidate image in the structured feature space. The LPMT manifold distances of the two feature spaces are weighted and fused and converted into a similarity score. This similarity score is then weighted and fused with the current ranking score corresponding to the candidate image to obtain the final ranking score of the candidate image. Based on the final ranking score of each candidate image, the similar images are ranked.

[0009] Secondly, this disclosure proposes an image deduplication system based on dual encoder feature extraction and two-stage retrieval, comprising: a coarse ranking module, configured to extract high-level semantic features from a query image and its corresponding geometrically transformed enhanced image using a first encoder; based on the query feature vector obtained by fusing the semantic features, retrieving the top N1 candidate images with the highest similarity from an image database to form a candidate image list; and a fine ranking module, configured to perform structured feature sorting on the candidate images in the candidate image list to obtain a rearranged candidate image list; performing overlapping sliding window operations on the current candidate image list, sorting the candidate images within each window according to structured information, and then weighted averaging the ranking scores of each candidate image across all windows to obtain the current ranking score of each candidate image. The top RN candidate images with the highest ranking scores are selected. For each candidate image, the LPMT manifold distance between the query image and the candidate image in the high-level semantic feature space is calculated, and the LPMT manifold distance between the query image and the candidate image in the structured feature space is also calculated. The LPMT manifold distances of the two feature spaces are weighted and fused and converted into a similarity score. This similarity score is then weighted and fused with the current ranking score corresponding to the candidate image to obtain the final ranking score of the candidate image. Based on the final ranking score of each candidate image, the similar images are ranked. The structured feature ranking is achieved by using a second encoder to extract the structured features of the query image and the candidate images to be ranked, and then concatenating the structured features of the query image and the candidate images to be ranked to obtain the ranking score of the candidate images to be ranked.

[0010] The beneficial effects of this invention are as follows: By extracting complementary features from images through a dual-encoder architecture, it possesses both high-level semantic understanding and fine-grained geometric discrimination capabilities, achieving efficient encoding of image features. After converting all images in the image library into two feature codes and storing them, only a small number of test image features need to be extracted during testing. Then, the similarity between the test image and all images can be calculated through simple matrix multiplication, greatly reducing the image comparison time. Through the LPMT manifold distance metric module and the list-wise reordering module, the manifold structure information of the feature space and cross-image dependencies are effectively utilized, significantly improving the detection capability for complex image transformations and local reuse behaviors. The detected target image can be a variant image that has been rotated, scaled, brightness adjusted, locally cropped, or even locally stitched and tampered with. The entire plagiarism detection process can be completed automatically by the computer without manual intervention, exhibiting strong robustness and practicality. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of an image deduplication method based on dual encoder feature extraction and two-stage retrieval, as implemented in one embodiment.

[0013] Figure 2 This is a schematic diagram of an image plagiarism detection system architecture based on dual encoder feature extraction and two-stage retrieval in one implementation.

[0014] Figure 3 This is a schematic diagram illustrating the display of query results in one implementation method. Detailed Implementation

[0015] Terminology explanation.

[0016] A patch descriptor is a standard term in computer vision and image processing. It refers to a numerical vector used to quantify a small region (i.e., a patch) in an image. In other words, it transforms an irregular, visually understandable small image patch (such as a small patch of skin texture or the area around a corner) into a fixed-length array that is easy for a computer to compare.

[0017] This invention proposes an image deduplication method based on dual encoder feature extraction and two-stage retrieval. Designed using deep learning and large-scale models, it is particularly suitable for detecting academic image forgery, verifying research integrity, and automatically identifying and locating various types of image tampering. The method includes: first, constructing a dual encoder architecture comprising CLIP and DINOv2 large-scale models to extract high-level semantic features and structural features of the image; second, designing a two-stage retrieval strategy, using high-level semantic features for rapid coarse screening, and then using structural features for precise reordering, achieving efficient and accurate global similarity image retrieval. In the fine-ranking stage, this invention further introduces a manifold distance metric module (LPMT) based on thermodynamic Markov transitions to replace traditional cosine similarity calculation. Furthermore, this invention introduces several innovative technical modules, including a list-wise reordering module based on long context sequence modeling.

[0018] The following provides a clear and complete description of how the technical solution of this case is implemented. Obviously, the described implementation methods are only a part of the implementation methods of this case, and not all of them. Based on the implementation methods in this case, all other implementation methods obtained by those skilled in the art without inventive effort are within the scope of protection of this application.

[0019] See Figure 1 An image deduplication method based on dual encoder feature extraction and two-stage retrieval, the processing steps of which include the following steps after inputting the query image.

[0020] Step 1: Image Preprocessing and Geometric Transformation Enhancement The input image undergoes standardized preprocessing and multi-view enhancement to eliminate background noise and illumination interference caused by differences in image acquisition environment and equipment. This addresses the problem of basic feature drift in images (such as medical micrographs, electrophoretic band images, etc.) caused by scanning artifacts, edge-independent white space, and non-uniform illumination. Simultaneously, multi-view enhancement constructs the local and global spatial correspondence of the image, providing scale invariance and anti-clipping properties for subsequent models, and improving the system's robustness against local tampering and collage fraud.

[0021] Image preprocessing includes: (1) white border removal, (2) letterbox scaling, (3) color normalization, and (4) geometric transformation enhancement. Details are as follows.

[0022] The white border removal step includes: for the input image First, convert it to grayscale. Then, the gray-level histograms of the four edge regions of the image are calculated, and the Otsu algorithm (Otsu's thresholding method) is used to determine the segmentation threshold between the background and the foreground. Edge detection is performed on the image, and regions where the gray-level value of the edge pixels is greater than the segmentation threshold and the continuous length exceeds 5% of the image's side length are identified as white edges, and cropped along this direction. If the image has multiple white edges (such as borders on all four sides), the above operation is recursively performed on the cropped image until the ratio of the remaining effective area to the original image reaches a preset value, such as not less than 80% of the original image. The image after removing white edges is denoted as [image name missing]. .

[0023] Image edge detection can employ edge detection algorithms such as Sobel and Canny. These algorithms mark pixels that match edge features. These marked pixels are called edge pixels.

[0024] The Letterbox scaling step includes: setting the target size as... , Let be the side length. In this embodiment, we take . Pixels, consistent with the input specifications of CLIP and DINOv2. For images after white border removal. Calculate its aspect ratio , For width, It is high. According to Determine the scaling factor based on the relationship with the target aspect ratio. Scale the image proportionally to Then, pixels with a grayscale value of 0 are used to fill the area around the scaled image, resulting in a size of [size missing]. square image S is the preset value.

[0025] The color normalization step includes: converting the image... Convert to RGB color space. For an image that was originally grayscale, duplicate all three channels to the same grayscale value. Then, perform global normalization on the image pixel values: ,in and These are the pre-calculated RGB channel mean and standard deviation on an image-based dataset. In this embodiment, the dataset used is ImageNet, and the calculated mean... and standard deviation They are respectively: Image after color normalization As a query image.

[0026] The geometric transformation enhancement is designed to be applied only to the query image: for the original input image In addition to the query image obtained through the standard preprocessing described above In addition, the following enhanced images are generated: white-edge removal enhanced image, center-cropped enhanced image, four-corner-cropped enhanced image, and random-cropped enhanced image.

[0027] The white border removal and enhancement image is an image obtained by performing only white border removal and letterbox scaling, without performing other transformations.

[0028] The center-cropping enhanced image generation step includes: in the image after removing white borders... Above, using the image center as the center point, crop out a region of a first preset size, for example, a region with an area equal to the original area (with a side length scaling factor of 1). Then scale it using Letterbox. .

[0029] The four-corner cropping enhancement image generation step includes: using the images after removing white edges... Using the top left, top right, bottom left, and bottom right corners as centers, crop out areas of a second preset size. For example, crop a square area with sides 50% of the original side length. Scale each area using Letterbox. This step yields four images.

[0030] The random cropping and enhanced image generation step includes: in the image after removing white borders... Above, randomly select a location and crop out an area of ​​the third preset size, such as a square area with a side length of 40% to 80% of the original side length, and then scale it down. .

[0031] During the query, for the eight images—the query image, the white-border-removed view, the center-cropped view, the four-corner-cropped view, and the random-cropped view—features are extracted from each image using the first encoder, and then fused into a unified query feature vector using max pooling.

[0032] Step 2: Dual encoder image feature extraction A dual-encoder architecture, comprising a first encoder and a second encoder, is employed. The first encoder uses the CLIP large model to extract high-level semantic features of the image; the second encoder uses the DINOv2 large model to extract entity-structured features reflecting the local structure, texture organization, and morphological layout of the image. Compared to traditional feature extraction methods that rely on edges, colors, or local descriptors (such as scale-invariant feature transforms) and conventional convolutional neural network (CNN) models that only focus on shape contours, this dual encoder can capture high-fine-grained cross-modal semantic priors and pixel-level self-similar structures from massive amounts of unlabeled data, thereby effectively distinguishing highly visually deceptive and sophisticated tampering in the feature space.

[0033] Specifically, the first encoder uses the ViT-B / 32 visual branch of CLIP to extract high-level semantic features (CLIP features). For an input image of size 224×224, ViT-B / 32 divides it into... Each image patch is a non-overlapping image block. Each patch is mapped to a 768-dimensional embedding vector through a linear projection layer, and a learnable positional encoding is added. These block embeddings, along with additional [CLS] tags, are then fed into a 12-layer Transformer encoder. Each Transformer layer contains a multi-head self-attention network (12 heads) and a feedforward network (intermediate layers with a dimension of 3072), and employs residual connections and layer normalization.

[0034] During forward propagation of the model, the final output vector corresponding to the [CLS] label serves as the global representation of the image. In this embodiment, this vector is extracted as the high-level semantic feature vector, with an original dimension of 768. A trainable linear projection layer is then appended to this output, compressing the dimension from 768 to 512; that is, the compressed high-level semantic feature vector has a dimension of 512. The eight enhanced image compressed high-level semantic feature vectors are then fused together using max pooling to obtain the final high-level semantic feature vector. .

[0035] The second encoder uses the ViT-S / 14 model of DINOv2 to extract structured features reflecting the local structure, texture organization, and morphological layout of the image, outputting a sequence of patch-level feature vectors. For a 224×224 input image, ViT-S / 14 divides it into... Each image block is linearly projected into a 384-dimensional embedding, which, after being encoded, is input into a 12-layer Transformer encoder. Each Transformer layer employs a 6-head self-attention network and a feedforward network with intermediate layers of 1536 dimensions. This invention extracts the final output vector corresponding to the [CLS] marker. As a structured feature vector, its dimension is 384, requiring no additional projection.

[0036] To ensure comparability between the two feature vectors in subsequent similarity calculations and to eliminate the impact of modulus differences on distance metrics, this invention respectively... and Perform L2 normalization:

[0037] After L2 normalization, the magnitude of both feature vectors is 1, and the cosine similarity is equivalent to the inner product.

[0038] Step 3: Two-stage retrieval: Similar image matching The first stage is the coarse screening stage, which uses high-level semantic feature vectors to retrieve the top N1 candidate images with the highest similarity to the query image from the image database based on cosine similarity.

[0039] Specifically, based on a unified query feature vector The algorithm uses cosine similarity to quickly retrieve the top N1 candidate images with the highest similarity to the query image from the image library, forming a candidate image list. For example, N1 = 200. Specifically, the image library uses FAISS to index and store the high-level semantic feature vectors of the images. During retrieval, the high-level semantic feature vector of the query image is compared with the high-level semantic features of each image stored in the FAISS (Facebook AI Similarity Search) index to calculate cosine similarity, obtaining the cosine similarity score between all images and the query image. Then, through the FAISS similarity search interface, the indexes and corresponding scores of the 200 candidate images with the highest similarity scores are returned.

[0040] The second stage is the fine ranking and re-ranking stage. For the candidate image list obtained in the coarse screening stage, a list-wise re-ranking module based on long context sequence modeling is used. The query image and the candidate image list are input as a whole, and the long context sequence model is used to capture the dependencies across images to obtain the re-ranking score.

[0041] Specifically, for the 200 candidate images obtained in the coarse screening stage, a list-wise reordering module based on long context sequence modeling is introduced. The list-wise reordering module sequentially performs local descriptor sequence extraction, sequence construction and position encoding, long context sequence model processing and execution of sliding window strategy.

[0042] The local descriptor sequence extraction includes: for both the query image and each candidate image, using the DINOv2 model to output patch-level features representing fine-grained structural units of each local region in the image, and using these as patch descriptors or local descriptors. The DINOv2ViT-S / 14 model divides the 224×224 input image into 256 image patches, with each patch corresponding to an output feature dimension of 384. Flattening these 256 patch features yields the local descriptor sequence corresponding to the candidate image. .

[0043] The sequence construction and location encoding include: converting the local descriptor sequence of the query image... Local descriptor sequences of all candidate images Concatenate along the length dimension of the sequence to form a sequence of length. The extremely long sequence. A special [SEP] marker is inserted at the end of the local descriptor sequence of each image, and a [CLS] marker is added at the beginning of the sequence, resulting in a final sequence of length. The sequence is then processed. Encoding information is appended to each patch feature to characterize its local spatial location within the image, its image affiliation, and its global order within the overall stitched sequence, preserving the spatial relationships and cross-image affiliation of the two-dimensional image structure after it is flattened into a one-dimensional sequence. Specifically, for the local descriptor of the p-th patch in the i-th image... Construct the patch position encoding within the image respectively. Image index coding Total sequence absolute position encoding The aforementioned encoded vectors are all learnable parameters that are automatically updated during training. Then, these three vectors are element-wise added to the local descriptor to obtain the sequence model input:

[0044] income The input long context sequence model is processed in a predetermined order, along with the [CLS] and [SEP] tags.

[0045] In the above, image index encoding is used to explicitly identify the image identity to which each local descriptor belongs, enabling the sequence model to distinguish the feature sources of the query image and each candidate image in cross-image stitching scenarios, thereby modeling the correspondence, competition, and difference relationships between images. Absolute sequence position encoding is used to characterize the global position of the local descriptor in the overall sequence after cross-image stitching, enabling the long context sequence model to maintain awareness of the sequence order and window boundaries, reducing the confusion of positional information caused by sequence flattening after multi-image stitching. Patch position encoding is used to preserve the local spatial layout within a single image. The three working together can simultaneously maintain the three types of structural information—"intra-image position—inter-image attribution—global sequence order"—when flattening the two-dimensional image patch sequence and inputting it into a one-dimensional long sequence model, thereby improving the ability to distinguish cross-image local reuse, stitching, and similar interference samples during reordering.

[0046] The long context sequence model processing includes: using the large language model Mamba as the backbone sequence model. For the input sequence, the Mamba sequence model performs layer-by-layer transformation through a selective state space layer and a feedforward network to obtain the correspondence, competition, and difference relationships between the query image and multiple candidate images at the local structural level. The hidden states corresponding to the [CLS] labels output by the model are fed into a multilayer perceptron head, with an output dimension of... The sorted score vector is used to obtain a list of candidate images for rearrangement.

[0047] The implementation of the sliding window strategy includes: performing an overlapping sliding window on the current candidate image list. Specifically, the current candidate images are divided into multiple fixed-length sublists, each sublist containing... 1 candidate image, with overlapping sublists. There are several candidate images. For example, if the image sequence in one sublist is 0-9, the image sequence in the next adjacent sublist is 7-16. For each window, the query image is concatenated with the candidate images within the window, and the ranking score within that window is obtained through the sequence model Mamba. The final score of each candidate image is the weighted average of the scores of that candidate image appearing in all windows, with the weights dynamically assigned by the sequence model Mamba based on the candidate image's position within the window. By employing a sliding window strategy, the local structural association strength between the query image and candidate images is first evaluated using each subset separately, and then the ranking results of the same candidate image in multiple windows are aggregated, thus balancing long-context comparison capability and computability.

[0048] The two-stage strategy described above not only ensures real-time retrieval in a large-scale image database by utilizing global high-level semantic features in the coarse screening stage, but also abandons the traditional point-wise independent scoring mechanism in the fine ranking stage. By using long context modeling, the candidate image set is regarded as a whole sequence, effectively capturing the implicit contrast clues and distribution competition relationships between candidate images. Thus, when there is a large amount of similar background interference, the true homologous tampered images can be accurately located.

[0049] Step 4: LPMT manifold distance measurement Next, in conjunction with LPMT manifold distance, the similarity between images at both semantic and structural levels is characterized using LPMT manifold distance. Specifically, this involves: ranking each candidate image based on its final score, selecting the top RN candidate images with the highest scores, and calculating the LPMT manifold distance between the query image and each candidate image. The list-wise reordering score of candidate images is weighted and fused with manifold distance to obtain the final comprehensive ranking, which in turn yields the similarity image ranking output. An example value for RN is 30. Details are as follows.

[0050] Step 4.1: Constructing the local state distribution.

[0051] For each image in the image library Let its structured feature vector be Then perform L2 normalization. Search the entire image database for images with... The set of neighbor indices is obtained by finding the K most similar neighbors (excluding the self). The feature vectors of these K neighbors and their corresponding features are... The similarity information is input into a lightweight MLP, and the output is a probability distribution. , representing an image The possibility of being in various local states.

[0052] Specifically: ,in This represents a vector concatenation operation. Let j be the structured feature vector of image j. Let be the cosine similarity between image i and image j (since the features have been normalized). The MLP contains two fully connected layers: the first layer takes the input dimension... Mapping to 128, the second layer maps 128 to the number of local states. (Settings of this invention) After softmax, And satisfy This distribution It can be understood as an image The "thermodynamic state" on the feature space manifold has an entropy value that reflects the degree of dispersion of the local neighborhood: the higher the entropy, the more uniformly the neighbors around the image are distributed, and it is located in a dense manifold region; the lower the entropy, the more unique the local structure of the image, and it is located in a sparse manifold region.

[0053] Step 4.2: Markov transfer cost calculation.

[0054] The probability distribution of each image i If considered as a state, then from the image The corresponding state transitions to the image The cost of the corresponding state is defined as the KL divergence between the two distributions: Construct a state transition graph with each state as a node, where edges represent the costs incurred during the transition between two states with a transition relationship. Cost This reflects the image The neighborhood structure of the corresponding state is "deformed" into an image. The information loss required for the neighborhood structure of the corresponding state is asymmetric.

[0055] To obtain a symmetrical distance metric, this embodiment defines a distance metric from the image. Corresponding state to image The cumulative cost of the minimum transition path for the corresponding state is shown below.

[0056]

[0057] In the above formula Let T represent the t-th node on a candidate path, where T represents the upper limit of the number of edges contained in this path.

[0058] An approximate solution method based on diffusion kernels is used to solve for the manifold distance. A symmetric affinity matrix is ​​constructed. ,in , is the scaling parameter (set to 0.5 times the median of the KL divergence of all non-self-paired pairs). For Normalizing the rows yields the Markov transition matrix. ,in diagonal matrix .image and The manifold distance between them is defined as: , and Let be the standard basis vectors corresponding to the i-th node and the j-th node, respectively. This represents the diffusion operator after the transition matrix has propagated through t edges.

[0059] In actual implementation, truncation to Step, error less than .

[0060] Using the above method, manifold distance calculation is performed in both the CLIP feature space and the DINOv2 feature space, including: obtaining the probability distribution of the query image in the high-level semantic feature space. Obtain the probability distribution of candidate images in the high-level semantic feature space. Calculate the minimum cumulative cost of the transition path from the query image to the candidate image as the LPMT manifold distance; obtain the probability distribution of the query image in the structured feature space. Obtain the probability distribution of candidate images in the structured feature space. The minimum cumulative cost of the transition path from the query image to the candidate image is calculated as the LPMT manifold distance.

[0061] Next, dual-feature space fusion is performed.

[0062] Manifold distance calculations will be performed on the CLIP feature space and the DINOv2 feature space respectively, and the results will be... and After normalization, the results are as follows: and The final distance is obtained by weighted summation: ,in For weights. Example values. .

[0063] Will Convert to similarity score .

[0064] Step 4.3: Output the sorting results of similar images.

[0065] The output score of the list-wise reordering module is weighted and fused with the LPMT similarity score to obtain the final refined ranking score:

[0066] in For weights, example values ​​are given. . Let j be the final ranking score for candidate image j. To query the similarity score between image i and candidate image j.

[0067] The final ranking score is used to output the sorting results of similar images.

[0068] By using the manifold distance LPMT method based on thermodynamic Markov transfer, the traditional Euclidean / cosine point-to-point metric is transformed into probability distribution transfer cost calculation on the feature manifold space. This method can effectively characterize the nonlinear topological structure in the high-dimensional sparse feature space. It not only overcomes the problem of direct measurement failure caused by complex image deformation (such as elastic distortion and viewpoint transformation), but also discovers deep correlations hidden on the feature manifold path, greatly enhancing the ability to identify deepfake and multi-layered tampered images.

[0069] See Figure 2 The schematic diagram illustrates the architecture of an image plagiarism detection system based on dual encoder feature extraction and two-stage retrieval, including a preprocessing module, a dual encoder module, a coarse screening stage module, and a fine sorting stage module.

[0070] The preprocessing module is configured to perform the following processing on the query image: white border removal, letterbox scaling, color normalization, and multi-view enhancement.

[0071] The dual encoder module is configured to use CLIP's ViT-B / 32 as the first encoder to extract high-level semantic features, and the final features are denoted as high-level semantic feature vectors. The ViT-S / 14 model of DINOv2 is used as the second encoder to extract structural morphological features, and the final features are denoted as structured feature vectors. Then on and L2 normalization is performed to obtain the feature vectors. and .

[0072] The coarse screening stage module is configured to perform matrix multiplication between the CLIP feature vector of the query image and the CLIP feature matrix of the image library stored in the FAISS index to obtain the cosine similarity score between all images and the query, and obtain the index and corresponding score of the TOP-200 candidate images based on the cosine similarity score.

[0073] The fine-ranking stage module is configured to obtain patch descriptors for the query image and each candidate image using the DINOv2 model, obtain local descriptor sequences based on the patch descriptors, input the local descriptor sequences corresponding to all images into the large language model Mamba-2.8B, and obtain the ranking scores for all images; based on the ranking scores, obtain a candidate image list, and use a sliding window strategy to obtain the final score for each candidate image; based on the final score of each candidate image, obtain the top RN corresponding candidate images from the image database, calculate the LPMT distance between the query image and each candidate image, and then obtain the final score, and finally obtain the similar image ranking output.

[0074] To verify the effectiveness of the image plagiarism detection system and method based on dual encoder feature extraction and two-stage retrieval proposed in this invention, a systematic experiment was conducted on a self-built database containing 100,000 academic images (covering various types such as Transwell, WB strip, electron microscopy, flow cytometry, scratch assay, immunofluorescence staining, and tissue staining).

[0075] Figure 3 The document displays some of the query images and their corresponding search results, along with similarity scores based on CLIP features, DINO features, and the method of this invention, as shown in Tables 1 and 2. Table 1 shows the similarity score for finding one most similar image from the image database based on one image, and Table 2 shows the similarity score for finding two most similar images from the image database based on one image.

[0076] Table 1

[0077] Table 2

[0078] In summary, for the original query image, the method of this invention constructs a two-layer feature representation for image deduplication tasks, a cross-image structured comparison sequence, and a multi-stage sorting mechanism to accurately retrieve target images that are highly similar to its content, including variant images that have been rotated, scaled, brightness adjusted, locally cropped, or even locally spliced ​​and tampered with.

[0079] Across all queried images, the mAP@10 (mean Average Precision) using the method of this invention reached 0.937, a 15.4% improvement over single CLIP feature retrieval (0.812) and a 7.8% improvement over single DINOv2 feature retrieval (0.869). For tampered images subjected to global transformations such as rotation, scaling, and brightness adjustment, the recall@20 using the method of this invention was 0.965; for locally tampered images containing splicing or copy-and-move, the recall@20 was 0.912. Regarding retrieval efficiency, a two-stage retrieval strategy was employed, with an average query time of 0.47 seconds (database size 100,000 images). The coarse screening stage, utilizing the FAISS index, compressed the candidate set from 100,000 to 200 images, taking approximately 0.09 seconds; the fine ranking stage performed LPMT distance calculation and list-wise re-sorting on the 200 candidate images, taking approximately 0.38 seconds. The overall retrieval speed meets the real-time requirements of actual academic image review.

[0080] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0081] For example, an image deduplication method based on dual encoder feature extraction and two-stage retrieval includes the following steps: For a query image and its corresponding geometrically transformed enhanced image, a first encoder is used to extract high-level semantic features; based on the query feature vector obtained by fusing the semantic features, the top N1 candidate images with the highest similarity are retrieved from the image database to form a candidate image list; the candidate images in the candidate image list are sorted by structured features to obtain a rearranged candidate image list; wherein, the structured feature sorting involves using a second encoder to extract structured features from the query image and the candidate images to be sorted, concatenating the structured features of the query image and the candidate images to be sorted, and using Vision... Mamba obtains the ranking scores of candidate images to be ranked; it performs overlapping sliding window operations on the current candidate image list, sorts the candidate images in each window according to structured information, and then calculates the weighted average of the ranking scores of each candidate image in all windows to obtain the current ranking score of each candidate image. It selects the top RN candidate images with the highest ranking scores; for each current candidate image, it calculates the LPMT manifold distance between the query image and the candidate image in the high-level semantic feature space, calculates the LPMT manifold distance between the query image and the candidate image in the structured feature space, weights and fuses the LPMT manifold distances of the two feature spaces and converts them into a similarity score, and weights and fuses the similarity score with the current ranking score corresponding to the candidate image to obtain the final ranking score of the candidate image. Based on the final ranking score of each candidate image, it obtains the ranking of similar images.

[0082] For example, this disclosure proposes an image deduplication system based on dual encoder feature extraction and two-stage retrieval, comprising: a coarse ranking module, configured to extract high-level semantic features from a query image and its corresponding geometrically transformed enhanced image using a first encoder, and retrieve the top N1 candidate images with the highest similarity from an image database based on the query feature vector obtained by fusing the semantic features, to form a candidate image list; and a fine ranking module, configured to perform structured feature sorting on the candidate images in the candidate image list to obtain a rearranged candidate image list; and to perform overlapping sliding windowing on the current candidate image list, sorting the candidate images in each window by structured information, and then weighting the ranking scores of each candidate image in all windows to obtain the current ranking of each candidate image. The ranking score is calculated by selecting the top RN candidate images with the highest ranking scores. For each candidate image, the LPMT manifold distance between the query image and the candidate image in the high-level semantic feature space is calculated, and the LPMT manifold distance between the query image and the candidate image in the structured feature space is also calculated. The LPMT manifold distances of the two feature spaces are weighted and fused and converted into a similarity score. This similarity score is then weighted and fused with the current ranking score corresponding to the candidate image to obtain the final ranking score of the candidate image. The similarity image ranking is obtained based on the final ranking score of each candidate image. The structured feature ranking is achieved by extracting the structured features of the query image and the candidate images to be ranked using a second encoder, concatenating the structured features of the query image and the candidate images to be ranked, and obtaining the ranking score of the candidate images to be ranked using Vision Mamba.

[0083] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk drive or magnetic tape drive. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0084] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0085] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0086] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0087] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0088] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0089] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0090] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0091] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0092] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

[0093] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An image deduplication method based on dual encoder feature extraction and two-stage retrieval, characterized in that the steps are as follows: include: For the query image and its corresponding geometric transformation enhanced image, the first encoder is used to extract high-level semantic features. Based on the query feature vector obtained by fusing the semantic features, the top N1 candidate images with the highest similarity are retrieved from the image library to form a candidate image list. The candidate images in the candidate image list are sorted by structured features to obtain a rearranged candidate image list; wherein, the structured feature sorting is performed by using a second encoder to extract the structured features of the query image and the candidate images to be sorted, and concatenating the structured features of the query image and the candidate images to be sorted to obtain the sorting score of the candidate images to be sorted. An overlapping sliding window is executed on the current candidate image list. The candidate images in each window are sorted by structured information. Then, the ranking scores of each candidate image in all windows are weighted and averaged to obtain the current ranking score of each candidate image. The top RN candidate images with the highest ranking scores are selected. For each candidate image, calculate the LPMT manifold distance between the query image and the candidate image in the high-level semantic feature space, calculate the LPMT manifold distance between the query image and the candidate image in the structured feature space, weight and fuse the LPMT manifold distances of the two feature spaces and convert them into a similarity score, weight and fuse the similarity score with the current ranking score corresponding to the candidate image to obtain the final ranking score of the candidate image, and obtain the similar image ranking based on the final ranking score of each candidate image.

2. The method according to claim 1, characterized in that, The query image has undergone preprocessing, and the preprocessing steps include: After converting the original view to grayscale, edge detection is performed to identify white edges. These white edges are then cropped until the remaining effective area's ratio to the original image reaches a preset value, resulting in the image with white edges removed. ; For images After scaling the Letterbox, fill the surrounding area with pixels of grayscale value 0, resulting in a size of [size missing]. square image S is a preset value; Image The image is converted to RGB color space and the pixels are normalized to obtain the query image. .

3. The method according to claim 1, characterized in that, Geometric transformation image enhancement includes white-edge removal enhancement, center cropping enhancement, corner cropping enhancement, and random cropping enhancement; where: The white-border-removed enhanced image is the image obtained after white-border removal and letterbox scaling. The scaled size is the same as the query image size; The center-cropped enhanced image is the image after removing the white borders. Above, using the center of the image as the center point, a region of the first preset size is cropped out, and then scaled to the same size as the query image using Letterbox; The four-corner cropped enhanced image is the image after removing the white borders. Using the top left, top right, bottom left, and bottom right corners as centers, crop out an area of ​​the second preset size, and then scale it using Letterbox to the same size as the query image; The randomly cropped enhanced image is the image after removing white borders. Above, a location is randomly selected, a region of the third preset size is cropped out, and then scaled to the same size as the query image.

4. The method according to claim 1, characterized in that, For images in the image library, FAISS is used to index and store the high-level semantic feature vectors of the images. During retrieval, the high-level semantic features of the query image are compared with the high-level semantic features of each image stored in the FAISS index to calculate the cosine similarity, thereby obtaining the top N1 candidate images with the highest similarity.

5. The method according to claim 1, characterized in that, The first encoder is the ViT-B / 32 vision branch of CLIP, and the second encoder is the ViT-S / 14 model of DINOv2.

6. The method according to claim 1, characterized in that, The LPMT manifold distance is the minimum cumulative cost of the transition path from one state to another, where each state is a probability distribution of each image, and the cost of transitioning between two states is the divergence between the two probability distributions. The steps for obtaining the probability distribution of each image are: first, based on the structured features of the image... Obtain from image library The K most similar neighbors are selected, where K is a preset value. Then, the structured features corresponding to these K neighbors and their similarity to each other are analyzed. The similarity information is input into the multilayer perceptron, and the multilayer perceptron outputs the probability distribution of the image.

7. The method according to claim 1, characterized in that, The steps for obtaining the similarity score include: The LPMT manifold distance of the query image in the high-level semantic feature space Normalization yields ; The LPMT manifold distance corresponding to the query image in the structured feature space. Normalization yields ; The final distance is obtained by weighted summation: , As weight; Will Convert to similarity score .

8. The method according to claim 1, characterized in that, The final ranking score is calculated as follows: In the formula, As weight, Let j be the final ranking score for candidate image j. To query the similarity score between image i and candidate image j.

9. An image deduplication system based on dual encoder feature extraction and two-stage retrieval, characterized in that, include: The coarse ranking module is configured to extract high-level semantic features from the query image and its corresponding geometric transformation enhanced image using the first encoder. Based on the query feature vector obtained by fusing the semantic features, the module retrieves the top N1 candidate images with the highest similarity from the image library to form a candidate image list. The fine-ranking module is configured to perform structured feature sorting on candidate images in the candidate image list to obtain a rearranged candidate image list; perform overlapping sliding windowing on the current candidate image list, sort the candidate images in each window according to structured information, and then calculate the weighted average of the ranking scores of each candidate image in all windows to obtain the current ranking score of each candidate image, and select the top RN candidate images with the highest ranking scores; for each current candidate image, calculate the LPMT manifold distance between the query image and the candidate image in the high-level semantic feature space, calculate the LPMT manifold distance between the query image and the candidate image in the structured feature space, weight and fuse the LPMT manifold distances of the two feature spaces and convert them into a similarity score, and weight and fuse this similarity score with the current ranking score corresponding to the candidate image to obtain the final ranking score of the candidate image, and obtain the similar image ranking based on the final ranking score of each candidate image; The structured feature ranking process involves extracting structured features from the query image and the candidate images to be ranked using a second encoder, and then concatenating these structured features to obtain a ranking score for the candidate images to be ranked.

Images