Screen image copyright protection method based on visual saliency multi-scale Hash
By employing a visual saliency multi-scale hashing method, the challenges of insufficient capture of screen-specific elements and balanced global and local details in screen image copyright protection are addressed, achieving highly efficient copyright protection.
Patent Information
- Application Number
- CN202510466973.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-15
- Publication Date
- 2025-11-07
AI Technical Summary
Existing visual hashing algorithms struggle to effectively capture screen-specific elements in screen image copyright protection, failing to balance global layout with local details, resulting in insensitivity to subtle tampering detection.
A visual saliency-based multi-scale hashing method is adopted. Key regions of screen images are detected by a visual saliency model. Combined with a dynamic multi-scale feature extraction network and cross-scale feature fusion, hash codes based on CP tensor decomposition are generated.
It improves the accuracy and robustness of screen image copyright protection, effectively captures key features of screen images, and enhances sensitivity to subtle tampering and overall performance.
Smart Images
Figure CN120912907A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of digital image processing, computer vision and information security, and particularly relates to a method for screen image content copyright protection and copy detection using deep learning and hash technology. BACKGROUND
[0002] With the popularization of digital scenarios such as remote office, online education and digital publishing, screen images such as screenshots, software interfaces and web page contents have become important carriers of information dissemination. However, screen images are easy to copy, intercept and tamper, which brings severe challenges to the protection of intellectual property rights of digital content. Traditional copyright protection technologies such as digital watermarking or encryption have limitations when applied to screen images. For example, watermarks may affect the visual experience or be easily removed by attacks, and encryption is not suitable for publicly displayed content.
[0003] In recent years, content-based image hash technology has attracted attention in the fields of image retrieval and copyright protection due to its efficiency and robustness to content modification. Visual hashing aims to generate a compact binary code (hash code) for an image, so that images with similar content have similar hash codes (usually measured by Hamming distance), while hash codes of content-independent images are significantly different. Existing visual hashing algorithms, including some deep hashing methods based on deep learning, have achieved some success on natural images. For example, image features are extracted by a convolutional neural network (CNN), and then a binary code is generated by a hash layer.
[0004] However, there are two key problems when directly applying existing visual hashing algorithms to screen images:
[0005] First, there are significant differences in statistical characteristics between screen images and natural images. Screen images usually contain a large amount of text, icons, window boundaries and other structured elements designed by humans, with sharp edges, large areas of uniform color blocks, and unique distortions such as moire patterns and screen reflections. The feature extraction mechanism designed for natural images often fails to effectively capture the key discriminative information of screen images, resulting in a decline in algorithm performance.
[0006] Second, many existing algorithms use a single scale feature extraction strategy, such as using only the deep or shallow features of a CNN. This approach is difficult to balance the global interface layout information (such as window arrangement) and local detail features (such as small text strokes and icon textures) of screen images at the same time. For possible minor tampering (such as modifying a small amount of text) or local content copying, a single scale feature may not be sensitive enough to meet the high-precision copyright detection requirements.
[0007] Therefore, there is an urgent need for a copyright protection method specifically designed for screen image characteristics that can effectively fuse multi-scale information and generate robust hash codes. SUMMARY
[0008] (1) Technical problems to be solved
[0009] In view of the problems in the prior art that the existing visual hash method applied to screen images has the problems of insufficient capture of screen-specific elements (text, graphics), inability to effectively distinguish uniform background interference, and difficulty in balancing global layout and local detail information, resulting in insensitivity to subtle tampering, the present application aims to provide a screen image copyright protection method based on visual saliency multi-scale hash to improve the accuracy and robustness of screen image copyright protection.
[0010] (2) Technical solutions
[0011] To solve the above technical problems, the technical scheme adopted by the present application is: a screen image copyright protection method based on visual saliency multi-scale hash, comprising the following steps:
[0012] Step 1: Visual saliency region detection.
[0013] An input screen image to be processed is input. A visual saliency model (SRD) is used to process the image. The model first calculates the edge density map and color sparsity map of the image respectively. Edge density calculation aims to quantify the structural complexity of local regions of the image, and high edge density usually corresponds to information regions such as text and icon boundaries. Color sparsity analysis is used to identify color regions that are relatively rare in the global color distribution, which may correspond to important UI elements (such as highlighted buttons and special icons). Then, a fusion module (such as a linear combination based on attention mechanism or learnable weights) is used to combine the edge density map and color sparsity map to generate a saliency weight map. The values of the weight map indicate the importance of each region of the image, and regions with higher values are considered to be visually salient regions, which help focus on key content and suppress the interference of smooth background regions in subsequent feature extraction.
[0014] Step 2: Multi-scale feature extraction and fusion.
[0015] The original screen image (or combined with the saliency weight map generated in the previous step) is input into a dynamic multi-scale feature extraction network (DFE). The network uses a dual-path or multi-path architecture.
[0016] Preferably, a detail path is included, which uses a smaller number of layers and smaller step size convolution layers (such as 3x3 convolution), mainly processes higher resolution inputs or feature maps, and aims to capture and preserve fine local details of the image, such as text strokes, small icon textures, etc., and outputs shallow detail features.
[0017] Preferably, one semantic path is included, which usually adopts a deeper, pre-trained convolutional neural network backbone (such as ResNet, VGG, etc.), extracts deep semantic information and global context information of the image (such as interface layout, element relative position, etc.) through multi-level down-sampling operations, and outputs deep semantic features.
[0018] Then, a cross-scale feature fusion mechanism is adopted. This mechanism effectively combines feature maps from different paths (detail path, semantic path) or different depth levels of the network.
[0019] Preferably, the fusion mechanism adopts cross-scale attention fusion. For example, the low-resolution semantic features are up-sampled to the same spatial resolution as the detail features, and then a module inspired by attention mechanisms (such as self-attention, deformable attention) (such as deformable attention gate DAG) is used to calculate spatial attention weights, dynamically determining whether to focus more on detail features or semantic features at each location according to local image content, and finally obtaining the fused multi-scale feature representation through weighted summation or concatenation, etc. The fusion feature contains rich local details and global structure information.
[0020] Step three: CP decomposition-based hash code generation.
[0021] The multi-scale fusion feature representation obtained in the previous step (may need to be arranged in shape through global average pooling, flattening, etc.) is constructed into a feature tensor. Then, the feature tensor is input into a CP (CANDECOMP / PARAFAC) tensor decomposition-based hash layer (CP-Hash Layer).
[0022] First, CP decomposition is performed on the feature tensor, which is approximated as the sum of R rank-1 tensors (i.e. R groups of factor vectors). CP decomposition can effectively capture the potential high-dimensional interaction information inside the feature tensor and compress the information into these low-dimensional factor vectors, and R is a pre-set decomposition rank.
[0023] Second, the factor vectors obtained by CP decomposition are processed. For example, the R groups of factor vectors can be concatenated along a certain dimension to form a longer vector.
[0024] Then, the processed vector is input into a mapping function, usually a small multi-layer perceptron (MLP), for nonlinear transformation and dimension reduction, and mapped to the length L (e.g. 64 bits, 128 bits, etc.) of the target hash code.
[0025] Finally, a binarization function is applied to the output of the MLP, most commonly the sign function, which maps values greater than or equal to 0 to +1 and values less than 0 to -1 (or 0 and 1), generating the final L-bit binary hash code.
[0026] Preferably, a composite loss function is used to optimize the neural network model corresponding to the method end-to-end when training the method. The loss function at least contains the following parts or their variants:
[0027] Similarity-preserving loss: aims to make the Hamming distance between the generated hash codes of content-similar screen image pairs as small as possible, and the Hamming distance between the hash codes of content-unsimilar image pairs as large as possible. Common forms include contrast loss, triplet loss, etc.
[0028] Binarization loss: acts on the continuous-valued output before the binarization function, encouraging these values to be far from 0 and closer to the target binary values (such as +1 and -1), in order to reduce the information loss in the binarization process.
[0029] Bit-balancing loss: encourages the distribution of each bit of the generated hash code over the entire dataset to be as balanced as possible (i.e. the mean is close to 0), in order to maximize the information entropy and discrimination ability of the hash code.
[0030] The network parameters are trained by minimizing the weighted sum of these loss terms.
[0031] The final binary hash code generated is the content fingerprint of the screen image, which can be used for efficient copyright comparison. For example, calculate the Hamming distance between the hash codes of two screen images, if the distance is less than a pre-set threshold, then they are determined to be similar in content or have a copy relationship.
[0032] (3) Advantages
[0033] Compared with the prior art, the present application has the following advantages:
[0034] 1. Strong targeting, effectively capturing key features of screen images: By introducing a visual saliency model based on edge density and color sparsity, the method can preferentially focus on information-rich areas such as text and icons in screen images, suppressing the interference of large-area uniform backgrounds, and improving the targeting and effectiveness of feature extraction.
[0035] 2. Balancing global and local, improving robustness: The dynamic multi-scale feature extraction and fusion architecture can capture both fine local details (such as text strokes) and overall layout structures (such as interface element arrangement) of screen images, making the generated hash code more sensitive to subtle content tampering while remaining robust to overall similarity.
[0036] 3. Efficient and discriminative hash code generation: By utilizing the hash layer based on CP tensor decomposition, not only can the feature dimension be effectively compressed and high-dimensional feature interactions be captured, but also compact and highly discriminative binary hash codes can be generated through end-to-end optimization of deep learning, which is suitable for fast retrieval and comparison of large-scale image libraries.
[0037] 4. Improved copyright protection accuracy: Based on the above advantages, the method can more accurately identify the copying and tampering behavior of screen images. Experiments on public datasets show that, compared with the baseline model, the average precision mean (mAP) and micro average precision (μAP) and other indicators are significantly improved, improving the overall performance of screen image copyright protection. BRIEF DESCRIPTION OF DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.
[0039] Figure 1 is a whole process schematic diagram of a screen image copyright protection method based on visual saliency multi-scale hashing provided by an embodiment of the present application.
[0040] Figure 2 is a structural schematic diagram of a visual saliency model (SRD) in an embodiment of the present application.
[0041] Figure 3 is a structural schematic diagram of a dynamic multi-scale feature extraction network (DFE) in an embodiment of the present application, which shows a detail path, a semantic path and a cross-scale fusion module.
[0042] Figure 4 is a structural schematic diagram of a hash layer based on CP tensor decomposition (CP-Hash Layer) in an embodiment of the present application. DETAILED DESCRIPTION
[0045] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.
[0046] Please refer to Figure 1 , Figure 1This paper illustrates the overall flow of a screen image copyright protection method based on visual saliency multi-scale hashing provided by an embodiment of the present invention. The method mainly includes three core stages: visual saliency region detection (module 100), multi-scale feature extraction and fusion (module 200), and hash code generation based on CP decomposition (module 300).
[0047] Phase 1: Visual Saliency Region Detection (Module 100)
[0048] Input a screen image I. The goal of this stage is to generate a saliency weight map S to guide subsequent feature extraction.
[0049] Reference Figure 2 The visual saliency model (SRD, internal structure of module 100) can be implemented as follows:
[0050] 1. Edge Density Calculation (Module 110): Apply an edge detection algorithm (such as the Canny operator) to the input image I to obtain the edge map E. Then, for each pixel location (x, y), calculate the edge pixel density in its neighborhood, for example, by weighted summation using a Gaussian kernel, to obtain the edge density map D. edge .
[0051] 2. Color Sparsity Analysis (Module 120): First, the color space of image I is quantized (e.g., clustered into K representative colors). Then, a global color histogram is plotted. For each quantized color c at pixel (x, y), a color sparsity score is calculated based on its global frequency.
[0052] (For example, the lower the frequency, the higher the score), thus obtaining the color sparsity map D. color .
[0053] 3. Saliency Map Fusion (Module 130): Fusion of the normalized edge density map D edge and color sparsity diagram D color The combination is achieved through a fusion function. For example, a linear combination S′=α·D is performed using learnable weights α and β. edge +β·D color and through
[0054] The sigmoid activation function yields the final saliency weight map S = sigmoid(S′). The weights α and β can be learned during model training.
[0055] Phase Two: Multi-scale Feature Extraction and Fusion (Module 200)
[0056] The original image I (or the image combined with the saliency map S) is input into the Dynamic Multiscale Feature Extraction Network (DFE, internal structure of module 200).
[0057] Referring to Figure 3 , the DFE can include:
[0058] 1. Detail Path (Module 210): composed of several layers of convolution (e.g. 3x3 convolution, small stride), processing high-resolution input, output shallow feature map F detail that preserves spatial details.
[0059] 2. Semantic Path (Module 220): using a deep CNN backbone network (e.g. ResNet-18), extracting deep semantic information of the image through multi-level convolution and down-sampling, output low-resolution feature map F semantic that contains global context.
[0060] 3. Cross-scale Feature Fusion (Module 230): first up-sample F semantic to the same spatial size as F detail . Then, use a cross-scale attention module (e.g. Deformable Attention Gate, DAG based on deformable convolution) to calculate an attention weight map A. This weight map indicates, at each spatial location, whether F detail or up-sampled F semantic is more important. The final fused feature F fused is obtained by weighted combination:
[0061] F fused = A⊙F detail + (1-A)⊙UpSample(F semantic ) (where ⊙ denotes element-wise multiplication).
[0062] Stage Three: Hash Code Generation Based on CP Decomposition (Module 300)
[0063] Send the fused feature F fused (maybe after global pooling, etc. operation becomes a vector or a fixed-size tensor) into the CP-decomposition-based hash layer (CP-Hash Layer, internal structure of Module 300).
[0064] Referring to Figure 4 :
[0065] 1. Feature Tensor Construction and CP Decomposition (Module 310): reshape F fused into a three-order or higher-order feature tensor T. Perform CP decomposition on T to approximate it as the sum of R rank-1 tensors: get R sets of factor vectors
[0066] 2. Hash Code Generation (Module 320): convert these factor vectors (e.g. convert all u r , vr , w r are concatenated and input into a small MLP. The MLP performs non-linear transformation and dimension reduction, outputting an L-dimensional continuous-valued vector h cont . Finally, the sign function H = sign(h cont ) is applied to binarize the continuous-valued vector into the final L-bit hash code H (elements are +1 or -1).
[0067] Training and application
[0068] The whole network model (including SRD, DFE, CP-HashLayer) is trained in an end-to-end manner. A dataset containing pairs of screen images (similarity / dissimilarity labels) is used. The loss function is usually a weighted sum of similarity-preserving loss quantization loss and bit-balancing loss : The network parameters are learned by minimizing the loss function using optimization algorithms such as gradient descent.
[0069] After training, for any input screen image, the method can quickly generate its corresponding binary hash code. In copyright protection applications, the hash codes of original copyrighted images can be stored. When a to-be-detected screen image is detected, its hash code is calculated, and the Hamming distance with the hash codes of the copyrighted images in the database is calculated. If the Hamming distance is less than a certain pre-set threshold, it is determined that the to-be-detected image is a copy or a tampered version.
[0070] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims
1. A screen image copyright protection method based on visual saliency multi-scale hashing, characterized in that, Includes the following steps: (a) Visual saliency region detection step: Input the screen image to be processed, and calculate the saliency weight map of the screen image through the visual saliency model (SRD). The visual saliency model integrates edge density calculation and color sparsity analysis to highlight key areas such as text and icons in the screen image. (b) Multi-scale feature extraction and fusion step: The screen image and / or its saliency weight map are input into the dynamic multi-scale feature extraction network (DFE). The network contains at least one detail path for extracting shallow detail features and one semantic path for extracting deep semantic features. Feature maps of different scales are obtained through parallel processing, and the features of different paths or different levels are adaptively aggregated using a cross-scale feature fusion mechanism to generate a fused feature representation. (c) Hash code generation step: The fused feature representation is constructed as a feature tensor. The feature tensor is processed using a CP-Hash Layer based on CP tensor decomposition. First, the feature information is compressed and high-dimensional interactions are captured by CP tensor decomposition technology. Then, the dimension is reduced by a mapping function (such as a multilayer perceptron). Finally, a binarization function (such as a sign function) is applied to generate the final binary hash code, which is used for subsequent screen image copyright detection and comparison.
2. The method according to claim 1, characterized in that: The visual saliency model in step (a) generates the saliency weight map by calculating the local edge density and global color sparsity of each pixel in the image and fusing the two using a learnable weight or attention mechanism.
3. The method according to claim 1, characterized in that: The Dynamic Multi-Scale Feature Extraction Network (DFE) in step (b) includes lightweight convolutional layers in its detail path to preserve high-resolution details, and employs a pre-trained backbone network (such as a convolutional neural network) to extract semantic information in its semantic path.
4. The method according to claim 1 or 3, characterized in that: The cross-scale feature fusion mechanism in step (b) employs cross-scale attention fusion or deformable attention gating (DAG), dynamically adjusting the fusion weights from detail path and semantic path features based on the feature content.
5. The method according to claim 1, characterized in that: In step (c), the hash layer based on CP tensor decomposition concatenates the multiple factor vectors obtained after CP decomposition and then processes them through the mapping function.
6. The method according to claim 1, characterized in that: The training process of the method is optimized using a composite loss function that includes similarity preservation loss, binary quantization loss, and bit balance loss.
7. The method according to any one of claims 1 to 6, characterized in that: The generated binary hash codes are stored, and copyright comparison or copy detection is performed by calculating the Hamming distance between the corresponding hash codes of different screen images.