An image invariant feature extraction system and method based on dynamic clustering learning

Abstract

The application provides an image invariant feature extraction system and method based on dynamic clustering learning, which comprises a primary image processing module, a cascade feature learning module and a feature coding module; the primary image processing module performs two-stage wavelet decomposition on an input image to obtain a first-stage low-frequency image and a second-stage low-frequency image; a first feature learning module in the cascade feature learning module takes the low-frequency image as input, performs wavelet decomposition, and obtains output by processing of a dynamic clustering unit, wherein the dynamic clustering unit is composed of a dynamic clustering space model, and a subsequent feature learning module takes the output of the previous module as input; the feature coding module maps the output of the last feature learning module to obtain an image invariant feature descriptor; the application is used for extracting an image descriptor with distinctiveness and robustness from an image, and judging the similarity of images according to the distance between image descriptors, and can be used for copy detection, image recognition and retrieval and the like.

Description

Technical Field

[0001] This invention relates to the fields of image processing technology and deep learning technology, and in particular to an image invariant feature extraction system and method based on dynamic clustering learning. Background Technology

[0002] With the continuous development of social media, a large number of users upload their own images to the internet every day. According to publicly available data from the Snapchat platform, billions of images are uploaded to its servers daily. [1] These images are frequently stolen, copied, and altered, severely infringing upon users' legitimate rights. Therefore, protecting the intellectual property rights of digital images has become a crucial issue that urgently needs to be addressed. To solve this problem, it is necessary to determine the similarity of user-uploaded images.

[0003] To determine the similarity between different images, it is usually necessary to first extract discriminative and robust invariant descriptors from the images, and then measure the similarity by the distance between the descriptors. [2] Early methods were mostly based on global or block-based statistics and frequency domain features: including calculating the mean or histogram of image blocks and constructing binary descriptors accordingly. These methods were robust to compression and slight color changes, but had limited robustness to large cropping, rotation, and affine transformations. [3] To enhance geometric invariance, locally invariant features are widely used: SURF [4] With ORB [5] Further improvements have been made in efficiency, and it is often combined with the visual bag-of-words framework to achieve large-scale similarity retrieval.

[0004] With the development of deep learning, deep hashing methods have achieved end-to-end feature learning, such as DPSH. [6] Pair-supervised joint optimization, HashNet [7] Stable training of signed function binary activations, and triplet-based supervision. [8] Several extensions of this approach have yielded superior retrieval performance compared to traditional handcrafted features and two-stage methods on datasets such as CIFAR-10, NUS-WIDE, and ImageNet. In recent years, hashing methods based on ViT have further improved the discriminativeness and robustness of hashing. [9] .

[0005] Overall, the core challenge facing existing image invariant descriptors / hashes remains "robustness and generalization": images transmitted over the network may undergo complex processing operations such as compression, resampling, cropping, smearing / graffiti, splicing, and watermarking, and out-of-distribution distortion will significantly weaken the robustness of the descriptors. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides an image invariant feature extraction system and method based on dynamic clustering learning. This invention utilizes a wavelet domain dynamic clustering state space model to enable image descriptors to characterize the essential visual features of both local and global images, exhibiting strong robustness. This invention can resist various image processing operations and also has strong discriminative power for images with different content.

[0007] The present invention is implemented using the following technical solution: An image invariant feature extraction system based on dynamic clustering learning, the system comprising a primary image processing module, a cascaded feature learning module, and a feature encoding module; wherein: The primary image processing module processes the input image. Perform two-stage wavelet decomposition, select the low-frequency subbands from both stages of wavelet decomposition, perform inverse wavelet transform, and scale the inverse transform results to match the original image. The same size was used to obtain the first-level low-frequency image. Second-level low-frequency image ; The first feature learning module in the cascaded feature learning module uses low-frequency images ( The previous module's output is used as the input for the current module; The feature encoding module will use the output of the last feature learning module. The mapping is performed to obtain image-invariant feature descriptors, where This represents the number of cascaded feature learning modules.

[0008] Furthermore, each level of the feature learning module includes a downsampling layer, a wavelet decomposition unit, a dynamic clustering unit, an attention gating unit, and a fusion unit; wherein: The downsampling layer performs spatial resolution compression on the input feature data to generate a downsampling result, denoted as . ; The wavelet decomposition unit Wavelet transform was performed to obtain the low-frequency feature maps. and high-frequency feature maps H； The dynamic clustering unit analyzes low-frequency feature maps. Clustering operations yield low-frequency clustering results ; The attention gating unit processes the input high-frequency feature map. Enhanced output high-frequency feature map ; The fusion unit performs low-frequency clustering results High-frequency feature map Perform a two-dimensional wavelet inverse transform to obtain the first... The output of each feature learning module .

[0009] Further, the dynamic clustering unit includes a pre-convolutional layer, a pooling layer, a dynamic clustering state space model, and a post-convolutional layer; the dynamic clustering state space model consists of clustering units, a state space model, and an upsampling layer; wherein: The pre-convolutional layer uses the low-frequency feature map As input, we obtain the convolution output. ; The pooling layer will output the convolution. by Divide the size and use average pooling to obtain There are cluster centers; wherein: the dynamic clustering state space model is... Each cluster center performs the following process: For each cluster center, collect data from its neighborhood. Cluster centers yield a set Then, the low-frequency feature map was processed. After partitioning and rearranging, we get Perform according to the following formula The dynamic clustering operation, in the... In the next iteration First, calculate the current cluster center. and correlation between ; in, Cluster center and The degree of correlation between them It is the number of cluster centers. It means Cluster centers in the neighborhood; Cluster centers are classified according to the following formula. The update will be performed as follows: in, It is the correlation degree of the normalized column; go through After the next iteration, the cluster centers will be... Global feature fusion is performed by inputting a dynamic clustering state space model; The output of the dynamic clustering state space model is upsampled and restored to generate a low-frequency subband. Feature maps of the same size; The upsampled and restored output is then input into a post-convolutional layer to obtain the low-frequency clustering result. .

[0010] This invention can also be implemented using the following technical solutions: An image invariant feature extraction method based on dynamic clustering learning includes the following steps: A perceptual image dataset is constructed using copied image pairs and irrelevant image pairs; The present invention provides an image invariant feature extraction system based on dynamic clustering learning, which inputs the original image and performs binarization processing to obtain an image invariant feature descriptor.

[0011] Beneficial effects The image descriptors extracted by this invention have strong robustness and can resist various image processing operations, while also having strong distinguishability between images with different content. The similarity of images can be determined based on the distance between image descriptors, and it can be applied to applications such as copy detection, image retrieval, and recognition. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the image invariant feature extraction system described in this invention. Detailed Implementation

[0013] The following is in conjunction with the appendix Figure 1 The present invention is described as follows: This embodiment proposes an image invariant feature extraction system and method based on dynamic clustering learning, as detailed below: like Figure 1 As shown, this invention provides an image invariant feature extraction system and method based on dynamic clustering learning. The system includes a primary image processing module, a cascaded feature learning module, and a feature encoding module. Each feature learning module includes a downsampling layer, a wavelet decomposition unit, a dynamic clustering unit, and an attention gating unit. The dynamic clustering unit includes a pre-convolutional layer, a pooling layer, a dynamic clustering state space model, and a post-convolutional layer. Wherein: The primary image processing module processes the input image through a two-stage wavelet transform. Decompose the image to obtain low-frequency images. ), and input it into the cascaded feature learning module; specifically: For the input image Perform two-stage wavelet decomposition, select the low-frequency subbands from both stages of wavelet decomposition, perform inverse wavelet transform, and scale the inverse transform results to match the original image. The same size was used to obtain the first-level low-frequency image. Second-level low-frequency image The first low-frequency image Second low frequency image The inputs are fed into a cascaded feature learning module. The first feature learning module uses low-frequency images (...) The previous module's output is used as input, and subsequent feature learning modules use the previous module's output as input. Each of the aforementioned feature learning modules includes a downsampling layer, a wavelet decomposition unit, a dynamic clustering unit, and an attention gating unit; specifically: With the first Taking a feature learning module as an example, the input is first spatially compressed using a downsampling layer, and the downsampling result is denoted as... After that The extraction of local and global features of the image is achieved sequentially through wavelet decomposition, dynamic clustering, and attention gating units: Wavelet decomposition unit pairs Wavelet transform was performed to obtain the low-frequency feature maps. and high-frequency feature maps H This invention does not impose specific restrictions on the type of wavelet.

[0014] The dynamic clustering unit includes a pre-convolutional layer, a pooling layer, a dynamic clustering state space model, and a post-convolutional layer. Specifically: Dynamic clustering units for low-frequency feature maps Clustering operations yield low-frequency clustering results ,include: The pre-convolutional layer uses the low-frequency feature map As input, we obtain the convolution output. ; Pooling layers will output convolutions. by Divide the size and use average pooling to obtain Cluster centers ( ); The cluster centers are input into the dynamic clustering state space model. The dynamic clustering state space model consists of clustering units, a state space model, and an upsampling layer, specifically: For each cluster center, collect data from its neighborhood. Cluster centers yield a set Then, the low-frequency feature map was processed. After partitioning and rearranging, we get Perform according to the following formula The dynamic clustering operation, in the... In the next iteration First, calculate the current cluster center. and correlation between ; in, Cluster center and The degree of correlation between them It is the number of cluster centers. It means Cluster centers in the neighborhood; Cluster centers are classified according to the following formula. The update will be performed as follows: in, It is the correlation degree of the normalized column; go through After the next iteration, the cluster centers will be... Global feature fusion is performed by inputting a dynamic clustering state space model; The output of the dynamic clustering state-space model is upsampled and restored to generate a low-frequency subband. Feature maps of the same size; The upsampled and restored output is then input into a post-convolutional layer to obtain the low-frequency clustering result. .

[0015] High-frequency feature maps obtained by attention-gated units from wavelet decomposition units For input ， Output enhanced high-frequency feature map This invention does not impose specific restrictions on the form of the attention gating unit.

[0016] Low-frequency clustering results With enhanced high-frequency features Perform a two-dimensional wavelet inverse transform to obtain the first... The output of each feature learning module .

[0017] The feature encoding module will take the output of the last feature learning module. Mapping is performed to obtain robust image feature descriptors, where N This refers to the number of cascaded feature learning modules. This invention does not impose specific restrictions on the form of the feature encoding modules.

[0018] Example 1 This invention utilizes copied image pairs and irrelevant image pairs to form a training set. Image invariant feature extraction systems, such as... Figure 1As shown, the primary image processing module decomposes the input image through two-stage wavelet transform to obtain low-frequency images. Each feature learning module includes a downsampling layer, a wavelet decomposition unit, a dynamic clustering unit, and an attention gating unit to achieve a joint representation of local and global image features. The joint representation is processed by the feature encoding module to output image descriptors. This invention is used to extract discriminative and robust image descriptors from images and to determine image similarity based on the distance between descriptors.

[0019] The specific steps for extracting image invariant features are as follows: 1) The primary image processing module processes the input image. Perform two-stage wavelet decomposition, select the low-frequency subbands from both stages of wavelet decomposition, perform inverse wavelet transform, and scale the inverse transform results to match the original image. The same size was used to obtain the first-level low-frequency image. Second-level low-frequency image The wavelet type selected in this embodiment is: .

[0020] The first low-frequency image Second low frequency image The inputs are fed into a cascaded feature learning module. The first feature learning module uses low-frequency images (...) The previous module's output is used as input, and subsequent feature learning modules use the previous module's output as input. Taking the first feature learning module as an example, the input is first compressed in terms of spatial resolution using a downsampling layer, and the downsampling result is denoted as... After that The extraction of local and global features of the image is achieved sequentially through wavelet decomposition, dynamic clustering, and attention gating units: Wavelet decomposition unit pairs Wavelet transform was performed to obtain the low-frequency feature maps. and high-frequency feature maps H ; Dynamic clustering units for low-frequency feature maps Clustering operations yield low-frequency clustering results ,include: The pre-convolutional layer uses the low-frequency feature map As input, we obtain the convolution output. ; Pooling layers will output convolutions. by Divide the size and use average pooling to obtain Cluster centers; The cluster centers are input into the dynamic clustering state space model. The dynamic clustering state space model consists of clustering units, a state space model, and an upsampling layer, specifically: For each cluster center, collect its The cluster centers of the neighborhood are obtained as a set. Then, the low-frequency feature map was processed. After partitioning and rearranging, we get Perform one dynamic clustering operation according to the following formula; in, Cluster center and The degree of correlation between them It is the number of cluster centers. It means Cluster centers in the neighborhood; Cluster centers are classified according to the following formula. The update will be performed as follows: in, It is the correlation degree of the normalized column; Cluster centers The dynamic clustering state space model is input for global feature fusion; Table 1 shows the specific network structure of the state space model in this embodiment.

[0021] Table 1. Network Structure of the State-Space Model

[0022] The output of the dynamic clustering state-space model is upsampled and restored to generate a low-frequency subband. Feature maps of the same size; The upsampled and restored output is then input into a post-convolutional layer to obtain the low-frequency clustering result. .

[0023] High-frequency feature maps obtained by attention-gated units from wavelet decomposition units For input ， Output enhanced high-frequency feature map Table 2 shows the specific network structure of the channel attention network in this embodiment.

[0024] Low-frequency clustering results With enhanced high-frequency features Perform a two-dimensional wavelet inverse transform to obtain the first... The output of each feature learning module .

[0025] Table 2 Attention Gating Unit Network Structure

[0026] The feature encoding module will take the output of the last feature learning module. Mapping is performed to obtain image-invariant feature descriptors, where 4 represents the number of cascaded feature learning modules. The specific parameters of the four feature learning modules in this implementation are shown in Table 3. These modules are then processed through fully connected layers. Perform feature encoding to obtain an image invariant feature descriptor with a preset length of 128.

[0027] Table 3 Specific parameters of the feature learning module in the embodiment

[0028] 2) The dataset selected in this embodiment is PIHD (Perceptual Image Hashing Dataset).

[12] It is a large-scale training / evaluation dataset built for the Perceptual Image Hashing (PIH) task, focusing on preserving the distortion of content in real-world applications while taking into account both robustness and discriminability.

[0029] Composition and size of the dataset: Original images: 1,200 in total; 1,000 of them are taken from each of the 1,000 categories in ImageNet-1K, and the remaining 200 are supplemented through a process of "random entity name - web crawling"; all images are preprocessed to a resolution of 224×224. Each original image serves as the center sample of a "batch".

[0030] Image copy pairs: Apply 48 content preservation distortions (35 single operations + 13 random combination operations) to each original image to generate one copy image, for a total of 48 images.

[0031] Content-independent image pairs: Each original image corresponds to 48 content-independent images, of which 24 are other images from the same semantic category as the original image, and the other 24 are random images with different semantics from the original image.

[0032] Total size: 97 images / batch × 1,200 batches = 116,400 images, divided into training / test / validation sets in a 7:2:1 ratio.

[0033] 3) The network was trained and tested using the PIHD dataset. The test results show that the accuracy of the present invention in the image copy detection task can reach 0.947 and the F1-score can reach 0.947. The present invention has good robustness to various distortion types and has high practical value.

Claims

1. An image invariant feature extraction system based on dynamic clustering learning, characterized in that: The system includes a primary image processing module, a cascaded feature learning module, and a feature encoding module; wherein: The primary image processing module processes the input image. Perform two-stage wavelet decomposition, select the low-frequency subbands from both stages of wavelet decomposition, perform inverse wavelet transform, and scale the inverse transform results to match the original image. The same size was used to obtain the first-level low-frequency image. Second-level low-frequency image ; The first feature learning module in the cascaded feature learning module uses low-frequency images ( The previous module's output is used as the input for the current module; The feature encoding module will use the output of the last feature learning module. The mapping is performed to obtain image-invariant feature descriptors, where This represents the number of cascaded feature learning modules.

2. The image invariant feature extraction system based on dynamic clustering learning according to claim 1, characterized in that, Each level of the feature learning module includes a downsampling layer, a wavelet decomposition unit, a dynamic clustering unit, an attention gating unit, and a fusion unit; wherein: The downsampling layer performs spatial resolution compression on the input feature data to generate a downsampling result, denoted as . ; The wavelet decomposition unit Wavelet transform was performed to obtain the low-frequency feature maps. and high-frequency feature maps H； The dynamic clustering unit analyzes low-frequency feature maps. Clustering operations yield low-frequency clustering results ; The attention gating unit processes the input high-frequency feature map. Enhanced output high-frequency feature map ; The fusion unit performs low-frequency clustering results High-frequency feature map Perform a two-dimensional wavelet inverse transform to obtain the first... The output of each feature learning module .

3. The image invariant feature extraction system based on dynamic clustering learning according to claim 2, characterized in that, The dynamic clustering unit includes a pre-convolutional layer, a pooling layer, a dynamic clustering state space model, and a post-convolutional layer; the dynamic clustering state space model consists of clustering units, a state space model, and an upsampling layer; wherein: The pre-convolutional layer uses the low-frequency feature map As input, we obtain the convolution output. ; The pooling layer will output the convolution. by Divide the size and use average pooling to obtain There are cluster centers; wherein: the dynamic clustering state space model is... Each cluster center performs the following process: For each cluster center, collect data from its neighborhood. Cluster centers yield a set Then, the low-frequency feature map was processed. After partitioning and rearranging, we get Perform according to the following formula The dynamic clustering operation, in the... In the next iteration First, calculate the current cluster center. and correlation between ; in, Cluster center and The degree of correlation between them It is the number of cluster centers. It means Cluster centers in the neighborhood; Cluster centers are classified according to the following formula. The update will be performed as follows: in, It is the correlation degree of the normalized column; go through After the next iteration, the cluster centers will be... Global feature fusion is performed by inputting a dynamic clustering state space model; The output of the dynamic clustering state space model is upsampled and restored to generate a low-frequency subband. Feature maps of the same size; The upsampled and restored output is then input into a post-convolutional layer to obtain the low-frequency clustering result. .

4. A method for extracting image invariant features based on dynamic clustering learning, characterized in that, Includes the following steps: A perceptual image dataset is constructed using copied image pairs and irrelevant image pairs; The method inputs the original image into the image invariant feature generation system optimized according to any one of claims 1-3, and performs binarization processing on the system output to obtain the image invariant feature descriptor.

Images