A fabric image robust retrieval method under a multi-view self-supervised product quantization strategy

Abstract

The application provides a fabric image robust retrieval method under a multi-view self-supervised product quantization strategy, a multi-view module is introduced, the module includes a view enhanced by an AutoAugment method, an adversarial sample view and an image high-frequency component view. The AutoAugment method can generate more diverse data changes, enabling the model to better learn different features and structures of fabric texture; fabric images usually have high complexity and diversity, and the addition of adversarial samples to model training can increase more noise and changes, which is one of the best ways to resist adversarial attacks; the image high-frequency component can make the edges, details and contrast of the fabric image clearer. Meanwhile, a multi-view cross quantization contrast loss function is designed, the code word and deep visual descriptor are learned together by comparing multiple views, which can effectively improve the robustness and generalization of the model. The application has good retrieval performance and high adversarial robustness.

Description

Technical Field

[0001] This invention belongs to the field of fabric image retrieval technology, specifically relating to a robust fabric image retrieval method under a multi-view self-supervised product quantization strategy. Background Technology

[0002] The textile industry holds a significant position globally. With the continuous advancement of technology in the information age and the ongoing development of the fashion and home furnishing markets, the demand for fabrics (such as lace and cloth) is constantly increasing. Fabrics are characterized by a rich variety of patterns and styles, complex structures, and large amounts of data. Accurately and quickly retrieving specific patterns or textures has become a major challenge in the design, management, and production processes of fabrics.

[0003] With the continuous advancement of digital image technology, researchers have begun to focus on how to utilize machine learning methods to achieve automated retrieval of fabric images. By extracting features and learning representations from fabric images, images can be transformed into computable feature vectors, thereby enabling similarity measurement and retrieval between fabric images. However, due to the complex and diverse textures of fabric images, as well as their unique properties such as scale variations, rotation transformations, and lighting changes, image retrieval presents certain challenges.

[0004] Due to the limited capabilities of traditional feature representation, research on image feature extraction based on convolutional neural network models has developed rapidly in the field of image retrieval in recent years. However, existing convolutional neural network models still have some problems in fabric image retrieval: (1) the feature vector dimension is large and the processing time is long; (2) the image annotation cost is high; (3) the model lacks robustness and is prone to failure due to adversarial attacks. This invention combines adversarial training with self-supervised product quantization and fully utilizes information from different views through a multi-view module to improve learning performance, and is applied to fabric image retrieval. This invention achieves good retrieval performance while also having high adversarial robustness. Summary of the Invention

[0005] This invention relates to the field of fabric image retrieval, specifically a robust fabric image retrieval method based on a multi-view self-supervised product quantization strategy. The method introduces a multi-view module, which includes views enhanced using the AutoAugment method, adversarial example views, and high-frequency component views. The AutoAugment method generates more diverse data variations, enabling the model to better learn different features and structures of fabric textures. Fabric images typically exhibit high complexity and diversity; incorporating adversarial examples into model training adds more noise and variation, representing one of the best existing methods for resisting adversarial attacks. High-frequency component views enhance the clarity of edges, details, and contrast in fabric images. A multi-view cross-quantization contrastive loss function is also designed, which compares multiple views to jointly learn codewords and depth visual descriptors, effectively improving the model's robustness and generalization. This invention achieves excellent retrieval performance while also exhibiting high adversarial robustness.

[0006] This invention proposes a deep image retrieval algorithm based on a multi-view self-supervised product quantization strategy, specifically targeting the characteristics of artificially generated fabric images. The method introduces a multi-view module, which includes views enhanced using the AutoAugment method, adversarial example views, and high-frequency component views. The AutoAugment method generates more diverse data variations, enabling the model to better learn different features and structures of fabric textures. Fabric images typically exhibit high complexity and diversity; incorporating adversarial examples into model training adds more noise and variation, representing one of the best existing methods for resisting adversarial attacks. High-frequency component views enhance the clarity of edges, details, and contrast in fabric images. A multi-view cross-quantization contrastive loss function is also designed, which compares multiple views to jointly learn codewords and deep visual descriptors, effectively improving the model's robustness and generalization. Experimental results on multiple datasets demonstrate the effectiveness of this method, significantly improving the robustness of the retrieval system compared to other advanced retrieval algorithms. This invention provides a novel solution for fabric image retrieval and is of significant importance for improving the performance of fabric image retrieval systems.

[0007] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0008] A robust fabric image retrieval method under a multi-view self-supervised product quantization strategy, characterized by the following steps:

[0009] Step S1: Create a dataset: Scan the lace samples into images, process them, and obtain one image for each type of lace sample; then divide the dataset into a training set and a test set.

[0010] Step S2: Construct a multi-view self-supervised product quantization network, including a multi-view module, a feature extractor, and a quantization head; the multi-view module includes: two views enhanced using the AutoAugment method, an adversarial example view, and a high-frequency component view of the image, for a total of four views; the feature extractor uses ResNet18 as the backbone network to extract the input image into a depth descriptor; the codebook in the quantization head divides the high-dimensional depth feature space into a Cartesian product of multiple subspaces, converting the depth descriptor into a quantization descriptor represented by codewords;

[0011] Step S3: Construct a multi-view cross-quantization contrastive loss function, and learn codewords and depth visual descriptors by comparing the four views in the multi-view module; when the views come from the same image, their depth descriptors and product quantization descriptors are considered correlated, while if the views come from different images, they are considered uncorrelated;

[0012] Step S4: Train the neural network constructed in step S2 using the training set established in step S1, and use the loss function in step S3 during the training process;

[0013] Step S5: After training is complete, use the trained feature extractor and codebook to build an image retrieval database, and convert all images in the dataset into binary codes to build a binary code retrieval database.

[0014] Step S6: When querying the image to be retrieved, use a feature extractor to extract the feature vector of the image to be retrieved, use Euclidean distance to calculate the similarity between the feature vector subvector and the codeword in the codebook, and output the n images with the highest similarity to the image to be retrieved as the retrieval results.

[0015] Further, in step S1, all lace samples are scanned as images. During scanning, light-colored lace is laid flat on black cardstock, while dark-colored samples are placed against a white cardstock background. The process of building the dataset includes the following steps:

[0016] Step S11: Scan each of the different lace samples to obtain one image of each type of lace sample;

[0017] Step S12: Rotate and crop the image so that the edge of the lace trim is parallel to the image border;

[0018] Step S13: Image scaling; To reduce computational load and to meet the input size requirements of the backbone convolutional neural network ResNet50, the image is scaled down to 224*224 pixels.

[0019] Further, in step S2, the multi-view self-supervised product quantization network specifically comprises:

[0020] For an input image x, four views are generated: two views enhanced using the AutoAugment method, an adversarial example view, and a high-frequency component view. The AutoAugment method uses an enhancement strategy trained on the ImageNet dataset. For the adversarial example view, an enhancement strategy is trained using... A constrained 10-step projective gradient descent algorithm generates adversarial perturbations, where ε = 8 / 255 controls the attack size and the step size is 2 / 255; for high-frequency component views, the perturbations are calculated using the following formula:

[0021]

[0022] in, and Represents the Fourier transform and its inverse transform; q h and q l The distinction is made through a hard threshold operation, with the threshold defined as 8;

[0023] The ResNet50 is used as the backbone feature extractor network to extract features for each input view; its projection head is a multilayer perceptron: the first fully connected layer maps the features to 512 dimensions, followed by a ReLU layer, and the second FC layer outputs a D-dimensional feature vector.

[0024] The quantization head reconstructs a D-dimensional feature vector into a D-dimensional quantized representation; the quantization head Q uses M codebooks, i.e. Among them, codebook C i Composed of K codewords: C m ={c m1 ,…,c mK}, c∈R DM Using a soft quantizer in the quantizer q m (·):

[0025]

[0026] Where τ q This is a non-negative temperature parameter used to measure the parameters of softmax. The squared Euclidean distance is used to measure the similarity between two inputs.

[0027] Furthermore, the loss function in step S3 specifically includes the following design:

[0028] (1) The loss function used is the multi-view contrast loss function:

[0029]

[0030] Where τ i(x) represents the i-th view of image x, and P(i) represents the view excluding τ. i The m-1 front views other than point x, N(i) represents the set of m augmented views of the b-1 batch samples other than point x, with a quantity of m(b-1);

[0031] (2) Two views augmented using the AutoAugment data augmentation method, plus adversarial example x adv and high-frequency component x H This yields a definition in (τ1(x), τ2(x), x adv ,x H Four-view cross-quantization adversarial contrast loss:

[0032]

[0033] Where ε>0 is the maximum perturbation size during adversarial training, θ F and θ Q These are the trainable parameters in the model;

[0034] (3) The objective function throughout the training process is:

[0035]

[0036] Furthermore, the training process in step S4 is as follows:

[0037] Step S41: Input the training set into the neural network in batches for training, and generate four views for each training image;

[0038] Step S42: Use the feature extractor to extract depth descriptors for each view, and the quantization head maps each depth descriptor to a quantization descriptor;

[0039] Step S43: Calculate the loss value of the current batch of data using the multi-view cross-quantization comparison loss function in step S3;

[0040] Step S44: Backpropagation of gradients, using the Adam optimizer to optimize network parameters θ F and θ Q .

[0041] Furthermore, in step S5, the retrieval database is established as follows:

[0042] Step S51: For each image x in the dataset n x is obtained through the trained feature extractor F. n depth descriptor Divide it into M equal-length subvectors.

[0043] Step S52: Calculate each sub-vector x separately. nm With codebook C m The squared Euclidean distance of each codeword in the search is used to find the subvector x. nm The closest code;

[0044] Step S53: Find the nearest neighbor codeword k * The index format is converted into binary code to generate a sub-binary code. Concatenate all the sub-binary codes to generate an M·log2(K) bit binary code.

[0045] Step S54: Repeat steps S51-S53 for all images in the image library to build a binary-coded retrieval database.

[0046] Furthermore, the image retrieval steps in step S6 are as follows:

[0047] Step S61: When a query image x is input... q When performing a retrieval, the same feature extractor F is used for x. q Feature extraction was performed to obtain Then use the same subvector partitioning process to... Divided into

[0048] Step S62: Using Euclidean distance, calculate the similarity between the sub-vectors of the query image and each codeword in all codebooks, and sum the distances of these sub-vectors to obtain the asymmetric distance between the query image and each database image.

[0049] Step S63: Based on the similarity calculation results, sort the candidate images, place the image with the highest similarity at the top, and return the first n images similar to the input image as the retrieval results as required.

[0050] Compared to existing technologies, this invention and its preferred embodiment propose a novel unsupervised fabric image retrieval method based on multi-view self-supervised product quantization. This method employs a product quantization strategy and is trained in a self-supervised manner. To improve the model's adversarial robustness and generalization ability, a multi-view module is introduced. This module incorporates AutoAugment, a data augmentation scheme more suitable for fabric images, and uses adversarial examples and high-frequency components of the image to create robustly perceptive and generalized perceptive unlabeled data views, establishing a fast retrieval method based on multi-view cross-quantization contrastive learning. Extensive experiments were conducted on two fabric datasets and one commonly used image retrieval dataset to demonstrate the effectiveness of the method compared to other state-of-the-art methods. Attached Figure Description

[0051] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0052] Figure 1 This is an example diagram illustrating how small perturbations in an image can have a profound impact on search results.

[0053] Figure 2 This is a schematic diagram of the MSPQ training process according to an embodiment of the present invention.

[0054] Figure 3 These are sample images from the test dataset.

[0055] Figure 4 This is a comparison chart of the retrieval mAP (%) for 19 different damage types on the lace dataset in an embodiment of the present invention.

[0056] Figure 5 This is a comparison of the method proposed in this embodiment of the invention with the retrieval results (32 bits) of SPQ on some adversarial query images on the lace dataset. The correct retrieval results are marked with green boxes. Detailed Implementation

[0057] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0058] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0059] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0060] Figure 1This paper presents an illustrative example illustrating how a lack of robustness can affect fabric image retrieval results. First, a self-supervised Product Quantization (SPQ) network was trained on the SJL dataset for large lace trims. SPQ is a state-of-the-art, fully unsupervised image retrieval method based on product quantization, using ResNet18 as the backbone to extract image features. It can be seen that when the query image is clean, SPQ can successfully retrieve images related to the query image from the sample database. However, when an attacker adds adversarial perturbations to the query image that are almost imperceptible to the human eye, SPQ will produce retrieval results completely irrelevant to the query image, causing the retrieval system to fail.

[0061] like Figure 1 The image illustrates an example of how small perturbations on an image can have a profound impact on retrieval results. By adding an adversarial perturbation of scale ε = 8 / 255 to the query image, the positive images are ranked much lower than before, although the differences between the perturbed images are difficult to detect. This perturbation is generated using projective gradient descent.

[0062] To address this, this invention proposes a multi-view self-supervised product quantization network, thereby constructing a fully unsupervised fabric image retrieval method independent of data labels. The robustness and generalization of the self-supervised product quantization algorithm are improved through data augmentation and adversarial learning. SPQ is used as the main image retrieval model, and a multi-view module is added on top of it. Multi-view learning can fully utilize information from different views to improve learning performance. The module introduces AutoAugment image augmentation technology as a more suitable augmentation scheme for fabric images. It also uses adversarial examples to create robust perceptual views, and high-frequency components of the image as unlabeled data views for generalization perception, establishing a fast multi-view contrastive learning retrieval method. AutoAugment is a data augmentation strategy based on neural architecture search optimization, which can achieve the best robustness while maintaining the accuracy of clean images. This is because the AutoAugment method can cover more damage types in the frequency domain, thereby improving the model's generalization ability. Training adversarial examples and normal samples together is an effective regularization that can improve the model's accuracy while effectively reducing the success rate of adversarial example attacks. Artificially generated fabric images, especially lace images, are distinguished primarily by their texture features. High-frequency components refer to areas of dramatic grayscale or color change in an image. Therefore, extracting high-frequency components for training can enhance texture details, enabling CNNs to capture more edge, detail, and contrast features in fabric images, further improving model performance. Furthermore, research has found that the relationship between high-frequency components and the spectrum of image data influences the generalization behavior of CNNs. In summary, this multi-view approach enhances the model's generalization ability and robustness to adversarial perturbations added to the query image, minimizing the impact of perturbations on the model's retrieval results.

[0063] This invention proposes a multi-view self-supervised product quantization method, which for the first time applies product quantization to the field of fabric image retrieval. Using SPQ as the baseline model, it introduces a multi-view module, combining adversarial training with self-supervised product quantization. Unlike traditional two-view contrastive learning methods, multi-view self-supervised product quantization can better explore multiple perspectives of the image and obtain more information. This makes it not only more suitable for fabric image retrieval tasks but also provides the model with better robustness and generalization.

[0064] The key idea behind MSPQ is to improve the robustness of the model by combining multi-view contrastive learning with adversarial training. While this idea originated from a NeurIPS paper (Qiu, Z., Su, Q., Ou, Z., Yu, J., and Chen, C., “Unsupervised hashing with contrastive information bottleneck,” arXiv preprint arXiv:2105.06138, (2021).), that paper focused on classification tasks. This invention innovatively applies it to ranking tasks, using MSPQ to combine adversarial training with self-supervised contrastive learning. This eliminates the need for image label information, significantly reducing manual annotation costs and making a substantial contribution to improving the robustness and versatility of fabric image retrieval systems.

[0065] The following details the multi-view self-supervised product quantization strategy for robust fabric image retrieval proposed in this invention:

[0066] This invention integrates multi-view components into a baseline model and proposes a novel multi-view adversarial contrastive learning framework based on product quantization. Figure 2 As shown, a multi-view cross-quantization loss function is given. By optimizing this loss function, a more robust retrieval model can be obtained.

[0067] Figure 2 This section describes the MSPQ training process. AutoAugment data augmentation is applied to image x to generate two transformed images. Adversarial examples and high-frequency components are also generated for x, resulting in four views of x. The feature extractor F extracts depth descriptors from each of the four views, and the quantization head Q converts these four depth descriptors into four product quantization descriptors. The cross-similarity between each depth descriptor and the product quantization descriptors is considered.

[0068] This invention employs SPQ, a model combining self-supervised learning and product quantization, as a baseline for deep, fully unsupervised image retrieval. For each input sample x, random sampling data augmentation is used to transform it into two different views τ1(x) and τ2(x). Features for each view are extracted using a feature extractor F, and then the features are reconstructed into a quantized representation using a quantization head Q.

[0069] During the quantization phase of SPQ, at the quantization head M codebooks are used, that is Where C i Composed of K codewords: C m ={c m1 ,…,c mK}, c∈RD / M PQ is a Cartesian product of a high-dimensional deep feature space divided into multiple subspaces in Q. Each codebook corresponding to a subspace represents several salient features representing an image dataset X. Each codeword in the codebook signifies a cluster center for different deep descriptors, with the aim of recording a frequently occurring local pattern. During quantization, similar attributes between images are shared by assigning the same codeword, while distinguishable features have different codewords.

[0070] Introduce soft quantization into the quantizer head, using the soft quantizer head q. m (·):

[0071]

[0072] Where τ q This is a non-negative temperature parameter used to measure the parameters of softmax. The squared Euclidean distance is used to measure the similarity between two inputs. In this way, the subquantized descriptor z nm =q m (x nm ;τ q C m This can be considered as belonging to C. m The output is an exponentially weighted sum of codewords. All codewords in the codebook are used to approximate the quantized output, with the codeword with the smallest distance contributing the most.

[0073] To learn both depth descriptors and codewords simultaneously, SPQ employs a cross-quantization contrastive learning scheme. If the views come from the same image, the depth descriptor and the product-quantized descriptor are considered correlated; otherwise, they are uncorrelated. The cross-quantization contrastive loss function for a pair of positive samples (τ1(x), τ2(x)) with respect to x is shown below:

[0074]

[0075] Where P(i) represents the front view excluding i, and N(i) represents the negative sample of point i; in order to reduce similarity and The redundancy between them means that the loss only calculates half of the negative samples in a batch, that is, N(i) includes (N... B -1) negative samples. Depth descriptor and quantization descriptors Cosine similarity between them, τ cqc It is a non-negative temperature parameter. Cosine similarity is used as a distance metric to avoid... and Norm deviation between them.

[0076] The following introduces the multi-view component: 1. The role of the AutoAugment data augmentation method.

[0077] Existing research indicates that data augmentation is crucial for training a high-quality unsupervised contrastive learning model; furthermore, data augmentation can effectively prevent robust overfitting during adversarial training.

[21] When enhancing the original image, SPQ adopts the data augmentation method from SimCLR. This image enhancement method emphasizes the importance of color distortion; however, it is mainly suitable for natural images. The acquisition process for fabric images, especially lace images, differs from that of natural images. This process follows certain norms and standards, and the background and lighting conditions are basically the same across different fabric images. Therefore, simple color distortion alone is no longer sufficient. Compared to the original SPQ, this invention uses AutoAugment as the image enhancement scheme.

[0078] AutoAugment is currently the most advanced augmentation scheme for standard cross-entropy training. Compared with traditional data augmentation methods, it is more diverse, including the basic augmentation methods commonly used in existing AT methods, as well as the Cutout operation. AutoAugment strategies trained on the ImageNet dataset mainly include color-based transformations, such as color inversion, contrast enhancement, and brightness adjustment, which can change the hue, saturation, and brightness attributes of an image, thereby increasing image contrast and sharpness while preserving the shape and texture of objects in the image. Simultaneously, this strategy also includes geometric transformations, such as rotation, translation, and cropping, which can change the position or orientation of objects in the image, thereby increasing image diversity and the number of challenging samples. In the experiments of this invention, the AutoAugment strategy found on the ImageNet dataset is used as the data augmentation strategy.

[0079] 2 Adversarial Example Views

[0080] In image retrieval tasks, generating an adversarial query image for a target retrieval system typically considers the following problem: given a clean query image x, the goal of generating the adversarial query image is to generate corresponding adversarial samples x by adding adversarial perturbations δ. adv =x + δ, and the final retrieval result returned by the deep network based on product quantization is semantically equivalent to x. q Irrelevant. Let F be a feature-dependent transformation function whose output contains the semantic information of the original features; let d(·,·) be used as the measure of F(x) and F(x). adv The similarity measurement function between (x, y, y) is used. Therefore, the core idea of ​​generating adversarial query images is to expand the adversarial semantic features F(x, y, y).adv The distance between the clean semantic feature F(x) and the clean semantic feature F(x) is defined by the metric function d.

[0081] Therefore, the problem of generating adversarial queries can be expressed as:

[0082]

[0083] st||δ||≤ε

[0084] Where d(·,·) is a similarity measure function, such as Euclidean distance; ||·|| is specified as an arbitrary norm, such as... or The norm, and ε controls the size of the attack.

[0085] If we use the cross-quantization contrastive loss function as a metric for x and x adv The similarity index between them is set as ||·||. If the norm is given, then the objective function (3) can be further expressed as:

[0086]

[0087] st||δ|| ∞ ≤ε

[0088] x adv Consider it as the third view of x.

[0089] 3 High-frequency component views

[0090] The solution provided in this invention also uses the high-frequency component (HFC) of the data as another additional comparative view. Existing research shows that HFC can enhance the texture details and edge parts of fabric images; moreover, the HFC data learned by CNNs is a major reason for achieving better generalization; secondly, when adversaries manipulate malicious samples to deceive model decisions, they usually focus on HFC. and This represents the Fourier transform and its inverse transform. Input image x = {x} L ,x H} can be decomposed into high-frequency components (HFC) x H and low-frequency components (LFC) x L :

[0091]

[0092] z L and z HThis is achieved through hard thresholding. Let z(i,j) represent the (i,j)th element of z, and c = (c1,c2) ​​represent the center point of the spectrum. By calculating the distance between the center point and c, the components z in the formula are filtered out. L and z H If d((i,j),(c i ,c j If z ≤ r, then z L (i,j)=z(i,j), z H (i,j) = 0; otherwise z L (i,j)=0,z H (i,j)=z(i,j). Where d(·,·) is the Euclidean distance between the two spatial coordinates, and r is a predefined distance threshold (r=8 in the experiment of the example).

[0093] Based on the above design, this invention employs multi-view cross-quantization to counteract contrast loss.

[0094] Based on the CrossQuantized Contrastive Learning (CQC) loss function used in SPQ self-supervised training mentioned above, this invention further extends it to a multi-view contrastive loss function.

[0095]

[0096] Where P(i) represents the value of τ. i The m-1 front views other than point x, and N(i) represents the set of m augmented views of the b-1 batch samples other than point x, with a quantity of m(b-1).

[0097] Two views enhanced with AutoAugment, plus adversarial examples and high-frequency components x separated from the original data x. H We obtained the definition in (τ1(x),τ2(x),x) adv ,x H Four-view cross-quantization adversarial contrast loss:

[0098]

[0099] Where ε>0 is the maximum perturbation size during adversarial training, θ F and θ Q These are the trainable parameters in the model. Therefore, the objective function throughout the training process is:

[0100]

[0101] Model optimization parameter θF and θ Q As the minimum participant, the resistance to disturbance δ=||xx adv || ∞ For the largest participant, the worst-case perturbation δ is obtained by maximizing the objective function, which maximizes the change in CQC loss on a single example x; the model parameters θ are obtained by minimizing the objective function. F θ Q This enables the model to resist adversarial perturbations on instance x. During training, the participants alternate between minimax games, minimizing and maximizing the perturbations until convergence.

[0102] This embodiment briefly describes the training process of the proposed multi-view self-supervised product quantization in Algorithm 1, where θ F and θ Q Let N and γ represent the trainable parameters of the feature extractor and quantization head, respectively, and γ be the learning rate. During training, the multi-view cross-quantization contrastive loss function with adversarial perturbation (Equation 8) from section 3.2 is used for training by comparing the depth descriptors and product quantization descriptors of different views. For a given mini-batch N... B Randomly select N from the database B The samples were augmented twice randomly using AutoAugment technology, and adversarial examples and high-frequency components were generated for each image, resulting in a total of 4N samples. B A data view. Consider the same image. The four independent views are related, while the other 4 (N) B -1) The views are unrelated and come from different images within a small batch.

[0103]

[0104]

[0105] The retrieval section follows the method of Product Quantization (PQ), mainly consisting of two steps: establishing an image retrieval database and querying the images to be retrieved. First, the retrieval database consists of a dataset... The binary encoding is composed of each image x in the dataset. n x is obtained through the feature extractor F trained in Section 3. n depth descriptor Divide it into M equal-length subvectors. Then calculate each subvector x separately. nm With codebook C m The squared Euclidean distance of each codeword is used to search for the subvector x. nmThe closest codeword. Next, the nearest neighbor codeword k. * The index format is converted into binary code to generate a sub-binary code. Finally, all the sub-binary codes are concatenated to generate an M·log2(K) bit binary code. Repeat this process for all images in the image library to build a binary-coded retrieval database.

[0106] During the query phase, when a query image x is input... q When performing image retrieval, the same F is used for x. q Feature extraction was performed to obtain Then use the same subvector partitioning process to... Divided into Using Euclidean distance, the similarity between the subvector of the query image and each codeword in all codebooks is calculated to construct a pre-computed lookup table. The distance calculation between the query image and the images in the library is an asymmetric approximation, which is accelerated by aggregating the lookup results. The image retrieval process is briefly described in Algorithm 2.

[0107]

[0108] Since MSPQ and SPQ use the same prediction model (the difference lies in the training loss), their prediction models have the same time complexity. To better express the time complexity during training, let O(n) be the time complexity. f The time complexity of gradient forward propagation is O. b For the time complexity of backpropagation, O u The time complexity for updating parameters. To compute adversarial perturbations, an additional time cost of O(n log n) is required. adv This is used to obtain gradients on content features. Therefore, the time complexity of the SPQ training process is O(n log n). f +O b +O u MSPQ, due to the addition of a multi-view module, has a time complexity of O(n log n).

[0109] 6×O f +6×O b +O adv +O u Typically, these four operations are linearly dependent, so the complexity of MSPQ is approximately 6 times that of SPQ.

[0110] Although the training time required per epoch increases during MSPQ training, the number of training epochs required for model convergence decreases. Empirically, the training time for MSPQ on the Lace dataset is roughly equivalent to that of SPQ, while on the Fabric dataset, the training time for MSPQ is approximately four times that of SPQ.

[0111] Based on the above design, the basic design scheme of the method of the present invention is formed as follows:

[0112] A robust fabric image retrieval method under a multi-view self-supervised product quantization strategy includes the following steps:

[0113] Step S1: Create the dataset: Scan all lace samples into images using a scanner. During scanning, lay light-colored lace flat on black cardstock, while using white cardstock as the background for dark-colored samples. To prevent distortion or interference from lighting conditions, maintain a constant external environment during the scanning process. Crops the images to obtain one image for each type of lace sample. Divide the dataset into training and test sets at an 80% to 20% ratio.

[0114] Step S2: Construct a multi-view self-supervised product quantization network, including a multi-view module, a feature extractor, and a quantization head. The multi-view module includes two views enhanced using the AutoAugment method, an adversarial example view, and a high-frequency component view of the image, for a total of four views. The feature extractor uses ResNet50 as the backbone network to extract depth descriptors from the input image; the codebook in the quantization head divides the high-dimensional depth feature space into Cartesian products of multiple subspaces, converting the depth descriptors into quantization descriptors represented by codewords.

[0115] Step S3: Construct the loss function during training. A multi-view cross-quantization contrastive loss function is constructed, which learns codewords and depth visual descriptors jointly by comparing the four views in the multi-view module. When views come from the same image, their depth descriptors and product quantization descriptors are considered correlated; otherwise, they are uncorrelated.

[0116] Step S4: Train the neural network constructed in step S2 using the training set established in step S1, using the loss function from step S3 during the training process.

[0117] Step S5: After training is complete, use the trained feature extractor and codebook to build an image retrieval database, and convert all images in the dataset into binary codes to build a binary code retrieval database.

[0118] Step S6: When querying the image to be retrieved, use a feature extractor to extract the feature vector of the image to be retrieved, use Euclidean distance to calculate the similarity between the feature vector subvector and the codeword in the codebook, and output the n images with the highest similarity to the image to be retrieved as the retrieval results.

[0119] The process of creating the dataset in step S1 includes the following steps:

[0120] Step S11: Scan each of the different lace samples to obtain one image of each type of lace sample.

[0121] Step S12: Rotate and crop the image so that the edge of the lace trim is parallel to the image border.

[0122] Step S13: Image scaling. To reduce computational load and meet the input size requirements of the ResNet50 backbone convolutional neural network, the image is scaled down to 224*224 pixels.

[0123] The network structure for step S2 is as follows:

[0124] For an input image x, four views are generated: two views enhanced using the AutoAugment method, an adversarial example view, and a view of the image's high-frequency components. The AutoAugment method uses an enhancement policy trained on the ImageNet dataset. For the adversarial example view, [the following is a list of views, likely related to image enhancement and its application]. A constrained 10-step projective gradient descent algorithm is used to generate adversarial perturbations, where ε = 8 / 255 controls the attack size and the step size is 2 / 255. For the high-frequency component view, the perturbations are calculated using the following formula:

[0125]

[0126] in, and This represents the Fourier transform and its inverse transform. q h and q l The distinction is made using a hard threshold operation, where the threshold is defined as 8.

[0127] The ResNet50 is used as the backbone feature extractor network to extract features for each input view. Its projector head is a multilayer perceptron, where the first fully connected layer maps the features to 512 dimensions, followed by a ReLU layer, and the second fully connected layer outputs a D-dimensional feature vector.

[0128] The quantization head reconstructs a D-dimensional feature vector into a D-dimensional quantized representation. The quantization head Q uses M codebooks, i.e. Among them, codebook C i Composed of K codewords: C m ={c m1 ,...,c mK}, c∈R D / M Using a soft quantizer in the quantizer head. m (·):

[0129]

[0130] Where τ qThis is a non-negative temperature parameter used to measure the parameters of softmax. The squared Euclidean distance is used to measure the similarity between two inputs.

[0131] The loss function in step S3 has the following characteristics:

[0132] (1) The loss function used is the multi-view contrast loss function:

[0133]

[0134] Where τ i (x) represents the i-th view of image x, and P(i) represents the view excluding τ. i The m-1 front views other than point x, and N(i) represents the set of m augmented views of the b-1 batch samples other than point x, with a quantity of m(b-1).

[0135] (2) Two views augmented using the AutoAugment data augmentation method, plus adversarial example x adv and high-frequency component x H This yields a definition in (τ1(x), τ2(x), x adv ,x H Four-view cross-quantization adversarial contrast loss:

[0136]

[0137] Where ε>0 is the maximum perturbation size during adversarial training, θ F and θ Q These are the trainable parameters in the model.

[0138] (3) The objective function throughout the training process is:

[0139]

[0140] The training process in step S4 is as follows:

[0141] Step S41: Input the training set into the neural network in batches for training, and generate four views for each training image.

[0142] Step S42: Use the feature extractor to extract depth descriptors for each view, and the quantization head maps each depth descriptor to a quantization descriptor.

[0143] Step S43: Calculate the loss value of the current batch of data using the multi-view cross-quantization comparison loss function in step S3.

[0144] Step S44: Backpropagation of gradients, using the Adam optimizer to optimize network parameters θ Fand θ Q .

[0145] In step S5, the retrieval database is established as follows:

[0146] Step S51: For each image x in the dataset n x is obtained through the trained feature extractor F. n depth descriptor Divide it into M equal-length subvectors.

[0147] Step S52: Calculate each sub-vector x separately. nm With codebook C m The squared Euclidean distance of each codeword is used to search for the subvector x. nm The closest code.

[0148] Step S53: Find the nearest neighbor codeword k * The index format is converted into binary code to generate a sub-binary code. Concatenate all the sub-binary codes to generate an M·log2(K) bit binary code.

[0149] Step S54: Repeat steps S51-S53 for all images in the image library to build a binary-coded retrieval database.

[0150] The steps for querying and retrieving images in step S6 are as follows:

[0151] Step S61: When a query image x is input... q When performing a search, the same F is used for x. q Feature extraction was performed to obtain Then use the same subvector partitioning process to... Divided into

[0152] Step S62: Using Euclidean distance, calculate the similarity between the sub-vectors of the query image and each codeword in all codebooks. Sum the distances of these sub-vectors separately to obtain the asymmetric distance between the query image and each database image.

[0153] Step S63: Based on the similarity calculation results, sort the candidate images, place the image with the highest similarity at the top, and return the first n images similar to the input image as the retrieval results as required.

[0154] The effectiveness of the design scheme of this invention is verified through a specific experiment: 1 Dataset

[0155] Lace Dataset: The data comes from a lace trading platform and contains 1250 types of lace fabrics. Each type of lace fabric was scanned to obtain one lace edge image, with varying image resolutions. 1000 images were randomly selected as the training set, and the remaining 250 were used as the test query set. All 1250 images were used to construct the retrieval database.

[0156] Figure 3 Some sample images from the dataset. The first row is from the Lace dataset, and the second row is from the Fabric 1.0 dataset.

[0157] Fabric 1.0: The dataset was collected from a large textile factory and contains 972 fabric patterns. Each fabric pattern has 24 images of the front and back sides from different angles, totaling 46,656 images of 256×256 pixels. Four images were selected for each fabric pattern: two front and two back sides, for a total of 3,888 images, as the retrieval database. One image was randomly selected from the four images for each fabric pattern as the test set, and the remaining three images were used as the training set, resulting in 2,916 training images and 972 test images.

[0158] Cifar10: Contains 60,000 images of size 32×32 across 10 class labels, with 6,000 images per class. 5,000 images per class are selected as the training set, and 100 images per class are selected as the query set. The entire training set of 50,000 images is used to construct the retrieval database.

[0159] Table 1. Composition of the three benchmark datasets

[0160]

[0161]

[0162] 2 Experimental Setup

[0163] Evaluation indicators

[0164] Mean Average Precision (mAP) and top-k Recall (R@k) were selected as evaluation metrics. mAP is the most commonly used evaluation metric in image retrieval, reflecting overall performance when measuring the retrieval ability of an algorithm. Precision is defined as the percentage of correctly retrieved images out of the total number of retrieved images. Recall is another performance metric used in image retrieval, calculated as the percentage of correctly retrieved images out of the total number of relevant images in the dataset. The mAP score calculated on all three datasets was mAP@1000.

[0165] Implementation details

[0166] Baseline methods are divided into two categories for comparison: (1) Deeply pre-trained unsupervised methods. These use a VGG16 model pre-trained on the ImageNet dataset as the backbone network and fine-tunes the model to generate encodings through unsupervised learning, such as CIBHash and MBE. (2) Deeply fully unsupervised methods. These jointly optimize visual features and encodings through unsupervised learning, such as HashGAN and SPQ.

[0167] ResNet18 was used as the backbone network. For data augmentation methods, AutoAugment, random cropping, random horizontal flipping, and CutOut were adopted. Specifically, for the Lace dataset and Fabric 1.0 dataset, the AutoAugment data augmentation strategy pre-trained on the ImageNet dataset was used.

[0168] During adversarial training, use A constrained 10-step PGD is used to generate adversarial perturbations, where ε = 8 / 255 and the step size is 2 / 255. When generating the high-frequency component view, a distance threshold r = 8 is set. The number of codewords in each codebook, the dimension of each codeword, and the number of codebooks are all set to K = 2. 4 D / M = 16 and M = 8.

[0169] During testing, for each dataset, the retrieval performance of the retrieval method on clean images was first calculated. Then, adversarial perturbations were added to the test set images using both FGSM and PGD attack methods to verify the robustness of the model. For the implementation of the PGD and FGSM attack methods, the targetless attack methods of both attacks from the Python library CleverHans were adopted. FGSM with ε = 0.1 and ε = 8 / 255 were used. A constrained PGD process of 20 steps is used to generate adversarial examples.

[0170] 3 Experimental Results

[0171] The performance of MSPQ was evaluated by comparing the retrieval results of MSPQ with those of other state-of-the-art unsupervised image retrieval methods on two fabric datasets: the Lace dataset and the Fabric dataset.

[0172] Table 2. mAP scores (%) of different retrieval methods on two fabric image datasets.

[0173]

[0174] To further verify the robustness of the proposed model against different white-box adversarial attacks, the mAP of the baseline model SPQ and the proposed MSPQ was compared on three datasets under two adversarial attacks (PGD and FGDM with 20 iterations). The experimental results are summarized in Table 3. This result shows that the method of the present invention is highly effective in improving the adversarial robustness of the retrieval model, and is effective not only for fabric images but also for natural images.

[0175] Table 3 shows the mAP scores (%) of SPQ and the proposed method on three benchmark datasets after being subjected to two adversarial attacks.

[0176]

[0177] Furthermore, 19 unpredictable attacks were subjected to the test set of the Lace dataset, with 5 severity levels for each attack, resulting in a new dataset, Lace-C. On this dataset, it was verified that the method of this invention is not only robust against adversarial attacks but also robust against other common unpredictable attacks. Figure 4 As shown, the method of this invention achieves improvements over the baseline method in most unforeseen attacks (except for some reduction in the four types of damage: brightness, elastic transform, frost, and snow). The proposed method shows an average improvement of 5.83% across all unforeseen attacks.

[0178] 4 Ablation Learning

[0179] Table 4 presents the effectiveness evaluation of each component of the proposed method on the lace dataset. It can be observed that when the query set consists entirely of clean samples, the evaluation metrics are highest after changing the data augmentation method of the baseline model to AutoAugment. This indicates that AutoAugment is indeed more suitable for fabric images, especially lace images, compared to mainstream contrastive learning data augmentation methods. However, when the query set consists of adversarial examples, the evaluation metrics are lower than the baseline model. Furthermore, adding a multi-view module to the baseline model for adversarial training results in a decrease in evaluation metrics on clean samples, but a slight increase on adversarial samples. Only by combining the multi-view module with AutoAugment can the best results be achieved on both clean and adversarial samples.

[0180] Table 4 shows the mAP scores (%) on the Lace dataset before and after adversarial attacks, when the baseline method (SPQ) is combined with a multi-view component or the AutoAugment (AA) data augmentation technique is used.

[0181]

[0182] exist Figure 5 The paper visualizes some of the retrieval results returned by the proposed method and SPQ for query images with added adversarial perturbations.

[0183] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0184] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0185] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0186] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0187] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0188] This patent is not limited to the above-described preferred embodiments. Anyone can derive other forms of robust fabric image retrieval methods under a multi-view self-supervised product quantization strategy based on the teachings of this patent. All equivalent changes and modifications made within the scope of the claims of this invention shall fall within the scope of this patent.

Claims

1. A robust fabric image retrieval method under a multi-view self-supervised product quantization strategy, characterized in that, Includes the following steps: Step S1: Create a dataset: Scan the lace samples into images, process them, and obtain one image for each type of lace sample; The dataset is then divided into a training set and a test set; Step S2: Construct a multi-view self-supervised product quantization network, including a multi-view module, a feature extractor, and a quantization head; The multi-view module includes: two views enhanced using the AutoAugment method, an adversarial example view, and a high-frequency component view of the image, for a total of four views; the feature extractor uses ResNet18 as the backbone network to extract the input image into a depth descriptor; the codebook in the quantization head divides the high-dimensional depth feature space into a Cartesian product of multiple subspaces, converting the depth descriptor into a quantization descriptor represented by codewords; Step S3: Construct a multi-view cross-quantization contrastive loss function, and learn codewords and depth visual descriptors by comparing the four views in the multi-view module; when the views come from the same image, their depth descriptors and product quantization descriptors are considered correlated, while if the views come from different images, they are considered uncorrelated; Step S4: Train the neural network constructed in step S2 using the training set established in step S1, and use the loss function in step S3 during the training process; Step S5: After training is complete, use the trained feature extractor and codebook to build an image retrieval database, and convert all images in the dataset into binary codes to build a binary code retrieval database. Step S6: When querying the image to be retrieved, use a feature extractor to extract the feature vector of the image to be retrieved, use Euclidean distance to calculate the similarity between the feature vector subvector and the codeword in the codebook, and output the n images with the highest similarity to the image to be retrieved as the retrieval results; In step S2, the multi-view self-supervised product quantization network specifically refers to: For the input image This generates four views: two views enhanced using the AutoAugment method, an adversarial example view, and a high-frequency component view. The AutoAugment method uses an enhancement strategy trained on the ImageNet dataset. For the adversarial example view, it uses... A constrained 10-step projective gradient descent algorithm generates adversarial perturbations, where the attack size is controlled by... Step size is For high-frequency component views, calculate using the following formula: ， in, and Represents the Fourier transform and its inverse transform; and The distinction is made through a hard threshold operation, with the threshold defined as 8; ResNet18 is used as the backbone feature extractor network to extract features for each input view; its projection head is a multilayer perceptron: the first fully connected layer maps the features to 512 dimensions, followed by a ReLU layer, and the second FC layer outputs a D-dimensional feature vector. The quantization head reconstructs a D-dimensional feature vector into a D-dimensional quantized representation; quantization head Use A codebook, that is The codebook Depend on Composed of individual code characters: , Use a soft quantizer in the quantizer head. : ， in This is a non-negative temperature parameter used to measure the parameters of softmax. The squared Euclidean distance is used to measure the similarity between two inputs. The loss function in step S3 specifically includes the following design: (1) The loss function used is the contrast loss function for multiple views: in Representing an image The One view, Indicates except Other A front view Indicates the division point Other Each batch of samples A collection of augmented views, with a quantity of... indivual; (2) Two views augmented using the AutoAugment data augmentation method, plus adversarial examples and high-frequency components , thus defined in Four-view cross-quantization adversarial contrast loss: in, It is the magnitude of the maximum perturbation during adversarial training. and These are the trainable parameters in the model; (3) The objective function throughout the training process is: 。 2. The robust fabric image retrieval method under a multi-view self-supervised product quantization strategy according to claim 1, characterized in that: In step S1, all lace samples are scanned as images. During scanning, light-colored lace is laid flat on black cardstock, while dark-colored samples are placed against a white cardstock background. The process of building the dataset includes the following steps: Step S11: Scan each of the different lace samples to obtain one image of each type of lace sample; Step S12: Rotate and crop the image so that the edge of the lace trim is parallel to the image border; Step S13: Image scaling; To reduce computational load and to meet the input size requirements of the backbone convolutional neural network ResNet18, the image is scaled down to 224*224 pixels.

3. The robust fabric image retrieval method under a multi-view self-supervised product quantization strategy according to claim 1, characterized in that: The training process in step S4 is as follows: Step S41: Input the training set into the neural network in batches for training, and generate four views for each training image; Step S42: Use the feature extractor to extract depth descriptors for each view, and the quantization head maps each depth descriptor to a quantization descriptor; Step S43: Calculate the loss value of the current batch of data using the multi-view cross-quantization comparison loss function in step S3; Step S44: Backpropagation of gradients, using the Adam optimizer to optimize network parameters. and .

4. The robust fabric image retrieval method under a multi-view self-supervised product quantization strategy according to claim 3, characterized in that: In step S5, the retrieval database is established as follows: Step S51: For each image in the dataset The feature extractor obtained through training get depth descriptor and divide it into equal-length subvectors ; Step S52: Calculate each sub-vector separately. With codebook The squared Euclidean distance of each codeword in the search is used to find the subvector. The closest code; Step S53: Find the nearest neighbor codeword The index format is converted into binary code to generate a sub-binary code. Concatenate all the sub-binary codes to generate one Binary encoding of bits ; Step S54: Repeat steps S51-S53 for all images in the image library to build a binary-coded retrieval database.

5. A robust fabric image retrieval method under a multi-view self-supervised product quantization strategy according to claim 4, characterized in that: The steps for querying and retrieving images in step S6 are as follows: Step S61: When a query image is input The same feature extractor is used when performing a search. right Feature extraction was performed to obtain Then, using the same subvector partitioning process, Divided into ; Step S62: Using Euclidean distance, calculate the similarity between the sub-vectors of the query image and each codeword in all codebooks, and sum the distances of these sub-vectors to obtain the asymmetric distance between the query image and each database image. Step S63: Based on the similarity calculation results, sort the candidate images, placing the image with the highest similarity at the top, and return the previous image as needed. Images similar to the input image are used as search results.

Images