An interactive clothing image retrieval method and system based on attribute feature unwinding

Abstract

The application discloses an interactive clothing image retrieval method and system based on attribute feature disentanglement, an image acquisition module acquires clothing image data to be retrieved, and transmits the acquired clothing image data to an image preprocessing module for processing, key feature information is extracted from the preprocessed image, various attribute information and feature information of the image are predicted based on the extracted feature information, the predicted attribute information is edited, modified and deleted, the extracted feature information is stored in a database, similar images are found from the database according to a keyword or a sample picture input by a user, and the similar images are returned to the user, so that the user can quickly and accurately find the clothing image that the user wants. The application solves the attribute change retrieval problem and the dimension disaster problem, and can solve the personalized retrieval of clothing images with different attributes and the dimension disaster problem in the multi-attribute joint retrieval through disentanglement expression and memory matrix blocks.

Description

Technical Field

[0001] This invention belongs to the field of image retrieval technology, specifically relating to an interactive clothing image retrieval method and system based on attribute feature untangling. Background Technology

[0002] With the rapid development of e-commerce and the fashion industry, the number of online shopping websites continues to climb. Because online shopping offers a convenient user experience, consumers increasingly prefer this method of shopping. Clothing is one of the most popular items in online shopping; consumers don't need to go to a store to try things on, they can easily find the styles they need simply by moving their fingers at home. Online shopping not only saves time but also offers more affordable prices and a wider selection of styles, making it very popular with consumers.

[0003] Currently, there are two main search methods on major shopping websites:

[0004] One approach is Text-Based Image Retrieval (TBIR), which aims to tag each image with keywords and then perform image retrieval by matching user-input keywords. However, text-based image retrieval has several drawbacks: 1) Unstable annotation quality: Manually annotating large databases is extremely costly, and the quality can fluctuate due to differences in annotator skills, affecting the accuracy of search results; 2) Users need to input keywords for matching, making it highly susceptible to subjective bias; 3) Semantic gap: A semantic gap may exist between images and tags. Even if two tags look very similar, they may differ semantically. For example, "sky blue" and "lake blue" both describe blue, but they depict completely different objects; 4) Diversity issues: Since an image can often be assigned multiple tags, too many related images may appear during the search, which may not be what the user expects.

[0005] Second is Content-Based Image Retrieval (CBIR), commonly known as "image search." In recent years, with the development of deep learning technology, CBIR has gradually become an important solution. CBIR identifies features by analyzing the image itself and then performs similarity matching based on these features. CBIR methods can overcome some of the problems faced by text-based image retrieval methods. However, current mainstream deep learning-based clothing image retrieval methods aim to retrieve results as similar as possible to the query image. The main steps are: 1) extracting features from user-uploaded images using deep learning neural networks; 2) calculating the distance between the uploaded image features and clothing features in the search database using a metric function (closer distances indicate greater similarity); 3) presenting the n closest images to the user. However, these methods rarely consider personalized user requirements. A single clothing similarity retrieval method is increasingly unable to meet users' demands for personalized clothing searches. Therefore, an interactive clothing image retrieval system with customizable attributes is more accurate, efficient, and saves users significant time and effort. However, in practical applications, existing methods often fail because clothing images contain a large number of attribute features that may be related or intertwined with each other. In other words, if you want to change one feature (such as changing the color of a T-shirt), you may end up changing other features at the same time (the texture may change from print to stripes).

[0006] In summary, how to construct a deep metric learning method for clothing image retrieval that can change one image attribute feature while minimizing the impact on other attribute features has become an urgent problem for researchers in this field. Summary of the Invention

[0007] To address the aforementioned technical problems in the existing technology, this invention provides an interactive clothing image retrieval method and system based on attribute feature untangling.

[0008] The technical problem to be solved by this invention is achieved by the following technical solution:

[0009] An interactive clothing image retrieval method based on attribute feature unwrapping includes the following steps:

[0010] S1: After preprocessing the original image, the image feature representation is extracted using a visual converter model;

[0011] S2: Define a list of clothing attributes, construct a subspace model for each attribute, and these subspace models are composed of a fully connected two-layer network. Input the image features extracted in step S1 into the subspace model. Each subspace model outputs a subspace model for each attribute. The subspace model is composed of a fully connected two-layer network to form the corresponding subspace feature representation.

[0012] S3: Construct a classification layer consisting of fully connected layers, taking the subspace feature representation from step S2 as input, and obtaining the predicted value of each attribute value of the given image as the output. The predicted value is used to compare with the true value of the image to train and fine-tune the subspace model in step S2.

[0013] S4: After training and optimizing the subspace model through step S3, connect the subspace feature representations in step S2 to form the unwrapped feature representation of the original image.

[0014] S5: Construct a memory matrix block model and operation vectors, multiply the output representation of the memory matrix block model and the operation vectors, and then add the unwrapped feature representation from step S4 to form the target image feature representation after the attribute changes expected by the user.

[0015] S6: Construct the model loss function, train and fine-tune the model from steps S2 and S5;

[0016] S7: Batch the clothing image dataset and sequentially obtain the unwrapped feature representation of the clothing images through steps S1, S2, and S4, and construct a feature database;

[0017] S8: Obtain the unwrapped feature representation of the image to be retrieved from the image uploaded by the user through steps S1, S2 and S4. Generate the operation vector by allowing the user to select the attribute to be changed through interaction. Obtain the final feature representation of the target image through step S5.

[0018] S9: Calculate the similarity between the target image feature representation from step S8 and all feature representations in the feature database from step S7 using an optimized supervised learning algorithm. Sort the images by similarity and output the top n images to the user.

[0019] Furthermore, step S1 is specifically as follows:

[0020] For the preprocessed image I n Image feature representation f is extracted through a visual converter model. n f n The calculation formulas are shown in equations (1), (2), (3), and (4) below:

[0021] X n =PatchEmbedding(I n (1)

[0022] H n,0 =X n (2)

[0023] H n,l=TransformerEncoderLayer(H n,l-1 (3)

[0024] f n =GlobalAveragePooling(H n,L (4)

[0025] Where l∈[1,L], l refers to the encoder number of the visual converter model, with a value range of 1 to L, and the Lth layer is the last layer; X n Image I n The output representation obtained after the PatchEmbedding process; H n,0 It is the input representation of the TransformerEncoderLayer process; H n,l This is the output representation of the l-th layer of the TransformerEncoderLayer process; n represents the image index, indicating that the currently processed image is the n-th image; H n,L This represents the output representation of the entire TransformerEncoderLayer process, specifically the output representation of the Lth layer.

[0026] Furthermore, in step S110, the PatchEmbedding process is as follows: the image is divided into multiple small blocks, and each small block is mapped to a patch vector through an embedding layer;

[0027] The TransformerEncoderLayer process is as follows: multiple Transformer encoder layers are used to process the patch vector in order to capture the interactions and dependencies between them;

[0028] The Global Average Pooling process is as follows: Global average pooling is performed on the output of the last encoder layer to obtain the image feature representation f of the entire image. n .

[0029] Furthermore, step S2 includes the following steps:

[0030] S210: Predefined attribute list: {a1,a2,…,a…} A}, where A is the number of attributes; each attribute value a is associated with a list of possible values. Related, where J a It is the total number of possible values ​​for attribute 'a';

[0031] S220: Construct a fully connected two-layer network φ for each attribute value a. a The image feature representation f obtained in step S1n Input to each attribute value φ a In a network, the image feature representation f n Mapping to a property-specific subspace representation: r n,a =φ a (f n ); where r n,a The following calculation formula is given:

[0032] r n,a =φ a (f n =ReLU(fW1+b1)W2+b2 (5)

[0033] Among them, f n Here, W1 is the weight matrix from the input layer to the hidden layer, b1 is the bias vector, ReLU(·) is the activation function, W2 is the weight matrix from the hidden layer to the output layer, b2 is the bias vector of the output layer, and φ is the input image feature representation. a This represents a fully connected two-layer network model.

[0034] Furthermore, step S3 specifically includes:

[0035] Construct a classification model consisting of a fully connected layer with softmax, and process the r obtained in step S220. n,a The image I is input into the classification model. n Attribute prediction values Attribute prediction value The following calculation formula is given:

[0036] z n,a =W3(r n,a )+b3 (6)

[0037]

[0038] Where W3 represents the weight matrix of the fully connected layer; b3 represents the bias vector of the fully connected layer. n,a The characteristic subspace representation r n,a The output representation obtained through the fully connected layer; the softmax process is to convert the z-axis of the fully connected layer output. n,a Convert to a probability distribution to represent the predicted value for each attribute.

[0039] Furthermore, step S4 specifically includes:

[0040] After training and optimizing the subspace model in step S3, the subspace feature representations from step S2 are concatenated to form the unwrapped feature representation r of the original image. n ; Untangled feature representation r nIt is obtained from the following calculation formula:

[0041] r n =(r n,1 ,…,r n,A (8)

[0042] Furthermore, step S5 specifically includes the following steps:

[0043] S510: Construct a memory matrix block model and operation vector i,

[0044] Memory Matrix Block Model It is obtained from the following calculation formula:

[0045]

[0046] in, The prototype embeddings of the j-th attribute value of attribute a are obtained through subsequent training.

[0047] The operation vector i is obtained by the following formula:

[0048] i = v p -v q =(i 1 i 2 ,…,i J (10)

[0049] Where i∈{-1,1,0}, corresponds to deleting an attribute value, adding an attribute value, or keeping it unchanged;

[0050] v p The attribute description of the target image is represented as follows:

[0051]

[0052] v q To query the attribute description of an image, it is represented as:

[0053]

[0054] S520: Calculate the feature representation r' of the target image to be retrieved. The feature representation r' of the target image is obtained by the following formula:

[0055]

[0056] Where, r q It is the untangled feature representation of the query image.

[0057] Further, step S6 specifically includes:

[0058] Construct an overall loss function Ltotal The model described in steps S2 and S5 is trained and optimized; the overall loss function L total It is calculated using the following formula:

[0059] L total =W cls ×L cls +W c ×L c +W lt ×L lt +W ndr ×L ndr +W mt ×L mt (14)

[0060] Where W represents the weights of each loss function; L cls L is the cross-entropy loss function, used to supervise the training in step S3; c L is the consistency loss function, whose purpose is to encourage semantic consistency. lt The label triplet loss function encourages images with similar attributes to have similar feature representations; L ndr The asymmetric regularization loss is used to train the prototype embeddings obtained in step S510 while preserving untangled properties; L mt The operation of the triplet loss function aims to encourage r' in step S520 to be as close as possible to a positive representation with the desired attributes.

[0061] Furthermore, in step S6, the cross-entropy loss function L cls The following calculation formula is given:

[0062]

[0063] Among them, y n,a Image I n The true value of attribute a, p(·) represents the conditional probability function, which represents the probability of an event occurring given certain conditions; Image I n The predicted value of the attribute on attribute a; N is the number of samples in the training set, and A is the number of attributes.

[0064] Furthermore, in step S6, the consistency loss function L c The following calculation formula is given:

[0065]

[0066] Where d(·) is the L2 distance; It is the memory matrix block model obtained in step S510; v q , and These are the attribute label vectors of the reference image, positive sample, and negative sample generated based on the operation task.

[0067] Furthermore, in step S6, the tag triplet loss function Llt has the following calculation formula:

[0068]

[0069] in, and , respectively, are the standard unwrapped representations of positive and negative samples, where m is the boundary parameter and d(·) is the L2 distance; positive samples are images with the same true attribute labels as the reference image.

[0070] Furthermore, in step S6, the operation asymmetric regularization loss function L ndr The following calculation formula is given:

[0071]

[0072]

[0073] in, Represents a matrix of all ones. This is represented as element-wise multiplication. represents off-diagonal elements; ||·||1 represents L1 norm calculation.

[0074] Furthermore, in step S6, the operational triplet loss function Lmt has the following calculation formula:

[0075]

[0076] in, and are the standard untangled feature representations of positive and negative samples, respectively, where m is the boundary parameter and d(·) is the L2 distance.

[0077] Furthermore, step S7 specifically includes:

[0078] The public or private clothing image dataset is processed sequentially through steps S1, S2, and S4 to extract the unwrapped feature representations of the images in batches. The unwrapped feature representations are then stored in a file named after the image file name. The corresponding file name of the image is used as an index in the database, forming the offline library generation subsystem of the retrieval system. Generally, the system only needs to initialize and generate the retrieval library once so that it can be quickly queried in the retrieval system in the future.

[0079] Further, step S8 includes the following steps:

[0080] S810: Obtain the unwrapped feature representation r of the image to be retrieved from the user-uploaded image through steps S1, S2, and S4. q ;

[0081] S820: Construct user interface software, through user interaction to select the attribute to be changed, the system automatically generates operation vector i based on the selection, and obtains the final target image feature representation through step S5; the above constitutes the online query subsystem of the retrieval system.

[0082] Further, step S9 specifically includes:

[0083] The similarity between the target image feature representation in step S8 and all feature representations in the feature database in step S7 is calculated using an optimized supervised learning algorithm. Based on the similarity, the top n images are output to the user.

[0084] Furthermore, in step S9, the optimized supervised learning algorithm includes the following steps:

[0085] 1. Establish a binary tree data structure: Insert all features from step S7 into the binary tree;

[0086] 2. Select query point: Select feature r from step S8. q ;

[0087] 3. Nearest Neighbor Search: Starting from the root node, recursively search the binary tree downwards; at each level, based on the query point r... q The distance between the current node and the target node determines whether to enter the left or right subtree, until a leaf node is reached;

[0088] The distance algorithm is used to measure the distance between two feature representations (samples), as shown in the following formula (20):

[0089]

[0090] Where r q,i and r p,i These represent the query point r respectively. p and sample point r p The feature value in the i-th dimension, d represents the number of feature dimensions; dist(·) represents the distance function.

[0091] 4. Randomization optimization: During the backtracking process, for each parent node, check whether there may be a closer sample in its other subtree; then, with a certain probability p, choose to continue searching downwards or prune; specifically, a random number t can be selected, if t>p, then continue searching downwards; otherwise, prune and continue backtracking.

[0092] 5. Update nearest neighbors: Record the currently visited nearest neighbor samples and continue searching for other unvisited nodes. If a node closer to the query point r is found... q If the sample is not found, then update the nearest neighbor sample;

[0093] 6. Backtracking: During the backtracking process, for each parent node, check if there might be a closer sample in its other subtree. If so, continue searching downwards.

[0094] 7. Verification Results: Finally, verify whether the nearest neighbor sample meets the requirements and return the results.

[0095] This invention also includes a retrieval system for an interactive clothing image retrieval method based on attribute feature unwrapping, characterized in that it includes:

[0096] The module comprises an image acquisition module, an image preprocessing module, a feature extraction module, an attribute prediction module, a feature database construction module, an attribute manipulation module, and a similarity retrieval module; among which:

[0097] The image acquisition module is used to acquire the clothing image data to be retrieved and transmit the acquired clothing image data to the image preprocessing module for processing.

[0098] The image preprocessing module is used to preprocess and optimize the acquired clothing image data, effectively reducing interference and noise in the image and improving the system's ability to recognize and understand the image;

[0099] The feature extraction module is used to extract key feature information from the preprocessed image. This feature information can reflect various attributes and characteristics of the image, which facilitates subsequent processing and retrieval.

[0100] The attribute prediction module is used to predict various attribute information and feature information of the image based on the extracted feature information. This attribute information provides input support for subsequent attribute operations.

[0101] The attribute manipulation module includes editing, modifying, and deleting predicted attribute information, allowing users to further filter and customize search results, thus improving the user experience.

[0102] The feature database module is used to store the extracted feature information into the database for subsequent retrieval and use, effectively improving the system's retrieval efficiency and accuracy.

[0103] The similarity search module finds the most similar images from the database based on user-input keywords or sample images and returns them to the user, allowing them to quickly and accurately find the clothing images they want and improving the system's intelligence.

[0104] Compared with existing clothing image retrieval methods, the method of the present invention has the following advantages:

[0105] 1. Advanced feature representation: Compared with traditional convolutional neural networks, this invention uses an advanced visual converter model, which makes the classification feature representation of clothing images more accurate.

[0106] 2. Accurate description of attribute information: By defining a list of clothing attributes and constructing a subspace model, this invention can more accurately describe the attribute information of clothing images and improve the accuracy of retrieval results.

[0107] 3. Solving the problems of attribute change retrieval and the curse of dimension: Through unentangled representation and memory matrix block model, this invention can solve the problem of personalized retrieval of clothing images with different attributes, as well as the problem of the curse of dimension that occurs when multiple attributes are jointly retrieved.

[0108] 4. Accurate similarity matching: By calculating and ranking similarity through an optimized supervised learning algorithm, this invention can more accurately match similar images, thus improving the accuracy of retrieval. Attached Figure Description

[0109] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with the embodiments of the invention to explain the invention, but do not constitute a limitation thereof. In the drawings:

[0110] Appendix Figure 1 This is a flowchart of the PatchEmbedding process provided in Embodiment 1 of the present invention;

[0111] Appendix Figure 2 The flowchart of TransformerEncoderLayer provided in Embodiment 1 of the present invention;

[0112] Appendix Figure 3 This is a flowchart of the image unwrapping representation provided in Embodiment 1 of the present invention;

[0113] Appendix Figure 4 The flowchart of the optimized supervised learning algorithm provided in Embodiment 1 of the present invention is shown below;

[0114] Appendix Figure 5 This is an overall flowchart of the interactive clothing image retrieval system provided in Embodiment 2 of the present invention;

[0115] Appendix Figure 6 This is a flowchart of the attribute prediction module provided in Embodiment 2 of the present invention;

[0116] Appendix Figure 7 This is a flowchart of the attribute operation module provided in Embodiment 2 of the present invention;

[0117] Appendix Figure 8This is a flowchart of the feature library building module provided in Embodiment 2 of the present invention;

[0118] Appendix Figure 9 This is a flowchart of the similarity retrieval module provided in Embodiment 2 of the present invention;

[0119] Appendix Figure 10 This is a schematic diagram of an interactive clothing image retrieval device provided in Embodiment 3 of the present invention. Detailed Implementation

[0120] To make the technical solution of the present invention clearer, the technical solution provided by the present invention will be described in detail below with reference to specific embodiments, and the present invention will be further illustrated with reference to the accompanying drawings. It is understood that the specific embodiments described herein are only some embodiments of the present invention, used to explain the present invention, and not to limit the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0121] Example 1

[0122] refer to Figures 1 to 4 The present invention provides an interactive clothing image retrieval method based on attribute feature unwrapping, the overall process of which is as follows:

[0123] S1: Preprocess the original image, including cropping, and then use a visual converter model to extract image feature representations.

[0124] Specifically, the preprocessing process for the original image is as follows:

[0125] Image scaling and cropping: The input image is scaled proportionally or cropped to a specified size. This is done to ensure the input image meets the model's input size requirements, while also avoiding the problem of excessive computation caused by images of different sizes.

[0126] Image normalization: This involves normalizing the pixel values ​​of an image to ensure they fall within a suitable range, such as [0,1] or [-1,1]. The purpose of this is to make the model more stable during training and to achieve faster convergence.

[0127] Color space conversion: Converting an image from one color space to another. For example, converting RGB color space to grayscale color space, or RGB color space to HSV color space. The purpose of this is to improve the model's robustness to color transformations.

[0128] Data augmentation: Applying various random transformations, such as rotation, scaling, translation, and flipping, to the training data to generate more training samples. The purpose of this is to improve the model's robustness and generalization ability to image transformations.

[0129] Specifically, step S1 includes the following steps:

[0130] For the preprocessed image I n Image feature representation f is extracted through a visual converter model. n f n The calculation formulas are shown in equations (1), (2), (3), and (4) below:

[0131] X n =PatchEmbedding(I n (1)

[0132] H n,0 =X n (2)

[0133] H n,l =TransformerEncoderLayer(H n,l-1 (3)

[0134] f n =GlobalAveragePooling(H n,L (4)

[0135] Where l∈[1,L], l refers to the encoder number of the visual converter model, with a value range from 1 to L, and the Lth layer is the last layer. X n Image I n The output representation obtained after the PatchEmbedding process; H n,0 It is the input representation of the TransformerEncoderLayer process; H n,l This is the output representation of the l-th layer of the TransformerEncoderLayer process; n represents the image index, indicating that the currently processed image is the n-th image; H n,L This represents the output representation of the entire TransformerEncoderLayer process, specifically the output representation of the Lth layer.

[0136] Specifically, the PatchEmbedding process is as follows: Figure 1 As shown: The image is divided into multiple patches, and each patch is mapped to a vector through an embedding layer;

[0137] Specifically, the TransformerEncoderLayer process is as follows: Figure 2 As shown: Multiple Transformer encoder layers are used to process the patch vectors in order to capture their interactions and dependencies;

[0138] Specifically, the Global Average Pooling process involves performing global average pooling on the output of the last encoder layer to obtain the feature representation f of the entire image. n .

[0139] S2: Define a list of clothing attributes and construct a subspace model for each attribute. For each attribute, we train a subspace model consisting of a fully connected two-layer network, taking the image feature representation from step one as input, and each subspace model outputs the corresponding subspace feature representation.

[0140] Specifically, step S2 includes the following steps:

[0141] S210: Predefined attribute list: {a1,a2,…,a…} A}, where A is the number of attributes. Each attribute value 'a' is paired with a list of possible values. Related, where J a It is the total number of possible values ​​for attribute a.

[0142] S220: Construct a fully connected two-layer network φ for each attribute value a. a f obtained in step S1 n Input to each attribute value φ a In the network, f is about to n Mapping to a property-specific subspace representation: r n,a =φ a (f n ).

[0143] Specifically, r n,a The following calculation formula is given:

[0144] r n,a =φ a (f n =ReLU(fW1+b1)W2+b2 (5)

[0145] Among them, f n Here, W1 is the weight matrix from the input layer to the hidden layer, b1 is the bias vector, ReLU(·) is the activation function, W2 is the weight matrix from the hidden layer to the output layer, b2 is the bias vector of the output layer, and φ is the input image feature representation. a This represents a fully connected two-layer network model.

[0146] S3: Construct a classification layer consisting of fully connected layers, taking the subspace feature representation from step S2 as input, and obtaining a predicted value for each attribute of a given image as output. This predicted value is used to train and fine-tune the subspace model from step S2 by comparing it with the true values ​​of the image.

[0147] Specifically, step S3 includes the following steps:

[0148] S310: Construct a classification model consisting of a fully connected layer with softmax, and apply the r obtained in step S2... n,a The image I is obtained by inputting it into the classification model. n Attribute prediction values

[0149] Specifically, The following calculation formula is given:

[0150] z n,a =W3(r n,a )+b3 (6)

[0151]

[0152] S4: After training and optimizing the subspace model through step S3, the subspace feature representations in step S2 are connected to form the unwrapped feature representation of the original image.

[0153] Specifically, step S4 includes the following steps:

[0154] S410: After training and optimizing the model through step S3, the subspace feature representations from step S2 are concatenated to form the unwrapped feature representation r of the original image. n The flowchart for untangling feature representation is as follows: Figure 3 As shown.

[0155] Specifically, r n The calculation formula is as follows:

[0156] r n =(r n,1 ,…,r n,A (8)

[0157] S5: Construct a memory matrix block model and operation vectors, multiply the output representation of the memory matrix block model and the operation vectors, and then add the unwrapped feature representation from step S4 to form the retrieval target image feature representation after the attribute changes expected by the user.

[0158] Specifically, step S5 includes the following steps:

[0159] S510: Construct a memory matrix block model And operation vector i.

[0160] Specifically, The following calculation formula is given:

[0161]

[0162] in, This represents the prototype embedding of the j-th attribute value of attribute a. These embeddings are obtained through subsequent training.

[0163] Specifically, the operation vector i is calculated using the following formula:

[0164] i = v p -v q =(i 1 i 2 ,…,i J (10)

[0165] Where i∈{-1,1,0}, corresponds to deleting an attribute value, adding an attribute value, or keeping it unchanged.

[0166] Specifically, v p The attribute description of the target image is represented as follows:

[0167]

[0168] Specifically, v q To query the attribute description of an image, it is represented as:

[0169]

[0170] S520: Calculate the feature representation r' of the target image to be retrieved.

[0171] Specifically, r' has the following calculation formula:

[0172]

[0173] Where, r q It is the untangled feature representation of the query image.

[0174] S6: Construct the model loss function, train and fine-tune the model from steps S2 and S5.

[0175] Specifically, step S6 includes the following steps:

[0176] S610: Construct the overall loss function L total The model described in steps S2 and S5 is then trained and optimized.

[0177] Specifically, L total The following calculation formula is given:

[0178] L total =W cls ×L cls +W c ×L c +W lt ×L lt +W ndr ×L ndr +W mt ×L mt (14)

[0179] Where W represents the weights of each loss function, and L... cls L is the cross-entropy loss function, which is used to supervise the training in step three; c L is the consistency loss function, whose purpose is to encourage semantic consistency. lt The label triplet loss function aims to encourage images with similar attributes to have similar feature representations; L ndr The asymmetric regularization loss is used to train the prototype embeddings obtained in step S510 while preserving untangling properties; L mt The operation of the triplet loss function aims to encourage r' in step S520 to be as close as possible to a positive representation with the desired properties.

[0180] Specifically, L cls The following calculation formula is given:

[0181]

[0182] Among them, y n,a Image I n The true value of attribute a, p(·) represents the conditional probability function, which represents the probability of an event occurring given certain conditions; Image I n The predicted value of the attribute on attribute a; N is the number of samples in the training set, and A is the number of attributes.

[0183] Specifically, L c The following calculation formula is given:

[0184]

[0185] Where d(·) is the L2 distance; It is the memory matrix block model obtained in step S510; v q , and These are the attribute label vectors of the reference image, positive sample, and negative sample generated based on the operation task.

[0186] Specifically, L lt The following calculation formula is given:

[0187]

[0188] in, and , respectively, are the standard unwrapped representations of the positive and negative samples, where m is the boundary parameter and d(·) is the L2 distance. A positive sample is an image with the same ground truth attribute labels as the reference image.

[0189] Specifically, L ndr The following calculation formula is given:

[0190]

[0191]

[0192] in, Represents a matrix of all ones. This is represented as element-wise multiplication. This indicates off-diagonal elements. ||·||1 indicates L1 norm calculation.

[0193] Specifically, L mt The following calculation formula is given:

[0194]

[0195] in, and are the standard untangled feature representations of positive and negative samples, respectively, where m is the boundary parameter and d(·) is the L2 distance.

[0196] S7: The clothing image dataset is processed in batches through steps S1, S2 and S4 to obtain the clothing image unwrapped feature representation database.

[0197] Specifically, step S7 includes the following steps:

[0198] Steps S1, S2, and S4 extract the unwrapped feature representations of publicly available or private clothing image datasets in batches, store them in files named after the image file names, and store the corresponding file names as indexes in the database, thus forming the offline library generation subsystem of the retrieval system. Generally, the system only needs to initialize and generate the retrieval library once so that it can be quickly queried in the retrieval system in the future.

[0199] S8: Obtain the unwrapped feature representation of the image to be retrieved from the user-uploaded image through steps S1, S2, and S4. Generate an operation vector by allowing the user to select the attributes to be changed, and obtain the final feature representation of the target image through step five.

[0200] Specifically, step S8 includes the following steps:

[0201] S810: Obtain the unwrapped feature representation r of the image to be retrieved from the user-uploaded image through steps S1, S2, and S4. q .

[0202] S820: Construct user interface software, allowing users to select the attributes to be changed through interaction. The system automatically generates an operation vector i based on the selection, and the final feature representation of the target image is obtained through step S5. The above constitutes the online query subsystem of the retrieval system.

[0203] S9: Calculate the similarity between the target image feature representation from step S8 and all feature representations in the feature database from step S7 using an optimized supervised learning algorithm. Sort the images by similarity and output the top n images to the user.

[0204] Specifically, in step S9, the optimized supervised learning algorithm is as follows: Figure 4 As shown, the steps are as follows:

[0205] 1. Establish a binary tree data structure: Insert all features from step S7 into the binary tree.

[0206] 2. Select the query point: Select feature r from step eight. q .

[0207] 3. Nearest Neighbor Search: Starting from the root node, recursively search downwards in the binary tree. At each level, based on the query node r... q The distance between the current node and the target node determines whether to enter the left or right subtree, until a leaf node is reached.

[0208] 4. Randomization Optimization: During backtracking, for each parent node, check if there might be a closer sample in its other subtree. Then, with a certain probability p, choose to continue searching downwards or prune. Specifically, a random number t can be selected; if t > p, continue searching downwards; otherwise, prune and continue backtracking.

[0209] 5. Update Nearest Neighbors: Record the currently visited nearest neighbor samples and continue searching for other unvisited nodes. If a node closer to the query point r is found... q If the sample is not found, then update the nearest neighbor sample.

[0210] 6. Backtracking: During backtracking, for each parent node, check if there might be a closer sample in its other subtree. If so, continue searching downwards.

[0211] 7. Verification Results: Finally, verify whether the nearest neighbor sample meets the requirements and return the results.

[0212] Specifically, a distance algorithm is used to measure the distance between two feature representations (samples), as shown in the following formula:

[0213]

[0214] Where r q,i and r p,i These represent the query point r respectively. p and sample point r p The feature value in the i-th dimension, d represents the number of feature dimensions; dist(·) represents the distance function.

[0215] Example 2

[0216] Appendix Figure 5 This is a schematic diagram of an interactive clothing image retrieval system provided in Embodiment 2 of the present invention. The device includes: an image acquisition module, an image preprocessing module, a feature extraction module, an attribute prediction module, a feature database construction module, an attribute operation module, and a similarity retrieval module.

[0217] The image acquisition module is used to acquire the clothing image data to be retrieved and then transmits the acquired clothing image data to the image preprocessing module for processing. This module can acquire clothing image data in various ways, such as downloading from the network, capturing from a camera, etc.

[0218] The image preprocessing module is used to preprocess and optimize the acquired images, such as denoising, smoothing, filtering, and enhancement. These operations can effectively reduce interference and noise in the images, improving the system's ability to recognize and understand images.

[0219] The feature extraction module is used to extract key feature information from the preprocessed image, such as color histograms, texture features, shape features, etc. This feature information can reflect various attributes and characteristics of the image, facilitating subsequent processing and retrieval.

[0220] Attribute prediction module, such as Figure 6 As shown, this is used to predict various attributes and features of an image based on extracted feature information, such as the texture, style, and shape of clothing. This attribute information provides input support for subsequent attribute operations.

[0221] Attribute manipulation modules, such as Figure 7 As shown, this module includes operations such as editing, modifying, and deleting attribute information. It allows users to further filter and customize search results, improving the user experience.

[0222] Feature library building module, such as Figure 8As shown, this module stores the extracted feature information in a database for subsequent retrieval and use. This module can effectively improve the system's retrieval efficiency and accuracy.

[0223] Similar search module such as Figure 9 As shown, based on user-input keywords or sample images, the system finds the most similar image from the database and returns it to the user. This module allows users to quickly and accurately find the clothing images they want, improving the system's intelligence.

[0224] Example 3

[0225] Appendix Figure 10 This is a schematic diagram of an interactive clothing image retrieval device according to Embodiment 3 of the present invention. The device includes: a user interface device, a data analysis device, a database management device, and a communication device.

[0226] The user interface is the main window for users to interact with the system. Users can enter keywords or upload sample images here to perform search operations.

[0227] The data analysis device is the core of the interactive clothing image retrieval device. It processes and analyzes the image information input by the user. After image preprocessing, the data analysis device extracts key feature information from the image and predicts the image's attributes based on this information. Simultaneously, it is also responsible for retrieving relevant image information from the database management device and performing similarity matching to return the most suitable result to the user.

[0228] The database management device is used to store and manage all data and information involved in the system, including user personal information, image feature information, etc. It is responsible for classifying, organizing, and managing this data, and provides efficient access interfaces and query functions to meet the system's need for rapid data response during operation. At the same time, it also plays a crucial role in data security protection, ensuring that sensitive data is not obtained or tampered with by malicious attackers.

[0229] The communication device is used for data transmission and communication between different devices to achieve information synchronization and real-time updates. It supports multiple communication methods and protocols to meet the needs of different scenarios and ensures the confidentiality and integrity of data during communication, preventing information leakage.

[0230] It should be noted that in the above embodiments for interactive clothing image retrieval devices, the various modules are divided according to functional logic and are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not intended to limit the scope of protection of the present invention.

[0231] The above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An interactive clothing image retrieval method based on attribute feature unwrapping, characterized in that, The retrieval method includes the following steps: S1: After preprocessing the original image, the image feature representation is extracted using a visual converter model; S2: Define a list of clothing attributes, construct a subspace model for each attribute, and these subspace models are composed of a fully connected two-layer network. Input the image features extracted in step S1 into the subspace model. Each subspace model outputs a subspace model for each attribute. The subspace model is composed of a fully connected two-layer network to form the corresponding subspace feature representation. S3: Construct a classification layer consisting of fully connected layers, taking the subspace feature representation from step S2 as input, and obtaining the predicted value of each attribute value of the given image as the output. The predicted value is used to compare with the true value of the image to train and fine-tune the subspace model in step S2. S4: After training and optimizing the subspace model through step S3, connect the subspace feature representations in step S2 to form the unwrapped feature representation of the original image. S5: Construct a memory matrix block model and operation vectors, multiply the output representation of the memory matrix block model and the operation vectors, and then add the unwrapped feature representation from step S4 to form the target image feature representation after the attribute changes expected by the user. S6: Construct the model loss function, train and fine-tune the models in S2 and S5; S7: The clothing image dataset is processed in batches through steps S1, S2, and S4 to obtain the unwrapped feature representation of the clothing images, and a feature database is constructed. S8: The image to be retrieved uploaded by the user is processed through steps S1, S2 and S4 to obtain the unwrapped feature representation of the retrieved image. The user selects the attribute to be changed through interaction to generate the operation vector. The final feature representation of the retrieved target image is obtained through step S5. S9: Calculate the similarity between the target image feature representation from step S8 and all feature representations in the feature database from step S7 using an optimized supervised learning algorithm. Sort the images by similarity and output the top n images to the user.

2. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 1, characterized in that, The specific steps of S1 are as follows: For the preprocessed image Image feature representation is extracted through a visual converter model. , The calculation formulas are shown in equations (1), (2), (3), and (4) below: in, , This refers to the encoder number of the visual converter model, with a value ranging from 1 to... , No. This is the last layer; Representing an image pass The output representation obtained after the process; yes Input representation of the process; yes The first step of the process Layer output representation; The index represents the image, indicating that the currently processed image is the [number]th [image]. Zhang Image; Indicates the whole The output representation of the process, i.e., the first The output representation of the layer.

3. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 1, characterized in that, Step S2 includes the following steps: S210: List of predefined attributes: ,in It is the number of attributes; each attribute value With list of possible values Related, among which It is an attribute The total number of possible values; S220: For each attribute value Build a fully connected two-layer network The image features obtained in step S1 are represented Input to each attribute value In networks, this means representing image features. Mapping to a property-specific subspace representation: ;in, The following calculation formula is given: in, It is the feature representation of the input image. It is the weight matrix from the input layer to the hidden layer. It is a bias vector. It is an activation function. It is the weight matrix from the hidden layer to the output layer. It is the bias vector of the output layer. This represents a fully connected two-layer network model.

4. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 3, characterized in that, Step S3 specifically includes: Build a classification model, which consists of a... The fully connected layer is constructed, and the result obtained in step S220 is... The image is input into the classification model and obtained. Attribute prediction values Attribute prediction value The following calculation formula is given: in, ; This represents the bias vector of the fully connected layer; Representation of characteristic subspace The output representation obtained through the fully connected layer; The process involves outputting the fully connected layer. Convert to a probability distribution to represent the predicted value for each attribute.

5. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 1, characterized in that, Step S4 specifically includes: After training and optimizing the subspace model in step S3, the subspace feature representations from step S2 are concatenated to form the unwrapped feature representation of the original image. Untangling feature representation It is obtained from the following calculation formula: 。 6. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 1, characterized in that, Step S5 specifically includes the following steps: S510: Construct a memory matrix block model and operation vector , Memory Matrix Block Model It is obtained from the following calculation formula: in, Represents attributes The The prototype embeddings of the attribute values ​​are obtained through subsequent training; operation vector It is obtained from the following calculation formula: in, This corresponds to deleting an attribute value, adding one, or leaving it unchanged; The attribute description of the target image is represented as follows: To query the attribute description of an image, it is represented as: S520: Calculate the feature representation of the target image to be retrieved It is obtained from the following calculation formula: in, It is the untangled feature representation of the query image.

7. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 1, characterized in that, Step S6 specifically includes: Construct an overall loss function To train and fine-tune the models described in S2 and S5; It is calculated using the following formula: in, These are the weights for each loss function; This is the cross-entropy loss function, used to supervise the training in step S3; This is the consistency loss function, whose purpose is to encourage semantic consistency. The label triplet loss function encourages images with the same attributes to have similar feature representations. The asymmetric regularization loss is used to train the prototype embedding in step S510 while maintaining the untangled nature. The purpose of operating the triplet loss function is to encourage the process in step S520. It should be as close as possible to a positive representation that has the desired properties.

8. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 1, characterized in that, Step S7 specifically includes: The public or private clothing image dataset is processed sequentially through steps S1, S2, and S4 to extract the unwrapped feature representations of the images in batches. The unwrapped feature representations are then stored in a file named after the image file name. The corresponding file name of the image is used as an index in the database, forming the offline library generation subsystem of the retrieval system. The system only needs to initialize and generate the retrieval library once so that it can be quickly queried in the retrieval system in the future.

9. The interactive clothing image retrieval method based on attribute feature untangling as described in claim 1, characterized in that, Step S8 includes the following steps: S810: Obtain the untangled feature representation of the image to be retrieved from the user-uploaded image through steps S1, S2, and S4. ; S820: Builds user interface software that allows users to select attributes to be changed through interaction, and the system automatically generates operation vectors based on the selection. The final feature representation of the target image is obtained through step S5; The above-mentioned S810 and S820 constitute the online query subsystem of the retrieval system.

10. A retrieval system for an interactive clothing image retrieval method based on attribute feature untangling, characterized in that: include: The module comprises an image acquisition module, an image preprocessing module, a feature extraction module, an attribute prediction module, a feature database construction module, an attribute manipulation module, and a similarity retrieval module; among which: The image acquisition module is used to acquire the clothing image data to be retrieved and transmit the acquired clothing image data to the image preprocessing module for processing. The image preprocessing module is used to preprocess and optimize the acquired clothing image data, effectively reducing interference and noise in the image and improving the system's ability to recognize and understand the image; The feature extraction module is used to extract key feature information from the preprocessed image. This feature information can reflect various attributes and characteristics of the image, which facilitates subsequent processing and retrieval. The attribute prediction module is used to predict various attribute information and feature information of the image based on the extracted feature information. This attribute information provides input support for subsequent attribute operations. The attribute manipulation module includes editing, modifying, and deleting predicted attribute information, allowing users to further filter and customize search results, thus improving the user experience. The feature database module is used to store the extracted feature information into the database for subsequent retrieval and use, effectively improving the system's retrieval efficiency and accuracy. The similarity search module finds the most similar images from the database based on user-input keywords or sample images and returns them to the user, allowing them to quickly and accurately find the clothing images they want and improving the system's intelligence.

Images