A hierarchical style-based conditional text-ebusiness picture retrieval method and system

Abstract

The present application relates to a kind of condition text-commerce picture retrieval method and system based on hierarchical style.The method includes the following steps:1. establish condition text-commerce picture database.2. the average feature of image block is interacted with image block feature by sliding window mechanism and two-stage self-attention mechanism, and the common feature of image block is obtained.3. the common feature is fused with image block feature through graph reasoning network, and useless image block feature is filtered out through local area filtering network, and the difference feature of image block is obtained.4. the common feature and difference feature are fused by low layer-high layer interaction to obtain style feature.5. with style feature as query sample, according to query sample retrieval target picture.The present application can fully exploit image fine-grained information and spatial context information, and learn the fusion alignment between the visual features containing style and the modified text, which achieves higher retrieval accuracy than existing methods.

Description

Technical Field

[0001] This invention relates to the field of multimedia retrieval, specifically to a hierarchical style-based conditional text-e-commerce image retrieval method and system. Background Technology

[0002] Conditional text-image retrieval is a highly flexible search method. Users can suggest modifications to a reference image and retrieve target images that meet their needs. Traditional single-modal image retrieval can only return images similar to the reference image, failing to meet users' customized requirements; traditional text-image cross-media retrieval struggles to accurately retrieve targets based on brief user suggestions. Unlike these two approaches, conditional text-image retrieval is more convenient and flexible, combining text and image modal information queries to obtain results that meet user needs.

[0003] The challenge of conditional text-image retrieval is how to address the inconsistency between text and image modalities and learn their inherent relationships. Because image and text data have different representations and distribution characteristics, and contain unequal amounts of semantic information, it is difficult to fuse and utilize features from different modalities.

[0004] To address the aforementioned issues, existing methods often project features from text and image modal data into a unified semantic space to learn cross-modal fusion representations and retrieve the closest target image based on these fusion features. For example, Chen et al. proposed the VAL method, which improves retrieval accuracy by more fully aligning the semantic information contained within the modalities through multi-level interactions between visual and text modalities. Kim et al. proposed the DCNet method (Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim. Dual compositional learning in interactive image retrieval. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI2021.), which improves the retrieval efficiency of conditional text-image by introducing text queries to minimize the difference between reference and target images in the unified space. Furthermore, mining the rich semantic information contained in images helps align the cross-modal fusion representation with the target image representation. For example, Baldrati et al. proposed the CLIP4Cir method (Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. Effective conditioned and composed image retrieval combining clip-based features. In IEEE / CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022). By introducing the CLIP model to extract features from images and text, a concise and efficient fusion model was designed to map the features of the two modalities to the same representation space.

[0005] However, none of the methods mentioned above take into account the style information contained in e-commerce images, such as urban style, sporty style, and campus style. This style information is crucial for meeting users' modification needs and improving their search experience. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a hierarchical style-based conditional text-e-commerce image retrieval method and system. The style of e-commerce images is defined as the commonalities and differences between local regions of the image. Based on this definition, this invention utilizes a two-stage self-attention mechanism, along with a graph inference network and a local region filtering network, to learn the common and difference features of local image regions, thereby modeling image style. This invention fuses style-enhanced image features and text features through intra-modal and inter-modal approaches, further aligning the fused features with the target features, achieving better retrieval accuracy.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] A hierarchical style-based conditional text-e-commerce image retrieval method includes the following steps, wherein steps (1)-(3) obtain semantic information containing e-commerce image style, and step (4) further integrates reference image features and modified text to retrieve target product images:

[0009] (1) Establish a conditional text-e-commerce image database, which contains triple data of reference image-modified text-target image;

[0010] (2) Obtain image blocks of reference images in the conditional text-e-commerce image database through a sliding window mechanism, and obtain the common features of the image blocks by interacting between the average features and the features of the image blocks through a two-stage self-attention mechanism.

[0011] (3) The common features obtained in step (2) are fused with the image patch features through a graph inference network, and the useless image patch features are filtered out using a local region filtering network to obtain the difference features of the image patch.

[0012] (4) The common features in step (2) and the difference features in step (3) are fused through low-level-high-level interaction to obtain style features. These are used as query examples. Target images are retrieved based on the query examples. That is, the features of target images in the conditional text-e-commerce image database are used as query targets. The similarity between the query examples and the query targets is calculated. Based on the similarity, a list of relevant results for the target modality data is obtained.

[0013] Furthermore, in the above-mentioned hierarchical style-based conditional text-e-commerce image retrieval method, step (1) the conditional text-e-commerce image database contains two modalities: images and text, and can include various clothing types and styles.

[0014] Furthermore, in the above-mentioned hierarchical style-based conditional text-e-commerce image retrieval method, the two-stage self-attention mechanism in step (2) trains a two-stage self-attention network on image patch features, maps each image patch to a common semantic space, and obtains the common features of the image patches.

[0015] Furthermore, in the above-mentioned hierarchical style-based conditional text-e-commerce image retrieval method, the graph inference network and local region filtering network in step (3) use multi-layer graph inference to associate the semantic information contained in different image blocks, and adaptively filter out the weakly associated useless image block features to obtain the differential features of the image blocks.

[0016] Furthermore, in the aforementioned hierarchical style-based conditional text-e-commerce image retrieval method, step (4) involves using style features as query examples and the features of the target image as the query target. For each query example, the similarity to all data of the query target is calculated, and then the data is sorted from largest to smallest similarity to obtain a list of relevant results.

[0017] Corresponding to the above method, the present invention also provides a hierarchical style-based conditional text-e-commerce image retrieval system, which includes:

[0018] The common feature extraction module is used to interact between the average features of image patches and the features of image patches using a sliding window mechanism and a two-stage self-attention mechanism to obtain the common features of image patches.

[0019] The difference feature extraction module is used to fuse common features with image patch features through a graph inference network, and use a local region filtering network to filter out useless image patch features to obtain the difference features of image patches.

[0020] The feature fusion module is used to fuse common features and dissimilar features through low-level-high-level interaction to obtain style features;

[0021] The retrieval module uses style features as query examples and target images as query targets. It calculates the similarity between the query examples and the query targets, sorts the similarities from highest to lowest, and obtains a list of relevant results for the target modality image data based on the similarity.

[0022] The advantages of this invention are as follows: Compared with existing methods, this method explicitly defines the style of e-commerce images and models the common and differential features of local regions of e-commerce images, thereby fully mining fine-grained information and spatial context information of the images. Furthermore, this invention improves the retrieval accuracy of conditional text-e-commerce images by fully integrating the correlation information between image-based style features and modified text-based text features through intra-domain and inter-domain cross-modal alignment.

[0023] The reason this method achieves the aforementioned effects is that it explicitly defines style as the commonalities and differences among image patches for e-commerce images with multiple styles. Based on this definition, a two-stage self-attention mechanism and a multi-layer graph inference network are used to model common and difference features respectively, fully mining fine-grained information and spatial context information within the image modality, thus improving the modeling ability between related region patches. A local region filtering network is used to adaptively filter image patches with weak interaction information, reducing the impact of weakly interactive image patches on the results and increasing the robustness of modeling style features. Furthermore, this method achieves better retrieval performance by mining the correlations between cross-modal features within and between domains. Attached Figure Description

[0024] Figure 1 This is a flowchart of a hierarchical style-based conditional text-e-commerce image retrieval method according to the present invention.

[0025] Figure 2 This is a schematic diagram of the complete network structure of the present invention. Wherein, α represents... The weight of the feature, β represents The weight of the feature, where Q represents the query feature. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0027] The present invention provides a hierarchical style-based conditional text-e-commerce image retrieval method, the process of which is as follows: Figure 1 As shown, it includes the following steps:

[0028] (1) Establish a conditional text-e-commerce image database, which contains triple data of reference image-modified text-target image.

[0029] The reference image is an e-commerce product image; the modification text contains suggestions for changes in color, size, style, etc.; and the target image is an e-commerce product image that is closest to the requirements of the modification text, based on the reference image.

[0030] The image feature vector is specifically designed using the RN50x4 model in the CLIP pre-trained model, with an input image size of 288×288 and an output image feature vector of 640 dimensions. The text feature vector is specifically designed using the Transformer model in the CLIP pre-trained model, which contains 12 layers and outputs a text feature vector T of 640 dimensions.

[0031] (2) By using a sliding window mechanism and a two-stage self-attention mechanism, the average features of the image patch are interacted with the features of the image patch to obtain the common features of the image patch.

[0032] This invention first uses a sliding window mechanism to obtain 13 image blocks from the reference images in the conditional text-e-commerce image database, with sizes of 2×2 and 3×3. Each image block's features are extracted using the image model in (1) to obtain image block features p. i This invention designs a two-stage self-attention mechanism to obtain common features of image patches. The first step uses a simple self-attention mechanism to average the features of all image patches as p. r The second step is to address p from the first step. r p is obtained using a multi-head attention mechanism. m Furthermore, the common features p of the image patches are calculated using the following formula. c :

[0033] p c =f c ([p m ,f q (p r )])

[0034] Among them, f c and f q Both are fully connected layers.

[0035] (3) The common features are fused with the image patch features through the graph reasoning network, and the useless image patch features are filtered out through the local region filtering network to obtain the difference features of the image patch.

[0036] This invention designs a graph connection network to associate image patch features to obtain image patch difference features. Specifically, each image patch feature p obtained in step (2) is... i and common features p c As graph nodes, the nodes in the graph reasoning network are established using the following formulas. And edge e:

[0037]

[0038]

[0039] Among them, W in and W out It is a fully connected layer, at the ingress node v in and outgoing node v out Information is exchanged between them.

[0040] After constructing the graph inference network, the nodes and edges in the graph inference network are updated using the following formula:

[0041]

[0042] Where L represents the number of layers in the graph reasoning network, This represents a fully connected layer. After passing through the graph reasoning network, the common feature p c By fully associating with image patch features to complete information interaction, preliminary differential features are obtained. To reduce the interference of unimportant image patches on the results, this invention designs a filtering network that adaptively calculates the importance weights of image patches and updates the filtered differential features. The calculation process is as follows:

[0043]

[0044]

[0045] Where σ is the Sigmoid function, BN is the batch normalization operation, and W f For a fully connected layer, v i These are image patch features.

[0046] Considering that the proportion of unimportant image blocks varies among different e-commerce products, this invention introduces two self-learning parameters, α and β, to balance the robustness of the invention in order to improve its robustness. and The importance of the difference features p was determined. d As shown in the following formula:

[0047]

[0048] (4) Use the image features of the target image as the query target. Use the style features obtained by fusing common features and difference features as the query example, calculate the similarity between the query example and the query target, and obtain the relevant result list of the target modality data based on the similarity.

[0049] This invention designs a fusion method for image style features and text cross-modal alignment, referred to as intra-domain and inter-domain cross-modal alignment. In the first stage, common features and dissimilar features of image patches are fused. First, the low-level interaction feature p is obtained using the following formula. l :

[0050] p l =f l ([f c (p c ),f d (p d )])

[0051] Among them, f l f c and f d This is a fully connected layer. By learning adaptive parameters θ, common and dissimilar features are fused to obtain style features p based on image patches. hAs shown in the following formula:

[0052] θ=f θ ([f c (p c ),f d (p d )])

[0053] p h =f h (p l )+θ·p c +(1-θ)·p d

[0054] f θ f c f d f h It is a fully connected layer mapping network.

[0055] In the second stage, style features p h Visual features V are obtained by concatenating with global image features. The style-integrated visual features V and the modified text features T are then fused using intra-domain and inter-domain cross-modal alignment. First, intra-domain attention features V′ and T′ are obtained through a self-attention mechanism, as shown in the following formula:

[0056]

[0057]

[0058] Where MLP stands for fully connected layer network, m is the number of reference images and modified text, T is the text features of the modified text, and t is the number of modified text features. i For each modified text, text features are defined. Two adaptive weights θ are learned using a fusion method similar to that in the first stage. self and θ com The fusion features are obtained as shown in the following formula:

[0059]

[0060]

[0061] F final =F self +F com

[0062] in, and It is a fully connected layer network. These are interaction features within the lower-level domain. For low-level inter-domain interaction features, F self For high-level domain interaction characteristics, F comThis refers to the interaction characteristics between higher-level domains.

[0063] The experimental results below show that, compared with existing methods, the hierarchical style-based conditional text-e-commerce image retrieval method of the present invention can achieve higher retrieval accuracy.

[0064] This embodiment uses the FashionIQ conditional text-e-commerce image dataset for experiments. This dataset was proposed in the paper "Fashion iq: A new dataset towards retrieving images by natural language feedback" (authors Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris, published at CVPR 2021). It contains three main classes: the training set contains 18,000 pairs of reference image-modified text-target image triples, the validation set contains 6,017 pairs, and the test set contains 6,119 pairs. The following three methods were tested for experimental comparison:

[0065] Existing Method 1: The VAL method in the paper "Image Search with Text Feedback by VisiolinguisticAttention Learning" (authors Yanbei Chen, Shaogang Gong, and Loris Bazzani).

[0066] Existing Method 2: The DCNet method in the paper "Dual compositional learning in interactive image retrieval" (authors Jongseok Kim, Youngjae Yu, Hoeseong Kim, and Gunhee Kim).

[0067] Existing Method 3: The CLIP4Cir method in the literature "Effective conditioned and composed image retrieval combining clip-based features" (authors Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo).

[0068] This invention: The method of this embodiment.

[0069] The experiment uses Recall@10, Recall@50, and their average, commonly used metrics in information retrieval, to evaluate the accuracy of the retrieval. Given a set of reference images and modified text, the system searches for the k closest target images. If the target image is included, the query score is 1; otherwise, it is 0. Recall@K is the average score of all queries in the test set. A higher Recall value indicates better retrieval results.

[0070] As shown in Table 1, this invention significantly improves upon R@10 and the average index. None of the three existing methods fully utilize the rich style information contained in e-commerce images. This invention models the style features of e-commerce images through a two-stage attention mechanism, as well as graph inference networks and filtering networks, and leverages intra-domain and inter-domain cross-modal alignment to mine the associations between reference image-modified text-target image triples, thus achieving better retrieval accuracy.

[0071] Table 1. Experimental results of the present invention

[0072] R@10 R@50 average Existing Method 1 24.15 46.61 35.40 Existing Method 2 27.78 53.89 40.84 Existing Method 3 35.39 59.03 47.21 This invention 37.78 60.64 49.21

[0073] In other embodiments, the method for extracting text features in step (1) of the present invention uses the CLIP pre-trained model to model text features, and can also use LSTM (Long-Short Term Memory) neural network and recurrent neural network (RNN) neural network as alternatives.

[0074] Another embodiment of the present invention provides a hierarchical style-based conditional text-e-commerce image retrieval system, comprising:

[0075] The common feature extraction module is used to interact between the average features of image patches and the features of image patches using a sliding window mechanism and a two-stage self-attention mechanism to obtain the common features of image patches.

[0076] The difference feature extraction module is used to fuse common features with image patch features through a graph inference network, and use a local region filtering network to filter out useless image patch features to obtain the difference features of image patches.

[0077] The feature fusion module is used to fuse common features and dissimilar features through low-level-high-level interaction to obtain style features;

[0078] The retrieval module uses style features as query examples and target images as query targets. It calculates the similarity between the query examples and the query targets, sorts the similarities from highest to lowest, and obtains a list of relevant results for the target modality image data based on the similarity.

[0079] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0080] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0081] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0082] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A hierarchical style-based conditional text retrieval method for e-commerce images, comprising the following steps: Establish a conditional text-e-commerce image database, containing triples of reference image-modified text-target image; Image patches of reference images in the conditional text-e-commerce image database are obtained through a sliding window mechanism. The common features of the image patches are obtained by interacting with the average features of the image patches through a two-stage self-attention mechanism. Common features are fused with image patch features through a graph inference network, and useless image patch features are filtered out using a local region filtering network to obtain the differential features of image patches. Style features are obtained by fusing common and dissimilar features through low-level-high-level interaction, and these features are used as query examples to retrieve target images. The two-stage self-attention mechanism is as follows: The image is segmented into multiple image patches, and features are extracted to obtain the image patch features p. i ; Using a simple self-attention mechanism, all image patch features p are processed. i The average as p r ; For p r p is obtained using a multi-head attention mechanism. m The common features p of image patches are calculated using the following formula. c : p c =f c ([p m ,f q (p r )]) Among them, f c and f q Both are fully connected layers; The method for constructing the differential features of the image patch is as follows: Each image patch feature p i and common features p c As graph nodes, points in the graph reasoning network are established using the following formulas. And edge e: Among them, W in and W out It is a fully connected layer, at the ingress node v in and outgoing node v out Information is exchanged between them; After constructing the graph inference network, the nodes and edges in the graph inference network are updated using the following formula: Where L represents the number of layers in the graph reasoning network, This represents a fully connected layer; after passing through a graph reasoning network, the common feature p c By fully associating with image patch features to complete information interaction, preliminary differential features are obtained. To reduce the interference of unimportant image patches on the results, a filtering network is designed to adaptively calculate the weights of image patches and update the filtered differential features. The calculation process is as follows: Where σ is the Sigmoid function, BN is the batch normalization operation, and W f For a fully connected layer, v i Image patch features; Two self-learning parameters, α and β, are introduced to balance and The importance of the difference features p was determined. d As shown in the following formula:

2. The method as described in claim 1, characterized in that, The conditional text-e-commerce image database contains triples of reference image-modification text-target image; the reference image is an e-commerce product image; the modification text contains suggestions for modification to color, size, and style; and the target image is an e-commerce product image that is closest to the requirements of the modification text, based on the reference image.

3. The method as described in claim 1, characterized in that, The two-stage self-attention mechanism trains a two-stage self-attention network on image patch features, mapping each image patch to a shared semantic space to obtain the common features of the image patches.

4. The method as described in claim 1, characterized in that, The process of fusing common and dissimilar features to obtain style features through low-level-high-level interaction involves fusing visual features containing style and textual features of modified text using intra-domain and inter-domain cross-modal alignment. This includes the following steps: a) Fuse common features and dissimilar features of image patches to obtain low-level interaction features p using the following formula. l : p l =f l ([f c (p c ),f d (p d )]) Among them, f l f c and f d It is a fully connected layer; b) By learning adaptive parameters θ, common and dissimilar features are fused to obtain style features p based on image patches. h , As shown in the following formula: θ=f θ ([f c (p c ),f d (p d )]) p h (f h (p. 100) l )+θ·p c +(1-θ)·p d Among them, f θ f h It is a fully connected layer; c) Style features p h Visual features V are obtained by concatenating with global image features. The style-inclusive visual features V and the modified text features T are then fused using intra-domain and inter-domain cross-modal alignment. Intra-domain attention features V′ and T′ are obtained through a self-attention mechanism, as shown in the following formula: Where MLP stands for fully connected layer network, m is the number of reference images and modified text, T is the text features of the modified text, and t is the number of modified text features. i Text features for each modified text; d) Learn two adaptive weights θ using the fusion methods of a) and b), respectively. self and θ com And obtain the fusion feature F final As shown in the following formula: F final =F self +F com in, and It is a fully connected layer network. These are interaction features within the lower-level domain. For low-level inter-domain interaction features, F self For high-level domain interaction characteristics, F com This refers to the interaction characteristics between higher-level domains.

5. The method as described in claim 1, characterized in that, The step of retrieving target images based on query samples involves using the features of target images in the conditional text-e-commerce image database as the query target, calculating the similarity between the query sample and the query target, and obtaining a list of relevant results for the target modality data based on the similarity.

6. A hierarchical style-based conditional text-e-commerce image retrieval system employing the retrieval method described in any one of claims 1 to 5, characterized in that, include: The common feature extraction module uses a sliding window mechanism and a two-stage self-attention mechanism to interact between the average features of the image patch and the features of the image patch to obtain the common features of the image patch. The difference feature extraction module fuses common features with image patch features through a graph inference network, and uses a local region filtering network to filter out useless image patch features to obtain the difference features of image patches. The feature fusion module is used to fuse common features and dissimilar features through low-level-high-level interaction to obtain style features; The retrieval module uses style features as query examples and target images as query targets. It calculates the similarity between the query examples and the query targets, sorts the similarities from highest to lowest, and obtains a list of relevant results for the target modality image data based on the similarity.

7. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the retrieval method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the retrieval method according to any one of claims 1 to 5.

Images