A cross-modal fashion commodity retrieval method and system based on multi-granularity feature fusion and a storage medium

Abstract

The application discloses a kind of multi-granularity feature fusion-based cross-modal fashion commodity retrieval method, system and storage medium, and the structured semantic enhancement module is converted into natural language description and extracts key visual attribute word sequence by commodity label;Attribute perception double-tower feature extraction network is constructed, wherein image tower adopts hierarchical visual Transformer to capture detail texture and global semantics, and text tower introduces attribute weighting layer based on static dictionary to strengthen visual related vocabulary;Independent pair probability alignment module is used, with pair Sigmoid loss function replacing traditional Softmax loss, independently optimizing the matching probability of each pair of image-text sample, significantly reducing memory occupation and adapting multi-label data;Attribute hard filtering and vector soft matching are fused in online retrieval stage, and retrieval accuracy and efficiency are considered.This application effectively improves the identification ability of fashion commodity fine-grained attribute and the accuracy and response speed of cross-modal retrieval.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, natural language processing and information retrieval technology, and specifically to a cross-modal fashion product retrieval method, system and storage medium based on multi-granularity feature fusion. Background Technology

[0002] With the rapid development of e-commerce, especially cross-border e-commerce in the fashion apparel sector, efficient and accurate product retrieval systems have become crucial for improving user experience and platform conversion rates. In this scenario, users often search for desired products by entering text descriptions or uploading reference images. This requires retrieval systems to possess strong cross-modal understanding and matching capabilities, that is, the ability to bridge the semantic gap between visual (images) and linguistic (text) communication.

[0003] Currently, the mainstream cross-modal fashion product retrieval technologies in the industry are mainly traditional retrieval methods based on keyword matching and cross-modal pre-trained model methods based on deep learning.

[0004] Traditional keyword-based retrieval methods primarily rely on structured tags (such as category, color, material, style, etc.) and title text manually or automatically labeled when products are listed. During retrieval, the system uses traditional information retrieval techniques such as inverted indexes to perform precise or fuzzy matching between user query terms and product tags. While this method is efficient and simple to implement, its inherent limitations are quite obvious. First, it heavily relies on the completeness and accuracy of the labeling and cannot understand semantic information beyond the tags. Second, it lacks semantic understanding capabilities and cannot handle synonyms, near-synonyms, or abstract descriptions (such as "commuter style"). Finally, for complex queries containing multiple attributes (such as "printed chiffon blouses suitable for vacation"), simple keyword combination matching cannot accurately measure the overall relevance between the product and the query, resulting in coarse recall results and limited precision.

[0005] To overcome the semantic limitations of keyword matching, researchers have introduced models such as CLIP (Contrastive Language-Image Pre-Training). These models perform contrastive learning on large-scale image-text pairs, mapping images and text to a unified vector space, making semantically similar image-text pairs closer in vector distance. During retrieval, the vector similarity between the query (text or image) and the products in the database is calculated separately. While this method has achieved significant success in general domains, it still has limitations when directly applied to fine-grained, complex fashion product retrieval. On the one hand, general pre-trained models lack the ability to perceive the professional attributes specific to the fashion field (such as fabric texture, cut, style, etc.), making it difficult to accurately capture the fine-grained visual features of products. On the other hand, to achieve high-quality generalization, standard contrastive learning requires extremely large batch sizes to cover sufficient negative samples, placing extremely high demands on computational memory. Furthermore, to clearly see the subtle textures and complex patterns of clothing, high-resolution images (e.g., 384x384 or higher) are required, which further leads to a quadratic increase in computational complexity. Under the constraint of limited engineering resources, this contradiction makes it difficult for the model to be effectively fine-tuned under high-resolution input, thus limiting its accuracy in capturing fine-grained features of commodities. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a cross-modal fashion product retrieval method, system, and storage medium based on multi-granularity feature fusion. This overcomes the deficiencies of existing technologies and can effectively improve the recognition capability of fine-grained attributes of fashion products, as well as the accuracy and response speed of cross-modal retrieval.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A cross-modal fashion product retrieval method based on multi-granularity feature fusion includes the following steps:

[0009] Step S1: Product data processing and semantic enhancement; obtain the image of the product to be retrieved and its corresponding structured attribute tags; input the structured attribute tags into the structured semantic enhancement module for processing, and output at least one piece of semantically enhanced text, the semantically enhanced text including a complete natural language description sentence generated by the global description generation unit and a sequence of key visual attribute words extracted by the attribute emphasis stream extraction unit;

[0010] Step S2: Multi-granularity feature extraction; Input the image of the product into an attribute-aware image encoder to extract image feature vectors; Input the semantically enhanced text into an attribute-aware text encoder to extract text feature vectors; Wherein, the attribute-aware image encoder adopts a hierarchical visual Transformer architecture based on a local window attention mechanism; The attribute-aware text encoder is a pre-trained language model based on Transformer and includes an attribute weighting layer, which is used to differentially weight word vectors according to the visual relevance of words;

[0011] Step S3: Model Training and Cross-Modal Alignment; Construct a training dataset consisting of product images and their corresponding semantically enhanced text pairs; Using this training dataset, jointly train the overall image encoder and text encoder from Step S2 using independent pairwise probability alignment; wherein, the independent pairwise probability alignment is optimized using a pairwise loss function based on the Sigmoid function, which independently calculates the matching probability of each pair of images and text in the training batch, and maps this probability to a binary classification output through the Sigmoid function, and independently optimizes the matching probability of the pair of images and text with binary cross-entropy loss, thereby updating the model parameters;

[0012] Step S4: Hybrid Retrieval: In response to the user's input retrieval request, if the request is text, key attributes are extracted for hard filtering, and the text is input into the text encoder trained in Step S3 to obtain the query text feature vector. Approximate nearest neighbor search is then performed in the attribute-filtered product vector pool. If the request is an image, the image is input into the image encoder trained in Step S3 to obtain the query image feature vector. Approximate nearest neighbor search is then performed in the full product vector database. The retrieval results are then returned.

[0013] Preferably, in step S1, the structured semantic enhancement module includes a global description generation unit and an attribute emphasis stream extraction unit;

[0014] The global description generation unit operates through a predefined rule template library and a string filling mechanism. Based on the fields contained in the structured attribute tags, it selects matching sentence templates from the rule template library and fills the field values ​​in the tags into the corresponding slots of the sentence templates to generate grammatically coherent natural language description sentences.

[0015] The attribute emphasis stream extraction unit operates through a pre-built visual attribute lexicon, which contains a set of adjectives strongly related to the visual appearance of the product; it matches the structured attribute tags with the visual attribute lexicon, extracts all matching key visual attribute words, and concatenates them into a sequence according to a predetermined format.

[0016] Preferably, in step S2, the attribute-aware image encoder specifically adopts the Swin Transformer model, which extracts features through a moving window self-attention mechanism and downsamples the feature map at different stages to simultaneously capture local detail features and global semantic features.

[0017] Preferably, in step S2, the attribute weighting layer assigns different weight coefficients to the embedding vectors of different words in the input text through a predefined weight mapping table. During the text encoding process, the embedding vector of each word is multiplied by the corresponding scalar weight obtained from the dictionary.

[0018] Preferably, in the weight mapping table, the weight of visually strongly related attribute words is greater than 1.0, and the weight of abstract descriptive words is less than 1.0.

[0019] Preferably, in step S3, the expression for the pairwise loss function is:

[0020]

[0021] in, and are the normalized image feature vector and text feature vector of the i-th image sample and the j-th text sample, respectively; t is the learnable temperature coefficient. For the Sigmoid function, For sample pairs The true matching label function takes a value of 1 when the i-th image matches the j-th text, and 0 otherwise.

[0022] Preferably, in step S4, the hard filtering specifically involves: parsing out terms that match preset key attribute categories from the search text input by the user, and filtering out a subset of products whose tags contain the terms in the database before vector retrieval, in order to narrow down the search scope.

[0023] This invention also discloses a cross-modal fashion merchandise retrieval system for implementing the above method, comprising:

[0024] The structured semantic enhancement module is used to receive the structured attribute tags of the product and output semantically enhanced text containing a global description sentence and a sequence of key attribute words;

[0025] An attribute-aware dual-tower feature extraction network, comprising an image encoder and a text encoder, is used to extract deep feature vectors of images and text, respectively.

[0026] The model training module integrates an independent pairwise probability alignment loss function for training the dual-tower feature extraction network;

[0027] The offline index building module is used to build a product image feature vector library using a trained image encoder, and to build an inverted index based on product tags;

[0028] An online hybrid search engine is used to receive and parse user queries, coordinate inverted index filtering and vector similarity calculation, complete the search task, and return the results.

[0029] Preferably, the online hybrid search engine includes:

[0030] The query parsing unit is used to identify key attribute words in the query text or extract feature vectors from the query image.

[0031] The initial screening and filtering unit, in text query scenarios, uses key attribute words and inverted indexes to quickly narrow down the range of candidate products;

[0032] The vector matching unit is used to calculate the similarity between the query vector and the feature vector of the candidate product;

[0033] The results synthesis and sorting unit is used to sort candidate products according to similarity and generate a final search results list.

[0034] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0035] This invention provides a cross-modal fashion product retrieval method, system, and storage medium based on multi-granularity feature fusion, which has the following beneficial effects: Through a structured semantic enhancement module, discrete tags from the e-commerce backend are transformed into coherent natural language descriptions, and key visual attribute words are extracted, effectively bridging the semantic gap between structured data and user natural language queries. This allows the model to more fully understand the multi-dimensional attributes of products. The attribute-weighted layer in the attribute-aware text encoder, and the hierarchical visual Transformer architecture for the image encoder capable of handling high-resolution input, significantly improve the model's sensitivity and discriminative power for fine-grained visual attributes, overcoming the shortcomings of traditional methods that focus on the global picture while neglecting local details. Through an independent pairwise probability alignment loss function, the training objective is decoupled from intra-batch normalized contrast to an independent binary classification task. This eliminates the dependence on extremely large batches, reducing training memory usage by approximately 30%-50%, thus supporting fine-tuning using higher-resolution images to capture clearer texture details under the same hardware conditions. Furthermore, the moving window attention mechanism employed by ImageTower reduces the computational complexity of self-attention from the quadratic level of the image size to the linear level, enabling efficient training on high-resolution inputs even with limited GPU memory. The collaborative work of the offline-built hybrid index and the online hybrid retrieval engine achieves a combination of "hard filtering" and "soft matching." During text retrieval, candidate sets are first quickly filtered using attribute tags, followed by vector similarity calculation. This strategy ensures accurate matching of key attributes such as color and category, while also achieving fuzzy matching of complex semantics such as style and design through vector retrieval. This significantly improves retrieval relevance while effectively controlling search latency, achieving the efficiency and effectiveness balance required for industrial applications. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of this invention or the prior art will be briefly introduced below.

[0037] Figure 1 The overall flowchart of the cross-modal fashion product retrieval method based on multi-granularity feature fusion in this invention;

[0038] Figure 2 Online retrieval flowchart of this invention;

[0039] Figure 3 Detailed flowchart of online hybrid retrieval (text-based image search) in this invention;

[0040] Figure 4 This invention presents an architecture diagram of a cross-modal fashion product retrieval system. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0042] Example 1, as Figures 1 to 3 As shown, a cross-modal fashion product retrieval method based on multi-granularity feature fusion is presented. This method is particularly suitable for image-based and text-based image-based search scenarios on fashion e-commerce platforms such as clothing and footwear, aiming to accurately match users' visual and semantic search intentions. This method mainly consists of two parts: an offline processing flow and an online retrieval flow. The offline processing flow is responsible for model training and data index construction. First, a cross-modal matching model is trained using historical product data (images + structured tags). The core of this model is an attribute-aware dual-tower feature extraction network, trained using an independent pairwise probability alignment module. After training, the image encoder in the model generates feature vectors for all products to be retrieved, and a vector index library and a tag-based inverted index are established. The online retrieval flow responds to users' real-time queries. For text queries, key attribute extraction and hard filtering are performed first, followed by vector similarity search; for image queries, vector similarity search is performed directly. The entire process is scheduled by a hybrid vector retrieval engine.

[0043] Specifically, the following steps are included:

[0044] Step 1: Product data processing and semantic enhancement;

[0045] Historical product data from the platform is collected, with each data entry containing a clear main product image and a structured set of attribute tags (JSON format). The structured attribute tags for each data entry are then input into the structured semantic enhancement module. This module comprises two parallel processing units: a global description generation unit and an attribute emphasis stream extraction unit.

[0046] The global description generation unit operates through a predefined rule template library and a string filling mechanism. Based on the fields contained in the structured attribute tags, it selects matching sentence templates from the rule template library and fills the field values ​​in the tags into the corresponding slots of the sentence templates to generate grammatically coherent natural language description sentences.

[0047] For example, template examples include: Template A: "A [Color] [Material] [Category] with [Pattern] pattern, featuring a [Neckline] neckline and [Sleeve_Length] sleeves, suitable for [Season]." Template B: "This is a [Season] [Category] made of [Material] in [Color]." The system selects the best-matching template (such as Template A) based on the fields present in the current tag. Then, it fills the tag values ​​into the slots, generating a global description: "A White Lace Dress with Solid pattern, featuring a V-neck neckline and Short sleeves, suitable for Summer." If a field (such as Pattern) is missing, an alternative template that does not contain that slot is selected.

[0048] The attribute emphasis stream extraction unit is connected to a pre-built visual attribute lexicon. This lexicon contains 500 high-frequency visual words that have been filtered by TF-IDF and manually reviewed, categorized into three types: "pattern" (e.g., floral, striped), "fabric" (e.g., chiffon, silk), and "cut" (e.g., v-neck, puff-sleeve). By scanning the structured attribute tags, all words matching the lexicon are extracted. For example, "Lace" and "V-neck" can be extracted from the tags. These words are then concatenated into an attribute emphasis sequence in the format "Tag Sequence:[word1], [word2], ...": "Tag Sequence: lace, v-neck".

[0049] Ultimately, the complete semantically enhanced text of the product consists of the aforementioned global descriptive sentence and attribute emphasis sequence, serving as the input on the text side.

[0050] Step 2: Construction of an attribute-aware dual-tower feature extraction network;

[0051] First, a dual-tower neural network model is constructed, consisting of an attribute-aware image encoder and an attribute-aware text encoder to process images and text respectively.

[0052] The image encoder uses the Swin Transformer-Base (Swin-B) model as its backbone network. The network comprises four stages. The input image (e.g., adjusted to 384x384 resolution) is first segmented into non-overlapping rectangular image patches. In each stage, a Shifted Window Multi-Head Self-Attention (MSA) module computes feature relationships within the local region, followed by patch merging (a downsampling operation) to expand the receptive field and increase the number of channels. This design allows Stages 1 / 2 to focus on local details such as cuff lace texture and button details, while Stages 3 / 4 can understand global semantic features such as the overall silhouette and length of the dress. The Shifted Window mechanism reduces the computational complexity of self-attention from quadratic to linear, thus supporting 384x384 or even higher resolution inputs with limited GPU memory.

[0053] Output: Take the global average pooling features from the last stage of the Swin Transformer, pass them through a projection layer, and then perform L2 normalization to obtain a 512-dimensional normalized image feature vector x.

[0054] The text encoder uses a BERT-Base pre-trained model. A lightweight attribute-weighted layer is added after the BERT model. This layer maintains a static token-weight mapping table, assigning weights to words, where visually relevant words are assigned 1.0, abstract words 0.5, and common words 1.0. For example, {“lace”: 1.5; “v-neck”: 1.5; “white”: 1.5; “dress”: 1.0; “summer”: 1.0; “with”: 1.0; “a”: 1.0; “suitable”: 0.5}. For each token in the input sequence, BERT outputs its word embedding vector. Then, the corresponding weight scalar is looked up according to the Token-weight mapping dictionary. (Unregistered terms have a default weight of 1.0). Then, a weighted operation is performed: This makes the model more sensitive to descriptive words about visual details.

[0055] Finally, mean pooling is performed on all weighted vectors, followed by a projection layer and L2 normalization to obtain the final 512-dimensional normalized text feature vector y.

[0056] Step 3: Model training based on independent pairwise probability alignment;

[0057] First, prepare the training dataset. ,in These are product images. It is its corresponding semantically enhanced text.

[0058] 31) Forward Propagation: For a mini-batch of data of size N, two sets of normalized feature vectors are obtained through the image encoder and the text encoder, respectively: and .

[0059] 32) Similarity Calculation and Probability Mapping: Calculate all possible image-text pairs within a batch. cosine similarity A learnable temperature coefficient t is introduced to scale the similarity scores. Then, a sigmoid function is applied to each scaled score. Map it to a matching probability between 0 and 1 .

[0060] 33) Loss Calculation: The pairwise sigmoid loss function is used. For each pair of samples... This is treated as an independent binary classification problem (match or mismatch). Define sample pairs. True matching label function ,like (i.e., the image and text are from the same product), then ,otherwise The loss for this sample pair is the binary cross-entropy:

[0061] ;

[0062] The loss for the entire batch is the average of all these independent losses:

[0063] ;

[0064] The loss function described above does not rely on the Softmax normalization of all samples within a batch, thus effectively decoupling the dependencies between samples. This significantly reduces memory usage during training, allowing for larger batches or higher resolution images. Furthermore, an image can maintain a high probability with multiple texts (positive samples) describing its different attributes simultaneously, perfectly suited to the multi-label coexistence characteristic of fashion products.

[0065] 34) Backpropagation and optimization: Calculating loss The parameters of the image encoder and text encoder are updated using optimizers such as AdamW relative to the gradients of all model parameters until the model converges.

[0066] Step 4: Offline index building;

[0067] 41) Vector Library Construction: After training, fix the parameters of the image encoder. Input all images of the products to be retrieved (e.g., 1 million items) into the image encoder sequentially to generate corresponding 512-dimensional feature vectors. These feature vectors and their corresponding product IDs are stored in a high-performance vector database (such as Milvus, FAISS) to construct an Approximate Nearest Neighbor (ANN) index.

[0068] 42) Inverted Index Construction: Simultaneously, based on the product's structured attribute tags (such as fields like Color, Material, Category, etc.), an inverted list is built for each attribute value. For example, for the attribute value "V-neck", its inverted list contains all product IDs with the collar type "V-neck". This index is used for subsequent "hard filtering".

[0069] Step 5: Online hybrid search;

[0070] Scenario A: Searching for images using text;

[0071] A1) The user enters the query text: "Find a vintage white lace dress for wedding".

[0072] A2) Query Parsing and Hard Filtering: The system first parses the query text and identifies key attribute words. For example, it identifies "white" (Color) and "lace" (Material). Using these words, it searches the inverted index to obtain a set of product IDs that simultaneously satisfy the conditions of "white color" and "lace material". This quickly narrows down the search scope and ensures accurate matching of basic attributes.

[0073] A3) Vectorization and Soft Matching: The complete query text is input into the trained text encoder to generate a query vector q. Then, a restricted ANN search is initiated in the vector database: specifying that the search should only be performed on the candidate set. Search within the corresponding product vectors and calculate the cosine similarity between the query vector q and these vectors.

[0074] A4) Result Return: The top-K most similar products (e.g., K=20) and their information (images, titles, prices, etc.) are returned to the user. Since semantics such as "vintage" and "for wedding" in the query are also encoded into the vector, the returned results not only match in terms of material and color, but also in style, which is closer to the semantics of "vintage wedding".

[0075] Scenario B: Image search;

[0076] B1) A user uploads a picture of a lace dress.

[0077] B2) Vectorization: The image is directly input into the trained image encoder to generate query vectors. .

[0078] B3) Vectorized retrieval: Searching and querying vectors within the full index of a vector database. The most similar vector.

[0079] B4) Result Return: Returns a list of visually similar products. Because the model is trained at high resolution and uses fine-grained feature fusion, it can effectively match details such as lace textures and skirt styles.

[0080] This invention, through a structured semantic enhancement module, transforms discrete tags from e-commerce backends into coherent natural language descriptions and extracts key visual attribute words, effectively bridging the semantic gap between structured data and user natural language queries. This enables the model to more fully understand the multi-dimensional attributes of products. The attribute-weighted layer in the attribute-aware text encoder, and the use of a layered Transformer capable of handling high-resolution input in the image encoder, significantly improve the model's sensitivity and discriminative power for fine-grained visual attributes, overcoming the shortcomings of traditional methods that neglect local details due to a focus on the global picture. By using the independent pairwise probability alignment (Sigmoid) loss function, the training objective is decoupled from intra-batch normalized contrast (Softmax) to an independent binary classification task. This eliminates the dependence on large batches, reducing training memory usage by approximately 30%-50%, thus supporting fine-tuning using higher resolution (e.g., 384x384) images under the same hardware conditions to capture clearer texture details. Furthermore, ImageTower employs a moving window attention mechanism, reducing the computational complexity of self-attention from quadratic to linear, enabling efficient training on high-resolution inputs even with limited GPU memory. The offline-built hybrid index (vector library + inverted index) works in conjunction with the online hybrid retrieval engine, combining "hard filtering" and "soft matching." During text retrieval, candidate sets are quickly filtered using attribute tags before vector similarity calculation. This strategy ensures accurate matching of key attributes such as color and category, while also achieving fuzzy matching of complex semantics like style and design through vector retrieval. This significantly improves retrieval relevance while effectively controlling search latency, achieving the efficiency and effectiveness required for industrial applications.

[0081] Example 2, as Figure 4As shown, the present invention also discloses a cross-modal fashion product retrieval system for implementing the method described in Embodiment 1, comprising: a structured semantic enhancement module, an attribute-aware dual-tower feature extraction network, a model training module, an offline index building module, and an online hybrid retrieval engine.

[0082] The structured semantic enhancement module, serving as the entry point to the data processing pipeline, is responsible for converting the structured attribute tags from the product backend into semantically rich text representations that the model can understand. Its implementation can be deployed as an independent microservice. Its input interface receives product attribute tags in JSON format.

[0083] The structured semantic enhancement module internally comprises a global description generation unit and an attribute emphasis stream extraction unit. The global description generation unit loads a predefined YAML-formatted rule template configuration file. This unit includes a template matching engine that selects the most suitable template based on the completeness of the input label fields and fills the slots with label values ​​using string interpolation to generate a fluent description. The attribute emphasis stream extraction unit connects to a MySQL or Redis database storing a pre-built visual attribute lexicon. This unit includes a fast matcher that scans all values ​​of the input labels, matches them against the lexicon to extract keywords such as "Silk" and "Striped," and formats them into attribute emphasis sequences. Finally, a complete semantically enhanced text is output, preserving both the global description and the attribute emphasis sequences for subsequent training or index building.

[0084] The attribute-aware dual-tower feature extraction network is the core computational model of the system. It consists of two parallel deep neural network encoders, specifically an image encoder and a text encoder, and is typically deployed on an inference server equipped with a GPU.

[0085] The image encoder uses either PyTorch or TensorFlow frameworks and loads a pre-trained SwinTransformer-Base model. The input image, after preprocessing (scaling to 384x384, normalization), is segmented into 16x16 patch sequences and input into the model. Feature output: The original model's classification head is removed, and the feature map output from the last stage (Stage 4) is taken. This feature map is then passed through a global average pooling layer and a learnable projection layer (linear layer, output dimension 512), and finally L2 normalized to output the image feature vector.

[0086] The text encoder is a modification of the BERT-Base model based on the Hugging Face Transformers library. A custom attribute-weighted layer is implemented after the BERT model. This layer loads a predefined weight mapping table (Pythondict or JSON file) and, during forward propagation, scales each token embedding output by the BERT element-wise (multiplying by the corresponding weight). Then, mean pooling is performed, followed by a projection layer (output dimension 512) and L2 normalization to output the text feature vector.

[0087] The model training module is responsible for driving the training process of the dual-tower network and integrating the training mechanism.

[0088] Its training task scheduling is as follows: the module accepts a training dataset configuration consisting of image-text pairs. The scheduler is responsible for data loading and distributed sampling (such as using DistributedSampler with multiple GPUs).

[0089] The implementation of the independent pairwise probabilistic alignment module is as follows: In each training iteration step, this module receives a batch of image feature vectors X and text feature vectors Y. First, it calculates the similarity matrix between all image-text pairs. ;in, Let Y be the transpose of the text feature vector Y. This is a matrix multiplication, the result of which is the dot product (inner product) of the image feature vector X and the text feature vector Y. Since the feature vectors are normalized, this dot product equals their cosine similarity, ranging from [−1, 1]. `t` is a learnable temperature coefficient used to scale the distribution of similarity values ​​to optimize the model's training dynamics. The similarity matrix is ​​then converted to a probability matrix `p` using the `torch.sigmoid` function. The loss calculation unit computes a pairwise Sigmoid loss based on the true matching matrix Z (1s on the diagonal, 0s elsewhere) using the `F.binary_cross_entropy` function. This loss is backpropagated to update the weights of the image encoder, text encoder, and temperature parameter `t`.

[0090] After training is complete, the final parameters, configuration files, and preprocessing logic of the image encoder and text encoder are packaged into a model version package and published to the model repository for index building and online service calls.

[0091] The offline index building module is responsible for constructing high-speed retrieval indexes for massive amounts of products. It includes a product image feature vector library and an inverted index built based on product tags.

[0092] The product image feature vector library pipeline is as follows: (1) Data reading: Read the image URLs and IDs of all products from the product database. (2) Batch inference: Call the published image encoder model service to generate 512-dimensional feature vectors for all products in batch inference mode. (3) Vector insertion: Write the (product ID, feature vector) pairs into a dedicated vector database (such as Milvus, Elasticsearch with vector plugin). Before insertion, create an index (such as IVF_FLAT, HNSW) according to the requirements of the vector database to accelerate the approximate nearest neighbor search.

[0093] The pipeline for building the tag inverted index is as follows: (1) Extract (product ID, attribute key-value pair) data from the product database in parallel. (2) Use the indexing engine of Apache Lucene or Elasticsearch to build an inverted list for each attribute value that has retrieval significance (such as Color:Blue, Material:Silk), and record all product IDs containing that attribute value.

[0094] The online hybrid retrieval engine receives and parses user queries, coordinates inverted index filtering and vector similarity calculation, completes the retrieval task, and returns the results. It includes: a query parsing unit, an initial screening and filtering unit, a vector matching unit, and a result synthesis and sorting unit.

[0095] The query parsing unit is used to identify key attribute words in the query text or extract feature vectors from the query image; specifically, it includes: (1) Text query: receiving user natural language queries (such as "blue striped silk blouse"). This unit integrates a lightweight keyword extraction model or a rule-based parser to identify terms related to predefined attribute categories (color, material, etc.). At the same time, it passes the complete query statement to the text encoder service to generate query vectors. (2) Image Query: Receive images uploaded by users and directly call the image encoder service to generate query vectors. .

[0096] The initial screening and filtering unit, in text query scenarios, utilizes key attribute words and inverted indexes to quickly narrow down the range of candidate products. Specifically, it includes: receiving attribute key-value pairs extracted by the query parsing unit; and initiating a multi-condition Boolean query on the tag inverted index to quickly retrieve the set of product IDs that simultaneously satisfy all specified attributes (e.g., material is silk and pattern is striped). This step is completed in milliseconds, narrowing the candidate set from millions to thousands or tens of thousands.

[0097] The vector matching unit is used to calculate the similarity between the query vector and the candidate product feature vector; specifically, it includes: (1) Text retrieval path: initiating an approximate nearest neighbor search with filtering conditions to the vector database. The query is... The filter condition is ID in The vector database calculates similarity in a limited candidate pool and returns the Top-K results. (2) Image retrieval path: Initiate an unfiltered approximate nearest neighbor search to the vector database, and the query is... Return the Top-K most similar vectors and their product IDs from the entire database.

[0098] The results synthesis and sorting unit is used to sort candidate products based on similarity and generate a final search results list. Specifically, it includes: receiving a list of (product ID, similarity score) returned by the vector matching unit; retrieving detailed product information (images, titles, prices, etc.) from the product center service based on the product ID; sorting the products in descending order of similarity score; encapsulating the information into a standard JSON API response format; and returning it to the front-end application.

[0099] Example 3: A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in Example 1.

[0100] In this embodiment, computer-readable storage medium refers to any entity or non-transitory medium capable of storing program code, including but not limited to:

[0101] Volatile memory: such as random access memory (RAM), used for temporary loading during program execution.

[0102] Non-volatile memory: such as read-only memory (ROM), solid-state drive (SSD), hard disk drive (HDD), USB flash drive, flash memory card, optical disc (CD-ROM, DVD), etc., used for long-term storage and distribution of programs.

[0103] Distributed storage systems: such as storage units in network attached storage (NAS), storage area networks (SAN), or cloud storage services.

[0104] The computer program stored on this medium contains a series of instructions that can be executed by a processor (such as a CPU or GPU). When these instructions are loaded into the memory of the computing device and executed by the processor, they will drive the computing device to fully run the cross-modal fashion product retrieval method based on multi-granularity feature fusion as defined in Example 1.

[0105] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal fashion product retrieval method based on multi-granularity feature fusion, characterized in that: Includes the following steps: Step S1: Product data processing and semantic enhancement; Obtain the image of the product to be searched and its corresponding structured attribute tags; The structured attribute labels are input into the structured semantic enhancement module for processing, and at least one piece of semantically enhanced text is output. The semantically enhanced text includes a complete natural language description sentence generated by the global description generation unit and a sequence of key visual attribute words extracted by the attribute emphasis stream extraction unit. The structured semantic enhancement module includes a global description generation unit and an attribute emphasis stream extraction unit; The global description generation unit operates through a predefined rule template library and a string filling mechanism. Based on the fields contained in the structured attribute tags, it selects matching sentence templates from the rule template library and fills the field values ​​in the tags into the corresponding slots of the sentence templates to generate grammatically coherent natural language description sentences. The attribute emphasis stream extraction unit operates through a pre-built visual attribute lexicon, which contains a set of adjectives strongly related to the visual appearance of the product; the structured attribute tags are matched with the visual attribute lexicon to extract all matching key visual attribute words, and then concatenated into a sequence according to a predetermined format. Step S2: Multi-granularity feature extraction; Input the image of the product into an attribute-aware image encoder to extract image feature vectors; Input the semantically enhanced text into an attribute-aware text encoder to extract text feature vectors; Wherein, the attribute-aware image encoder adopts a hierarchical visual Transformer architecture based on a local window attention mechanism; The attribute-aware text encoder is a pre-trained language model based on Transformer and includes an attribute weighting layer, which is used to differentially weight word vectors according to the visual relevance of words; The attribute-aware image encoder specifically adopts the Swin Transformer model, which extracts features through a moving window self-attention mechanism and downsamples the feature map at different stages to capture both local detail features and global semantic features simultaneously. The attribute weighting layer assigns different weight coefficients to the embedding vectors of different words in the input text through a predefined weight mapping table. During the text encoding process, the embedding vector of each word is multiplied by the corresponding scalar weight obtained from the dictionary. Step S3: Model Training and Cross-Modal Alignment; Construct a training dataset consisting of product images and their corresponding semantically enhanced text pairs; Using this training dataset, jointly train the image encoder and text encoder from Step S2 using independent pairwise probability alignment; wherein, the independent pairwise probability alignment is optimized using a pairwise loss function based on the Sigmoid function, which independently calculates the matching probability of each pair of images and text in the training batch, and maps the probability to a binary classification output through the Sigmoid function, and independently optimizes the matching probability of the pair of images and text with binary cross-entropy loss, thereby updating the model parameters; The expression for the pairwise loss function is as follows: ； in, and are the normalized image feature vector and text feature vector of the i-th image sample and the j-th text sample, respectively; t is the learnable temperature coefficient. For the Sigmoid function, For sample pairs The true matching label function takes a value of 1 when the i-th image matches the j-th text, and 0 otherwise; Step S4: Hybrid Retrieval: In response to the user's input retrieval request, if the request is text, key attributes are extracted for hard filtering, and the text is input into the text encoder trained in Step S3 to obtain the query text feature vector. Approximate nearest neighbor search is then performed in the attribute-filtered product vector pool. If the request is an image, the image is input into the image encoder trained in Step S3 to obtain the query image feature vector. Approximate nearest neighbor search is then performed in the full product vector database. The retrieval results are then returned.

2. The cross-modal fashion product retrieval method based on multi-granularity feature fusion according to claim 1, characterized in that: In the weight mapping table, the weight of visually strongly related attribute words is greater than 1.0, while the weight of abstract descriptive words is less than 1.

0.

3. The cross-modal fashion product retrieval method based on multi-granularity feature fusion according to claim 1, characterized in that: In step S4, the hard filtering specifically involves parsing the search text input by the user to extract terms that match the preset key attribute categories, and before vector retrieval, filtering out a subset of products in the database whose tags contain the terms to narrow down the search scope.

4. A cross-modal fashion merchandise retrieval system for implementing the method of any one of claims 1-3, characterized in that, include: The structured semantic enhancement module is used to receive the structured attribute tags of the product and output semantically enhanced text containing a global description sentence and a sequence of key attribute words; An attribute-aware dual-tower feature extraction network, comprising an image encoder and a text encoder, is used to extract deep feature vectors for images and text, respectively. The model training module integrates an independent pairwise probability alignment loss function for training the dual-tower feature extraction network; The offline index building module is used to build a product image feature vector library using a trained image encoder, and to build an inverted index based on product tags; An online hybrid search engine is used to receive and parse user queries, coordinate inverted index filtering and vector similarity calculation, complete the search task, and return the results.

5. The system according to claim 4, characterized in that: The online hybrid search engine includes: The query parsing unit is used to identify key attribute words in the query text or extract feature vectors from the query image. The initial screening and filtering unit, in text query scenarios, uses key attribute words and inverted indexes to quickly narrow down the range of candidate products; The vector matching unit is used to calculate the similarity between the query vector and the feature vector of the candidate product; The results synthesis and sorting unit is used to sort candidate products according to similarity and generate a final search results list.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 3.

Images