A graphic-text matching method based on clothing hierarchical feature representation and association learning

Abstract

The present invention relates to a graphic-text matching method for clothing hierarchical feature representation and association learning, belonging to the fields of computer vision and artificial intelligence. The present invention first constructs a hierarchical feature representation module comprising CNN, Faster-RCNN, cascaded pyramid network and RNN by combining a custom clothing structure, respectively extracting the global, style and structural features of the input clothing image and the description, subject and label features of the input clothing text, and performing hierarchical representation; then, combining quadruple comparison and association reasoning, the global and description, style and subject, and structure and label features of the clothing image and text are associated and learned, respectively obtaining three relationship vectors of global, style and structure labels; finally, the similarity is calculated based on the three relationship vectors for graphic-text matching, and the final clothing graphic-text matching result is obtained after optimized sorting. The present invention can be effectively used for cross-modal fashion retrieval with high accuracy and efficiency.

Description

Technical Field

[0001] The present invention relates to a graphic-text matching method for clothing layered feature representation and association learning, and belongs to the fields of computer vision and artificial intelligence. Background Art

[0002] The development of cross-modal retrieval technology in the field of clothing retrieval has greatly promoted online shopping for fashion apparel. Cross-modal retrieval aims to achieve mutual retrieval between two or more different modal data. Images and text, as the most important data in multimodality, have received widespread attention. Due to the diversity, heterogeneity, and complexity of multimodal data, semantic understanding of cross-modal data is particularly important for matching. In cross-modal fashion retrieval, clothing image-text matching requires not only focusing on local clothing information, but also on clothing structure and attributes, as well as semantic understanding between images and text, making clothing image-text matching difficult. Global-based methods map individual data features from each modality from their respective old spaces to a common isomorphic subspace and use Euclidean distance or cosine similarity in the new space to compare distances between different items. However, accuracy is affected by the accuracy of feature representation in the common space. Methods based on visual language pre-training models require the use of pre-trained models with a Transformer architecture with a large number of parameters. In contrast, local-based methods perform attentive matching on multiple object-level information of a single item, achieving alignment at a more fine-grained level. For example, among the known clothing image-text matching methods, Gao ( <Proceedings of the 43rd International ACM SIGIR Conference on Research andDevelopment in Information Retrieval> ,2020:2251-2260) proposed FashionBERT to achieve cross-modal fashion retrieval and use spatially evenly divided image blocks as image features. Zhuge (<Proceedings of theIEEE / CVF Conference on Computer Vision and Pattern Recognition> ,2021:12647-12657) continued cross-modal research in the fashion field using the BERT network, proposed the Kaleido-BERT visual-language pre-training model, and introduced the Kaleido strategy to generate image features of different scales, enabling Transformer to better learn multimodal representations in the fashion field. Kenneth (<SIGIR 2021Workshop on eCommerce.ACM.> ,2021) replaced the previous spatial segmentation method with multi-level feature representation, and achieved a significant improvement in image-text matching performance while reducing the number of parameters.<arXiv preprint arXiv:2207.08150,2022.> 2022)) proposed a novel fashion-centric visual-language representation learning framework FashionViL, which uses multi-view comparative learning for attribute classification.Among the known similarity reasoning methods, Gao (<IEEE Transactions on Pattern Analysis and MachineIntelligence> , 2020) used a graph reasoning network based on similarity pyramid and constructed similarity graph for reasoning to improve fashion retrieval. Diao(<Proceedings of the AAAI Conference on ArtificialIntelligence.> ,2021,35(2):1218-1226) Combining similarity graph reasoning and attention filtering, vector-based similarity is applied to multimodal data, improving the interpretability and performance of image-text matching. The above-mentioned known clothing image-text matching methods mainly focus on representing features from a global or single aspect and use Euclidean distance to measure items.

[0003] Due to the diversity of clothing styles, textures, structures, and accessories, and the limitations of existing methods in mining and associating clothing structure information and clothing attribute information, how to detect significant clothing information in images and match it with text is an urgent problem to be solved. The present invention uses a customized clothing structure to construct a hierarchical feature representation module including CNN, Faster-RCNN, cascaded pyramid network, and RNN. It extracts the global, style, and structural features of the input clothing image and the description, subject, and label features of the input clothing text, and performs hierarchical representation. Furthermore, through quadruple comparison and associative reasoning, it obtains three relationship vectors of global, style, and structural labels of the three layers of features: global and description, style and subject, and structure and label. These are used to achieve the final clothing image-text matching, effectively improving the precision and accuracy of cross-modal fashion retrieval. Summary of the Invention

[0004] The present invention provides a graphic-text matching method based on clothing hierarchical feature representation and association learning, which can be effectively used for cross-modal fashion retrieval, solve the problem of missing clothing information and semantics in graphic-text matching, and improve the accuracy of clothing graphic-text matching.

[0005] The solution of the present invention is: a graphic-text matching method for clothing layered feature representation and association learning, the specific steps of the method are as follows:

[0006] Step 1: By combining the customized clothing structure, a hierarchical feature representation module consisting of CNN, Faster-RCNN, cascaded pyramid network, and RNN is constructed to extract the global, style, and structural features of the input clothing image and the description, subject, and label features of the input clothing text, and perform hierarchical representation.

[0007] Step 2: Combine four-fold comparison and associative reasoning to learn the association between the three features of clothing images and text: global and description, style and subject, and structure and label, and obtain three relationship vectors: global, style, and structure label.

[0008] Step 3: Calculate the similarity based on the three relationship vectors to perform image-text matching, and obtain the final clothing image-text matching result after optimizing the sorting.

[0009] The specific process of Step 1 is as follows:

[0010] First, the input fashion clothing multimodal dataset D = {D1,...,D k ,...,D d Clothing image I in k ∈D k Japanese clothing text T k ∈D k , where d represents the number of image-text pairs in the dataset. A global feature extraction network combining CNN and RNN is used to perform the first-layer feature representation of clothing. By constructing a binary fusion with gate control, the original features f of the clothing image extracted are transformed into origin and the cropping feature f crop Generate global features f of clothing images whole ; For the extracted word feature sequence {w1,...,w j ,...,w o After global average pooling, the description feature g of the clothing text is obtained whole , o represents the number of words in the text.

[0011] Then, Faster-RCNN and attribute aggregation are combined to perform the second-layer feature representation of clothing. Based on the pre-trained Faster-RCNN, the input clothing image I k Perform style detection and generate category features f class , coordinate feature f position and the region of interest feature f r Clothing style characteristics style ={r1,...,r i ,...,r m}, Where [,] represents the splicing operation; then NLP dependency analysis tools are used to analyze the clothing text T k Get the subject of each sentence and its dependency relationship, combined with the word sequence {w1,...,w j ,...,w o}get the subject {o1,...,o j ,...,o x} and attribute dependencies {a1,...,aj ,...,a y}, and by defining attribute aggregation with gated attention, several attribute dependencies are fused to the corresponding subject to obtain the subject feature g subject ={s1,..,s j ,...,s n},in σ is the activation function, C is the number of attribute dependencies corresponding to the subject, W s is the mapping matrix.

[0012] Finally, the third level of clothing feature representation is performed by combining the customized clothing structure and cascade pyramid. The clothing structure based on the key points of the symmetrical area of the clothing (including the sleeves, collar, chest, waist, hips, and legs) is defined, and the cascade pyramid and the ResNet network without the fully connected layer are used to extract the features of the corresponding clothing structure area in the input clothing image to obtain the structural feature f structure ={l1,...,l i ,...,l p}, where p represents the number of structures, l i Represents the i-th clothing structure feature. Then, through NLP grammar analysis, the text adjectives and labels corresponding to the clothing image are mined to obtain two parts of label features respectively. and where k j It is obtained by concatenating the jth adjective code and its attribution code. t and q represent the number of clothing labels in the two parts respectively. After merging, the label feature of the clothing text is obtained.

[0013] The specific process of Step 2 is as follows:

[0014] First, based on the three-layer features of clothing images and text obtained in Step 1, namely, global and description, style and subject, structure and label, we define the InfoNCE loss function from image to text. Perform a quadruple comparison between each pair of eigenmodes, where τ is the temperature hyperparameter, is a set of negative examples of text that do not match I, sim(I,T + )=(W f f) T (W g+ g + ), W f With W g+ is the mapping matrix.

[0015] Secondly, by calculating the high-dimensional correlation where |·| 2The sum ||·||2 represents the element-by-element square difference and the second normal form, W is the parameter matrix, x and y are two eigenvectors, and the initial correlation relationship between the three layers of features is obtained: the initial global relationship R whole =s(f whole ,g whole ; W1), initial style relationship (where r = m × n, i∈m,j∈n), initial structure label relationship (where z = p × q, i∈p,j∈q).

[0016] Then, graph convolution is used to initialize the relationship R in the second layer. style , the third layer initial structure label relationship R stru-lab and all three-level relations R all =R whole ∪R style ∪R stru-lab Build the graph structure on it and get the style relationship graph G α =(V α ,E α ), structure label relationship diagram G β =(V β ,E β ) and merged relationship graph G all =(V all ,E all ), where V α =R style Represents the initial style relationship as graph G α The node set, E α Representation graph G α The edge set of Indicates; V β =R stru-lab Represents the initial structure label relationship as graph G β The node set, E β Representation graph G β The edge set of Indicates; V all =R all Represents the relationship between global, style and structure labels as graph G all The node set, E all Represents the edge set G of the graph all , using the adjacency matrix express.

[0017] Finally, define the graph convolution operation in Update the feature vector for the l+1th graph convolution of the i-th node, and is the learnable parameter matrix, A ij is the adjacency weight of the i-th node to the j-th neighbor node, N i is the set of neighboring nodes of node i, g j is the jth node vector in the neighborhood; first, the style relationship graph G α , structure label relationship diagram G β Perform the first associative reasoning to obtain a global reasoning relationship One-time inference style relationship and one-time inference structure label relationship Then merge the relationship graph G all Perform the second associative reasoning to obtain the secondary reasoning global relationship Secondary Reasoning Style Relationship and secondary reasoning structure label relationship By layered attention fusion reasoning, we obtain the global relationship vectors Style Relationship Vector and structure label relationship vector in, Given R={s1,...,s i ,...,s D}, the fusion weight is δ(·) represents the Sigmoid function, BN(·) represents batch normalization, and W m is a learnable transformation matrix.

[0018] The beneficial effects of the present invention are:

[0019] 1. Most existing methods are suitable for general use (social, daily life) but cannot meet the fashion industry's requirements for clothing style, structure, and attributes, and cannot capture specific clothing information. This invention utilizes layered clothing feature representation and corresponding association learning to mine specific clothing layered information such as clothing style, structure, and label, and associates them at three different granularities to address the problem of missing clothing information and semantics in image-text matching, thereby improving the accuracy of clothing image-text matching.

[0020] 2. Known methods have limitations in mining and characterizing clothing structure information and clothing attribute information. The present invention defines hierarchical information of clothing and performs feature representation of clothing's global, style and structural labels in three layers. Among them, style features and structural features significantly represent fashion semantics. Combined with global features, it can characterize clothing features from multiple aspects and further improve the representation granularity of both clothing image and text modal data.

[0021] 3. Most known methods directly calculate pooling or maximum cosine similarity for image-text association matching, resulting in the matching accuracy being affected by the representation accuracy of the modal feature vector in the public representation space. The present invention constructs multi-level, high-dimensional association relationships from feature vectors, obtains global, style and structural label association relationships from hierarchical feature representations, and combines graph reasoning to learn the final association degree, which can effectively associate images and texts, thereby improving the association degree and matching accuracy of clothing images and texts. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a flow chart of the present invention;

[0023] Figure 2 A flow chart showing the first level of features of the present invention;

[0024] Figure 3 A flow chart showing the second level features of the present invention;

[0025] Figure 4 The third level feature of the present invention is represented by a flow chart;

[0026] Figure 5 This is a flowchart of clothing association learning of the present invention;

[0027] Figure 6 This is an example diagram of the clothing image-text matching retrieval result of the present invention. DETAILED DESCRIPTION

[0028] Example 1: Figures 1-6 As shown in FIG, a method for image-text matching based on clothing hierarchical feature representation and association learning includes:

[0029] Step 1: By combining the customized clothing structure, a hierarchical feature representation module consisting of CNN, Faster-RCNN, cascaded pyramid network, and RNN is constructed to extract the global, style, and structural features of the input clothing image and the description, subject, and label features of the input clothing text, and perform hierarchical representation.

[0030] Step 2: Combine four-fold comparison and associative reasoning to learn the association between the three features of clothing images and text: global and description, style and subject, and structure and label, and obtain three relationship vectors: global, style, and structure label.

[0031] Step 3: Calculate the similarity based on the three relationship vectors to perform image-text matching, and obtain the final clothing image-text matching result after optimizing the sorting.

[0032] The specific process of Step 1 is as follows:

[0033] First, the input fashion clothing multimodal dataset D = {D1,...,Dk ,...,D d Clothing image I in k ∈D k Japanese clothing text T k ∈D k , where d represents the number of image-text pairs in the dataset. A global feature extraction network combining CNN and RNN is used to perform the first-layer feature representation of clothing. By constructing a binary fusion with gate control, the original features f of the clothing image extracted are transformed into origin and the cropping feature f crop Generate global features f of clothing images whole ; For the extracted word feature sequence {w1,...,w j ,...,w o After global average pooling, the description feature g of the clothing text is obtained whole , o represents the number of words in the text.

[0034] Then, Faster-RCNN and attribute aggregation are combined to perform the second-layer feature representation of clothing. Based on the pre-trained Faster-RCNN, the input clothing image I k Perform style detection and generate category features f class , coordinate feature f position and the region of interest feature f r Clothing style characteristics style ={r1,...,r i ,...,r m}, Where [,] represents the splicing operation; then NLP dependency analysis tools are used to analyze the clothing text T k Get the subject of each sentence and its dependency relationship, combined with the word sequence {w1,...,w j ,...,w o}get the subject {o1,...,o j ,...,o x} and attribute dependencies {a1,...,a j ,...,a y}, and by defining attribute aggregation with gated attention, several attribute dependencies are fused to the corresponding subject to obtain the subject feature g subject ={s1,..,s j ,...,s n},in σ is the activation function, C is the number of attribute dependencies corresponding to the subject, W s is the mapping matrix.

[0035] Finally, the third level of clothing feature representation is performed by combining the customized clothing structure and cascade pyramid. The clothing structure based on the key points of the symmetrical area of the clothing (including the sleeves, collar, chest, waist, hips, and legs) is defined, and the cascade pyramid and the ResNet network without the fully connected layer are used to extract the features of the corresponding clothing structure area in the input clothing image to obtain the structural feature f structure ={l1,...,l i ,...,l p}, where p represents the number of structures, l i Represents the i-th clothing structure feature. Then, through NLP grammar analysis, the text adjectives and labels corresponding to the clothing image are mined to obtain two parts of label features respectively. and where k j It is obtained by concatenating the jth adjective code and its attribution code. t and q represent the number of clothing labels in the two parts respectively. After merging, the label feature of the clothing text is obtained.

[0036] The specific process of Step 2 is as follows:

[0037] First, based on the three-layer features of clothing images and text obtained in Step 1, namely, global and description, style and subject, structure and label, we define the InfoNCE loss function from image to text. Perform a quadruple comparison between each pair of eigenmodes, where τ is the temperature hyperparameter, is a set of negative examples of text that do not match I, sim(I,T + )=(W f f) T (W g+ g + ), W f With W g+ is the mapping matrix.

[0038] Secondly, by calculating the high-dimensional correlation where |·| 2 The sum ||·||2 represents the element-by-element square difference and the second normal form, W is the parameter matrix, x and y are two eigenvectors, and the initial correlation relationship between the three layers of features is obtained: the initial global relationship R whole =s(f whole ,g whole ; W1), initial style relationship (where r = m × n, i∈m,j∈n), initial structure label relationship (where z = p × q, i∈p,j∈q).

[0039] Then, graph convolution is used to initialize the relationship R in the second layer. style , the third layer initial structure label relationship R stru-lab and all three-level relations R all =R whole ∪R style ∪R stru-lab Build the graph structure on it and get the style relationship graph G α =(V α ,E α ), structure label relationship diagram G β =(V β ,E β ) and merged relationship graph G all =(V all ,E all ), where V α =R style Represents the initial style relationship as graph G α The node set, E α Representation graph G α The edge set of Indicates; V β =R stru-lab Represents the initial structure label relationship as graph G β The node set, E β Representation graph G β The edge set of Indicates; V all =R all Represents the relationship between global, style and structure labels as graph G all The node set, E all Represents the edge set G of the graph all , using the adjacency matrix express.

[0040] Finally, define the graph convolution operation in Update the feature vector for the l+1th graph convolution of the i-th node, and is the learnable parameter matrix, A ij is the adjacency weight of the i-th node to the j-th neighbor node, N i is the set of neighboring nodes of node i, g j is the jth node vector in the neighborhood; first, the style relationship graph G α , structure label relationship diagram G β Perform the first associative reasoning to obtain a global reasoning relationship One-time inference style relationship and one-time inference structure label relationship Then merge the relationship graph G allPerform the second associative reasoning to obtain the secondary reasoning global relationship Secondary Reasoning Style Relationship and secondary reasoning structure label relationship By layered attention fusion reasoning, we obtain the global relationship vectors Style Relationship Vector and structure label relationship vector in, Given R={s1,...,s i ,...,s D}, the fusion weight is δ(·) represents the Sigmoid function, BN(·) represents batch normalization, and W m is a learnable transformation matrix.

[0041] Example 2: Figure 1 As shown, the specific steps of this method are as follows:

[0042] Step 1: First, input the fashion clothing multimodal dataset D = {D1,...,D k ,...,D d Clothing image I in k ∈D k Japanese clothing text T k ∈D k , where d represents the number of image-text pairs in the dataset, and a global feature extraction network combining CNN and RNN is used to perform the first-layer feature representation of clothing. Figure 2 As shown in the figure, CNN selects the ResNet101 network pre-trained on the ImageNet dataset without the fully connected layer, and RNN selects the bidirectional gated recurrent unit (bi-GRU). By constructing a binary fusion with gate, the original features of the extracted clothing image f origin and the cropping feature f crop Generate global features f of clothing images whole ; For the extracted word feature sequence {w1,...,w j ,...,w o After global average pooling, the description feature g of the clothing text is obtained whole , o represents the number of words in the text.

[0043] Then, if Figure 3 As shown in Figure 2, Faster-RCNN and attribute aggregation are combined to perform the second-layer feature representation of clothing. Based on the pre-trained Faster-RCNN, the input clothing image I k Perform style detection and generate category features f class , coordinate feature f position and the region of interest feature fr Clothing style characteristics style ={r1,...,r i ,...,r m}, Among them, [,] represents the splicing operation, and its style categories are shown in Table 1; then the NLP dependency analysis tool is used to analyze the clothing text T k Get the subject of each sentence and its dependency relationship, combined with the word sequence {w1,...,w j ,...,w o}get the subject {o1,...,o j ,...,o x} and attribute dependencies {a1,...,a j ,...,a y},like Figure 3 The structure of attribute aggregation is a star structure O={o i ,a i1 ,...,a ic}(o i Indicates the i-th subject, a ic represents the corresponding c-th attribute dependency), and by defining attribute aggregation with gated attention, several attribute dependencies are fused to the corresponding subject to obtain the subject feature g subject ={s1,..,s j ,...,s n},in σ is the activation function, C is the number of attribute dependencies corresponding to the subject, W s is the mapping matrix.

[0044] Table 1

[0045]

[0046]

[0047] Finally, the third layer of clothing feature representation is performed by combining the customized clothing structure and cascade pyramid. Figure 4 As shown in the figure, the clothing structure based on the key points of the symmetrical area of clothing is defined (including sleeves, collar, chest, waist, hips, and legs, as shown in Table 2). The cascade pyramid and the ResNet-50 network without the fully connected layer are used to extract the features of the corresponding clothing structure area in the input clothing image to obtain the structural features f structure ={l1,...,l i ,...,l p}, where p represents the number of structures, l iRepresents the i-th clothing structure feature. Then, through NLP grammar analysis, the text adjectives and labels corresponding to the clothing image are mined to obtain two parts of label features respectively. and where k j It is obtained by concatenating the jth adjective code and its attribution code. t and q represent the number of clothing labels in the two parts respectively. After merging, the label feature of the clothing text is obtained.

[0048] Table 2

[0049]

[0050] Step 2: First, based on the three-layer features of clothing images and text obtained in Step 1, namely, global and description, style and subject, structure and label, we define the InfoNCE loss function from image to text. Perform a quadruple comparison between each pair of eigenmodes, where τ is the temperature hyperparameter, is a set of negative examples of text that do not match I, sim(I,T + )=(W f f) T (W g+ g + ), W f With W g+ is the mapping matrix. Combining the above formula, the first level of contrastive learning is performed between the two modalities of image and text: Perform second contrastive learning in the image-text modality: (where g entity It is a text description feature that removes function words and retains content words), and performs third-level comparative learning between the first and second layer features: (in Perform the fourth level of contrastive learning between the first and third layer features: (in ).

[0051] Secondly, if Figure 5 As shown on the left, by calculating the high-latitude correlation where |·| 2 The sum ||·||2 represents the element-by-element square difference and the second normal form, W is the parameter matrix, x and y are two eigenvectors, and the initial correlation relationship between the three layers of features is obtained: the initial global relationship R whole =s(f whole ,g whole ; W1), initial style relationship (where r = m × n, i∈m,j∈n), initial structure label relationship (where z = p × q, i∈p,j∈q).

[0052] Then, graph convolution is used to initialize the relationship R in the second layer. style , the third layer initial structure label relationship R stru-lab and all three-level relations R all Build the graph structure on it and get the style relationship graph G α =(V α ,E α ), structure label relationship diagram G β =(V β ,E β ) and merged relationship graph G all =(V all ,E all ), where V α =R style Represents the initial style relationship as graph G α The node set, E α Representation graph G α The edge set of Indicates; V β =R stru-lab Represents the initial structure label relationship as graph G β The node set, E β Representation graph G β The edge set of Indicates; V all =R all =R whole ∪R style ∪R stru-lab Represents the relationship between global, style and structure labels as graph G all The node set, E all Represents the edge set G of the graph all , using the adjacency matrix Indicated by the relationship R # For graph G * The node set of V * =R # ,*∈{α,β,all},#∈{style,stru-lab,all},V all =R whole ∪R style ∪R stru-lab , and E * Representation graph G * The edge set of The adjacency matrix calculation steps are as follows: First, the similarity of the node pairs is calculated using the following formula: Where W q and Wk is the parameter matrix, h i and h j Taken from node feature V * , D w is the latitude of the feature. Then calculate the attention weight of node i to node j: represents the exponential function with the natural constant e as the base, N i represents the domain node set of node i), and finally the entire adjacency matrix is obtained using the attention weight: A ij =α ij .

[0053] Finally, define the graph convolution operation Update the feature vector for the l+1th graph convolution of the i-th node, and is the learnable parameter matrix, A ij is the adjacency weight of the i-th node to the j-th neighbor node, N i is the domain node set of node i; Figure 5 As shown, first, the style relationship diagram G α , structure label relationship diagram G β Perform the first associative reasoning to obtain a global reasoning relationship One-time inference style relationship and one-time inference structure label relationship Then merge the relationship graph G all Perform the second associative reasoning to obtain the secondary reasoning global relationship Secondary Reasoning Style Relationship and secondary reasoning structure label relationship By layered attention fusion reasoning, we obtain the global relationship vectors Style Relationship Vector and structure label relationship vector in, Given R={s1,...,s i ,...,s D}, the fusion weight is δ(·) represents the Sigmoid function, BN(·) represents batch normalization, and W m is a learnable transformation matrix.

[0054] Step 3, first, Figure 5 As shown on the right, the three relationship vectors obtained in Step 2 are concatenated and sent to the linear projection layer to obtain the correlation score for image-text matching, which is expressed as r score =W proj [r whole ,r style ,rstru-lab ], where W proj is a learnable mapping matrix.

[0055] Then, the bidirectional ranking loss is used to train the image-text matching model: L(I,T)=[γ-S rf (I,T)+S rf (I,T - )] + +[γ-S rf (I,T)+S rf (I - ,T)] + , where γ represents the threshold, S rf (·) represents the association prediction function implemented using the hierarchical association learning model.

[0056] Finally, after the entire model is trained, the model is used to calculate the matching score between the query input and the candidate data in the inference phase, and image-text matching is performed based on the score.

[0057] like Figure 6 As shown in the figure, the visual image text matching results of clothing data, the left side is the text query result obtained by inputting the image, and the right side is the image query result obtained by inputting the text; Figure 6 For example, the input image (a) yields three candidate texts. The first selected text segment fully matches the query image, describing the clothing style in the image as a denim-like jogger jacket, the color as blue, and the copper-colored button. The semantics of the long sleeves, jacket, and collar in the next two segments are also relevant to the input image. Figure 6 For example, in (f), the input text (f) yields three candidate images. From these candidate images, we observe that the inventive method successfully identifies semantics such as the style being a T-shirt, the structure being short sleeves, and the color being white. Furthermore, the image selected in the leftmost box best matches the query, successfully capturing the black chest logo, corroborating the structural label association in the inventive method. Other examples show numerous associations in terms of style, structure, and semantic labels, demonstrating the present invention's ability to effectively associate images and text.

[0058] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A graphic-text matching method based on clothing hierarchical feature representation and association learning, characterized by: The steps include: Step 1: By combining the customized clothing structure, a hierarchical feature representation module consisting of CNN, Faster-RCNN, cascaded pyramid network, and RNN is constructed to extract the global, style, and structural features of the input clothing image and the description, subject, and label features of the input clothing text, and perform hierarchical representation. Step 2: Combine four-fold comparison and associative reasoning to learn the association between the three features of clothing images and text: global and description, style and subject, and structure and label, and obtain three relationship vectors: global, style, and structure label. Step 3: Calculate similarity based on the three relationship vectors to perform image-text matching, and obtain the final clothing image-text matching result after optimizing the sorting; The specific process of Step 1 is as follows: First, the input fashion clothing multimodal dataset D = {D1,...,D k ,...,D d Clothing image I in k ∈D k Japanese clothing text T k ∈D k , where d represents the number of image-text pairs in the dataset. A global feature extraction network combining CNN and RNN is used to perform the first-layer feature representation of clothing. By constructing a binary fusion with gate control, the original features of the clothing image f are extracted origin and the cropping feature f crop Generate global features f of clothing images whole ; For the extracted word feature sequence {w1,...,w j ,...,w o After global average pooling, the description feature g of the clothing text is obtained whole , o represents the number of words in the text; Then, Faster-RCNN and attribute aggregation are combined to perform the second-layer feature representation of clothing; based on the pre-trained Faster-RCNN, the input clothing image I k Perform style detection and generate category features f class , coordinate feature f position and the region of interest feature f r Clothing style characteristics style ={r1,...,r i ,...,r m }, Where [,] represents the splicing operation; then NLP dependency analysis tools are used to analyze the clothing text T k Get the subject of each sentence and its dependency relationship, combined with the word sequence {w1,...,w j ,...,w o }get the subject {o1,...,o j ,...,o x } and attribute dependencies {a1,...,a j ,...,a y }, and by defining attribute aggregation with gated attention, several attribute dependencies are fused to the corresponding subject to obtain the subject feature g subject ={s1,..,s j ,...,s n },in σ is the activation function, C is the number of attribute dependencies corresponding to the subject, W s is the mapping matrix; Finally, the third layer of clothing feature representation is performed by combining the customized clothing structure and cascade pyramid. The clothing structure based on the key points of the symmetrical area of the clothing is defined, including the sleeves, collar, chest, waist, hips, and legs. The cascade pyramid and the ResNet network without the fully connected layer are used to extract the features of the corresponding clothing structure area in the input clothing image to obtain the structural feature f structure ={l1,...,l i ,...,l p }, where p represents the number of structures, l i represents the i-th clothing structure feature; Then, NLP grammar analysis is used to mine the text adjectives and labels corresponding to the clothing images, and two parts of label features are obtained respectively. and where k j It is obtained by concatenating the jth adjective code and its attribution code. t and q represent the number of clothing labels in the two parts respectively. After merging, the label feature of the clothing text is obtained. The specific process of Step 2 is as follows: First, based on the three-layer features of clothing images and text obtained in Step 1, namely, global and description, style and subject, structure and label, we define the InfoNCE loss function from image to text. Perform a quadruple comparison between each pair of eigenmodes, where τ is the temperature hyperparameter, is a set of negative examples of text that do not match I, sim(I,T + )=(W f f) T (W g+ g + ), W f With W g+ is the mapping matrix; Secondly, by calculating the high-dimensional correlation where |·| 2 The sum ||·||2 represents the element-by-element square difference and the second normal form, W is the parameter matrix, x and y are two eigenvectors, and the initial correlation relationship between the three layers of features is obtained: the initial global relationship R whole =s(f whole ,g whole ; W1), initial style relationship Where r = m × n, Initial structure label relationship Where z = p × q, Then, graph convolution is used to initialize the relationship R in the second layer. style , the third layer initial structure label relationship R stru-lab and all three-level relations R all =R whole ∪R style ∪R stru-lab Build the graph structure on it and get the style relationship graph G α =(V α ,E α ), structure label relationship diagram G β =(V β ,E β ) and merged relationship graph G all =(V all ,E all ), where V α =R style Represents the initial style relationship as graph G α The node set, E α Representation graph G α The edge set of Indicates; V β =R stru-lab Represents the initial structure label relationship as graph G β The node set, E β Representation graph G β The edge set of Indicates; V all =R all Represents the relationship between global, style and structure labels as graph G all The node set, E all Represents the edge set G of the graph all , using the adjacency matrix express; Finally, define the graph convolution operation in Update the feature vector for the l+1th graph convolution of the i-th node, and is the learnable parameter matrix, A ij is the adjacency weight of the i-th node to the j-th neighbor node, N i is the set of neighboring nodes of node i, g j is the jth node vector in the neighborhood; first, the style relationship graph G α , structure label relationship diagram G β Perform the first associative reasoning to obtain a global reasoning relationship One-time inference style relationship and one-time inference structure label relationship Then merge the relationship graph G all Perform the second associative reasoning to obtain the secondary reasoning global relationship Secondary Reasoning Style Relationship and secondary reasoning structure label relationship By layered attention fusion reasoning, we obtain the global relationship vectors Style Relationship Vector and structure label relationship vector in, Given R={s1,...,s i ,...,s D }, the fusion weight is δ(·) represents the Sigmoid function, BN(·) represents batch normalization, and W m is a learnable transformation matrix.

Images