Clothing image retrieval method and device based on local feature block matching, equipment and storage medium

Abstract

The application discloses a garment image retrieval method and device based on local feature block matching, equipment and storage medium, the method comprises: global feature extraction is carried out to the query image, and the initial image in the database is matched based on the global feature, and the first image sequence containing multiple candidate images is obtained by screening, the local feature block in the image pair composed of the query image and each candidate image is mined through the reordering network, the reordering similarity score is obtained, the first image sequence is reordered based on the reordering similarity score, and the target garment image matched with the query image is determined based on the second image sequence reordered; since the application pays attention to the dependence relationship between the internal features of the image and the local matching information between the image pairs, the perception ability of the local difference in the garment image is enhanced, the inter-class false matching is reduced, the local feature retrieval ability of the occluded image is improved, and accurate and effective retrieval in a large-scale garment image dataset is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of clothing image retrieval, and in particular to a clothing image retrieval method and device based on local feature block matching, an equipment and a storage medium. BACKGROUND

[0002] In recent years, with the increasing demand for efficient and accurate visual retrieval systems in various industries, image retrieval tasks have received extensive attention. Image retrieval technology can be used in augmented reality applications, smart home devices, and e-commerce, etc. With the rapid development of e-commerce, the online sales of clothing products have significantly expanded, and the amount of clothing image data on the network has increased dramatically. How to accurately retrieve target clothing images from massive clothing image data has become a popular research topic. Currently, clothing image retrieval mainly includes two approaches: text-based image retrieval (i.e., "text-to-image") and content-based image retrieval (i.e., "image-to-image"). "Text-to-image" relies on manual annotation, and users only need to input relevant keywords of the clothing in the search box to filter out the corresponding clothing images. Therefore, it has the disadvantages of unclear description and strong subjectivity, which makes the accuracy of the retrieval results not high. "Image-to-image" does not rely on text annotation, mainly analyzes the visual content (color, texture, shape, target, etc.), extracts image features, and compares the similarity with the images in the database to automatically retrieve and return the most similar set of clothing pictures. "Image-to-image" avoids the subjectivity of manual annotation and improves the accuracy of retrieval, becoming the mainstream technology in the field of clothing image retrieval.

[0003] However, clothing image retrieval has the problem of fine-grained recognition, which means that due to the small difference between image classes and the large difference within the same class, different classes can only be distinguished by small differences, and individuals of the same class may have occlusion, deformation, etc. due to different shooting conditions such as different imaging distances, observation angles, backgrounds, lighting and weather conditions, showing large visual differences. Therefore, the current content-based image retrieval technology is limited by the fine-grained feature distinction difficulty and the serious image occlusion caused by different shooting angles and posture changes in the clothing image retrieval scene, which cannot effectively extract high-discriminative feature representations of images, resulting in the inability to accurately and effectively retrieve in large-scale clothing image datasets. SUMMARY

[0004] The main purpose of the present application is to provide a clothing image retrieval method and device based on local feature block matching, an equipment and a storage medium, which aims to solve the technical problem that the prior art cannot effectively extract high-discriminative feature representations of clothing images, resulting in the inability to accurately and effectively retrieve in large-scale clothing image datasets.

[0005] To achieve the above object, the application provides a garment image retrieval method based on local feature block matching, which comprises the following steps:

[0006] inputting a query image into a global retrieval module to extract global features and obtain first global features;

[0007] performing similarity matching between the first global features and second global features of each initial image in a database to obtain a global matching result, wherein the second global features are obtained by pre-extracting global features of the initial image by the global retrieval module;

[0008] selecting multiple candidate images from the initial images based on the global matching result, and sorting the candidate images based on global similarity between each candidate image and the query image to obtain a first image sequence;

[0009] pairing the query image with each candidate image in the first image sequence to generate multiple image pairs;

[0010] inputting the multiple image pairs into a reordering network to obtain a reordering similarity score of each image pair, wherein the reordering network comprises a local feature block matching module, and the local feature block matching module is used for performing local feature matching between the query image and the candidate image in each image pair;

[0011] reordering the first image sequence based on the reordering similarity score to obtain a second image sequence, and determining a target garment image matched with the query image based on the second image sequence.

[0012] Optionally, the reordering network further comprises a spatial relative attention module;

[0013] the spatial relative attention module is used for performing spatial enhancement processing on the image pairs input into the reordering network to output spatial enhancement features;

[0014] the spatial enhancement processing process of the spatial relative attention module comprises:

[0015] inputting an original feature into a convolution layer, generating a first initial feature and a second initial feature based on an original feature dimension of the original feature, and the original feature is a local image feature of an image pair input into the reordering network;

[0016] calculating spatial attention based on the first initial feature and the second initial feature to obtain spatial position attention weights of the image pair;

[0017] performing spatial enhancement processing on the image pair based on the spatial position attention weights to output spatial enhancement features.

[0018] Optionally, the reordering network further comprises a bidirectional attention module;

[0019] The bidirectional attention module is configured to perform feature iterative optimization processing on the spatial enhanced features output by the spatial relative attention module, and output a pair of bidirectional attention features when the number of iterations reaches an optimization number threshold, the pair of bidirectional attention features comprising bidirectional attention output features of the query image and the candidate image.

[0020] The processing process of the bidirectional attention module comprises a self-attention stage and a cross-attention stage.

[0021] The processing process of the self-attention stage comprises:

[0022]

[0023]

[0024]

[0025]

[0026] wherein, represents an intermediate feature of the self-attention stage, represents layer normalization, represents a multi-head self-attention unit, , and represent a query matrix, a key matrix and a value matrix respectively, which are obtained by linear transformation of the input feature of the self-attention stage, represents connecting the feature and the layer normalization output result in residual connection, represents a target output feature of the self-attention stage, represents a feedforward network, which is composed of a multi-layer perceptron, represents connecting the intermediate feature and the output result of the feedforward network in residual connection, represents a feature splicing operation, represents the output of the th attention head, represents the number of attention heads, represents attention, represents an output projection weight matrix, , and represent the query matrix, the key matrix and the value matrix of the th attention head respectively, represents a scaled dot-product attention function.

[0027] The processing procedure of the cross-attention stage includes:

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] wherein, denotes the self-attention output of the query image , , and denote the cross-attention parameters of the query image, the cross-attention parameters including a query matrix, a key matrix and a value matrix, , and denote the cross-attention weight matrix of the query image, denotes the self-attention output of the candidate image , , and denote the cross-attention parameters of the candidate image, , and denote the cross-attention weight matrix of the candidate image, denotes the cross-attention output of the query image, denotes the target output feature of the cross-attention stage of the query image, denotes the intermediate feature of the cross-attention stage of the candidate image, denotes the target output feature of the cross-attention stage of the candidate image.

[0035] Optionally, the local feature matching procedure of the local feature block matching module includes:

[0036] calculating the spatial position dot product between the bidirectional attention output feature of the query image and the bidirectional attention output feature of the candidate image in the bidirectional attention feature pair output by the bidirectional attention module, and generating an initial confidence matrix based on the spatial position dot product;

[0037] performing Softmax normalization on the initial confidence matrix along the row and column respectively to obtain a bidirectional confidence matrix;

[0038] perform local feature matching on the local feature blocks of the query image and the candidate image based on a preset similarity threshold and the bidirectional confidence matrix, to obtain a local feature matching result, the local feature matching result being used to calculate a re-ranking similarity score of the image pair, the local feature matching result including a matching set and matching confidence of each matching pair in the matching set, bidirectional confidence of the feature blocks matched with each other in each matching pair in the matching set being higher than the preset similarity threshold, and the matching pair being composed of the local feature blocks matched with each other between the query image and the candidate image.

[0039] Optionally, the re-ranking network further includes a similarity score module.

[0040] The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, to determine the re-ranking similarity score of the image pair.

[0041] The similarity scoring process of the similarity score module includes:

[0042] The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, to determine the re-ranking similarity score of the image pair.

[0043] The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, to determine the re-ranking similarity score of the image pair.

[0044] Optionally, the similarity score module is configured to aggregate the matching confidence of each matching pair in the matching set based on the local feature matching result output by the local feature block matching module, to obtain an aggregated confidence.

[0045] The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, to determine the re-ranking similarity score of the image pair.

[0046] The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, to determine the re-ranking similarity score of the image pair.

[0047] The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, to determine the re-ranking similarity score of the image pair.

[0048] Optionally, the global retrieval module is constructed based on a Swin-Transformer model.

[0049] The loss function of the global retrieval module is a contrastive loss function, and a mathematical expression of the contrastive loss function is:

[0050]

[0051] wherein, represents a contrastive loss function, is a sample batch size, is a sample class label, and respectively represent a sample index, is a cosine similarity between feature vectors of a positive sample image pair, is a cosine similarity between feature vectors of a negative sample image pair, and are boundaries of a positive sample pair and a negative sample pair, respectively.

[0052] In addition, to achieve the above object, the application further provides a garment image retrieval device based on local feature block matching, which comprises:

[0053] a feature extraction module, configured to input a query image into a global retrieval module to perform global feature extraction and obtain first global features;

[0054] a global matching module, configured to perform similarity matching between the first global features and second global features of each initial image in a database, to obtain a global matching result, wherein the second global features are obtained by performing global feature extraction on the initial images by the global retrieval module in advance;

[0055] an image sorting module, configured to filter a plurality of candidate images from the initial images based on the global matching result, and sort the candidate images based on global similarities between the candidate images and the query image, to obtain a first image sequence;

[0056] an image pairing module, configured to pair the query image with each candidate image in the first image sequence respectively, to generate a plurality of image pairs;

[0057] a reordering and scoring module, configured to input the plurality of image pairs into a reordering network to obtain a reordering similarity score of each image pair, wherein the reordering network comprises a local feature block matching module, and the local feature block matching module is configured to perform local feature matching on the query image and the candidate image in each image pair;

[0058] a target image retrieval module, configured to reorder the first image sequence based on the reordering similarity score, to obtain a second image sequence, and determine a target garment image matched with the query image based on the second image sequence.

[0059] In addition, to achieve the above object, the application further provides a garment image retrieval device based on local feature block matching, which comprises a memory, a processor and a garment image retrieval program based on local feature block matching stored in the memory, the processor is used for running the garment image retrieval program based on local feature block matching, and the computer program is configured to realize the steps of the garment image retrieval method based on local feature block matching as described above.

[0060] In addition, to achieve the above object, the application further provides a computer readable storage medium, which stores a computer program, and the computer program realizes the steps of the garment image retrieval method based on local feature block matching when executed by a processor.

[0061] In addition, to achieve the above object, the application further provides a computer program product, which comprises a computer program, and the computer program realizes the steps of the garment image retrieval method based on local feature block matching when executed by a processor.

[0062] The application extracts the global feature of the query image and performs global retrieval to obtain the first image sequence preliminarily matched with the query image, performs local feature block matching through the reordering network, and outputs the reordering similarity score, thereby fully mining the local feature in the garment image, enhancing the perception ability of the local difference in the garment image, reducing the inter-class false matching, improving the local feature retrieval ability of the occluded image, and realizing accurate and effective retrieval in a large-scale garment image dataset. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

[0064] Figure 1 It is the structural schematic diagram of the garment image retrieval device based on local feature block matching of the hardware running environment involved in the embodiment scheme of the present application.

[0065] Figure 2 It is the flowchart of the first embodiment of the garment image retrieval method based on local feature block matching of the present application.

[0066] Figure 3 It is the processing process schematic diagram of the spatial relative attention module in an embodiment of the garment image retrieval method based on local feature block matching of the present application.

[0067] Figure 4 It is the schematic diagram of the relative position coding in an embodiment of the garment image retrieval method based on local feature block matching of the present application.

[0068] Figure 5 It is the structural schematic diagram of the bidirectional attention module in an embodiment of the garment image retrieval method based on local feature block matching of the present application.

[0069] Figure 6 It is the dual-softmax calculation process schematic diagram in an embodiment of the garment image retrieval method based on local feature block matching of the present application.

[0070] Figure 7 It is the position importance weight example diagram in an embodiment of the garment image retrieval method based on local feature block matching of the present application.

[0071] Figure 8 It is the processing process schematic diagram of the garment image retrieval reordering model in an embodiment of the garment image retrieval method based on local feature block matching of the present application.

[0072] Figure 9 It is the schematic diagram of the hard negative sample pair in an embodiment.

[0073] Figure 10 It is the comparative schematic diagram of the garment image retrieval results of different methods in an embodiment.

[0074] Figure 11 It is the structural block diagram of the first embodiment of the garment image retrieval device based on local feature block matching of the present application.

[0075] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0076] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0077] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a clothing image retrieval device based on local feature block matching, which is part of the hardware operating environment of the embodiment of the present invention.

[0078] like Figure 1 As shown, the clothing image retrieval device based on local feature block matching may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0079] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on clothing image retrieval devices based on local feature block matching, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0080] like Figure 1 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a clothing image retrieval program based on local feature block matching.

[0081] exist Figure 1The network interface 1004 is mainly used for data communication with a network server; the user interface 1003 is mainly used for data interaction with a user; the processor 1001 and the memory 1005 in the garment image retrieval device based on local feature block matching can be arranged in the garment image retrieval device based on local feature block matching, the garment image retrieval device based on local feature block matching calls the garment image retrieval program based on local feature block matching stored in the memory 1005 through the processor 1001, and executes the garment image retrieval method based on local feature block matching provided by the embodiment of the application.

[0082] The embodiment of the application provides a garment image retrieval method based on local feature block matching. Figure 2 , Figure 2 The embodiment of the application provides a garment image retrieval method based on local feature block matching.

[0083] In the embodiment, the garment image retrieval method based on local feature block matching comprises the following steps:

[0084] Step S10: inputting a query image into a global retrieval module to perform global feature extraction, and obtaining first global features.

[0085] It should be noted that the embodiment is applied to garment image retrieval, and the garment image retrieval has a fine-grained recognition problem, that is, different classes can only be distinguished through small differences due to small inter-class differences and large intra-class differences, and individuals of the same class can have occlusion, deformation and other problems due to different shooting conditions such as different imaging distances, observation angles, backgrounds, illuminations and weather conditions, and show large visual differences.

[0086] Therefore, in order to accurately and effectively retrieve in a large-scale garment image dataset, the embodiment obtains robust global features by performing preliminary global feature extraction on the image through the global retrieval module, improves the accuracy of the initial retrieval result, performs local feature block matching through the reordering network, and outputs a reordering similarity score, thereby fully mining local features in the garment image, enhancing the perception ability of local differences in the garment image, reducing inter-class false matching, improving the local feature retrieval ability of the occlusion image, and achieving accurate and effective retrieval in a large-scale garment image dataset.

[0087] It should be understood that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a terminal electronic device capable of realizing the above functions, etc. The following takes a garment image retrieval device based on local feature block matching (referred to as retrieval device) as an example to illustrate the embodiment and the following embodiments.

[0088] It should be noted that the query image can be a garment-related image (such as a garment photo, a screenshot, etc. taken by a user) input by a user for finding a matching object. The global feature can be a high-dimensional vector representation obtained after the image is processed by a deep network, which can capture the overall semantic information of the image, and is a feature vector representing the overall attribute of the image, covering macro information such as color distribution, overall contour, main texture, style keynote, etc.

[0089] It should be noted that the global retrieval module is constructed based on a deep learning neural network. The first global feature can be a global feature vector extracted by the global retrieval module from the query image.

[0090] Further, in order to improve the global feature extraction efficiency, help the network learn a more robust global representation, and avoid detail loss, the global retrieval module is constructed based on a Swin-Transformer model.

[0091] The loss function of the global retrieval module is a contrastive loss function, and the mathematical expression of the contrastive loss function is:

[0092]

[0093] wherein, represents the contrastive loss function, is a sample batch size, is a sample category label, and respectively represent sample indexes, is a cosine similarity between feature vectors of a positive sample image pair, is a cosine similarity between feature vectors of a negative sample image pair, and are boundaries of the positive sample pair and the negative sample pair, respectively.

[0094] It should be noted that the global retrieval module adopts the contrastive loss function. The contrastive loss maximizes the cosine similarity of the positive sample pair while minimizing the similarity of the negative sample pair, which helps the network learn a more robust global representation.

[0095] In a specific implementation, a single image For input, a Swin Transformer model is used as a backbone network to extract global features For global image retrieval. Swin Transformer uses swin blocks to sequentially perform window-based multi-head self-attention (Win-MSA) and shift window division, which can effectively model the global interaction between each local point. Swin Transformer introduces a hierarchical structure that is more suitable for integrating details from different scales and capturing robust global semantic information on a single-scale input image. In this embodiment, the cosine similarity between the global features of the query image and the candidate image is ranked as the ranking result of global image retrieval.

[0096] Step S20: Similarity matching between the first global feature and the second global feature of each initial image in the database is performed to obtain a global matching result.

[0097] It should be noted that the second global feature is obtained by pre-extracting the global feature of the initial image by the global retrieval module.

[0098] It should be noted that the initial image can be a set of clothing images pre-stored in the database for matching with the query image. The second global feature can be a global feature vector pre-extracted and stored by the global retrieval module for all initial images. The global matching result can be a set containing the global similarity score of the query image and each initial image.

[0099] In some embodiments, similarity matching can be to calculate the similarity between the query image and the global feature vector of each initial image by a specific algorithm (such as cosine similarity, Euclidean distance, Manhattan distance, etc.), thereby quantifying the correlation between images, calling all second global features stored in the database, and calculating the similarity one by one with the first global feature, recording the similarity score of each matching group, and forming a complete global matching result.

[0100] Step S30: Based on the global matching result, multiple candidate images are selected from the initial images, and the candidate images are ranked based on the global similarity between each candidate image and the query image to obtain a first image sequence.

[0101] It should be noted that the candidate image can be an image selected from the initial image that meets the set condition in terms of global similarity with the query image. The global similarity can be a quantitative value reflecting the similarity of the image as a whole, which is calculated by the global feature. The first image sequence can be an image queue formed by ranking the candidate images according to the global similarity with the query image.

[0102] ​In some embodiments, the retrieval device can set a global similarity threshold (such as a cosine similarity ≥ 0.5), filter out the initial images above the threshold as candidate images, arrange all the candidate images in descending order of global similarity scores, the higher the score, the higher the ranking, and output the ranked queue as the first image sequence, the sequence length can be set according to requirements (such as the first 100, the first 200).

[0103] Step S40: Pairing the query image with each candidate image in the first image sequence respectively to generate a plurality of image pairs.

[0104] It should be noted that the image pair is a binary combination composed of a query image and a single candidate image in the first image sequence, for example, the image pair format is (query image, candidate image).

[0105] Step S50: Inputting the plurality of image pairs into a reordering network to obtain a reordering similarity score of each image pair.

[0106] It should be noted that the reordering network includes a local feature block matching module, which is used for local feature matching of the query image and the candidate image in each image pair.

[0107] In a specific implementation, in order to fully exploit the similarity relationship of the query-candidate image pair, the reordering network adopts the image pair as input. The feature map is the feature output of the image pair in the second stage of the Swin Transformer, which can obtain better local details and capture sufficient context information, wherein the spatial size is , and the channel number is . Then, the relative position weight of the feature block is dynamically calculated through the spatial relative attention to strengthen the relative position constraint in space. Then, the feature map is stretched to a one-dimensional vector in the spatial dimension, at this time , through multiple interleaved execution of self-attention and cross-attention, it is converted into a more discriminative feature mapping . Then, the local feature block matching is performed according to the feature , and based on the local feature block matching result, the final reordering similarity score is calculated through position importance weighting, and the global image retrieval result is optimized.

[0108] Further, in order to effectively capture the spatial relationship of the clothing image, in an embodiment, the reordering network further includes a spatial relative attention module;

[0109] The spatial relative attention module is configured to perform spatial enhancement processing on the image pair input to the reordering network, and output spatial enhancement features.

[0110] The spatial enhancement processing procedure of the spatial relative attention module includes:

[0111] The original features are input to a convolutional layer, and first initial features and second initial features are generated based on an original feature dimension of the original features, the original features being local image features of the image pair input to the reordering network;

[0112] Spatial attention is calculated based on the first initial features and the second initial features, and spatial position attention weights of the image pair are obtained;

[0113] The image pair is subjected to spatial enhancement processing based on the spatial position attention weights, and spatial enhancement features are output.

[0114] It should be noted that the embodiment introduces a spatial relative attention mechanism (SRA) to capture the spatial relationship of the garment image.

[0115] Referring to Figure 3 , Figure 3 The processing procedure of the spatial relative attention module is shown in FIG. 6. In the process of local matching, the relative position of the local feature blocks of the image will affect the accuracy of matching. Two pairs of matching blocks are selected from the query image and the candidate image. Since the difference in the relative position of the two local regions in different images is smaller, the matching is more reliable. On this basis, the embodiment proposes a spatial relative attention mechanism. Since the relative position is the spatial correlation between regions, the SRA is established on a spatial attention module, the relative position encoding is introduced, the direction perception embedding of separating the horizontal and vertical components is performed, and the spatial relationship of the garment image is effectively captured. Even if there is a problem of occlusion, the geometric consistency can still be maintained.

[0116] As shown in FIG. 7, given a local feature Figure 3 , first, it is input to a convolutional layer to generate two new feature maps B and C, where , and then they are reshaped to , where is the number of pixels. After that, C is transposed and matrix multiplication with B is calculated, a softmax layer is applied to calculate spatial attention, and at this time, the relative position components in the horizontal and vertical directions need to be added:

[0117]

[0118] wherein, represents the i ​The relative position of the first position to the second position, j The influence of the relative position of the first position to the second position, is the position coding of the horizontal component dx of the relative position, is the position coding of the vertical component dy of the relative position, as shown in Figure 4 , Figure 4 is a schematic diagram of the relative position coding, and dx and dy respectively represent the horizontal and vertical relative distances from the source point of the feature map B to the target point of the feature map C.

[0119] At the same time, the feature A is input into the convolutional layer to generate a new feature map , which is reshaped into . Then the transposes of D and S are matrix multiplied, and the result is reshaped into , multiplied by a scale parameter α, to obtain the final output as follows:

[0120]

[0121] Further, in order to accurately mine and aggregate the key information between the images and the image pairs, in an embodiment, the reordering network further comprises a bidirectional attention module;

[0122] The bidirectional attention module is configured to perform feature iterative optimization processing on the spatial enhanced features output by the spatial relative attention module, and output a bidirectional attention feature pair when the number of iterations reaches an optimization number threshold, the bidirectional attention feature pair comprising bidirectional attention output features of the query image and the candidate image.

[0123] The processing process of the bidirectional attention module includes a self-attention stage and a cross-attention stage.

[0124] The processing process of the self-attention stage includes:

[0125]

[0126]

[0127]

[0128]

[0129] wherein, represents the intermediate feature of the self-attention stage, represents layer normalization, represents a multi-head self-attention unit, , and Let represent the query matrix, key matrix, and value matrix, respectively, which are obtained by linear transformation of the input features during the self-attention stage. Indicates features Connect to the residual of the layer normalized output. This represents the target output feature during the self-attention phase. This represents a feedforward network, which consists of a multilayer perceptron. Indicates intermediate features A residual connection is made with the output of the feedforward network to represent the feature concatenation operation. Indicates the first The output of each attention head, Indicates the number of attention heads, indicating attention. This represents the learnable output projection weight matrix. , and They represent the first The query matrix, key matrix, and value matrix of each attention head. This represents the scaled dot product attention function;

[0130] The processing steps of the cross-attention phase include:

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137] in, Indicates query image Self-attention output, , and The cross-attention parameters represent the query image, and these parameters include the query matrix, key matrix, and value matrix. , and This represents the cross-attention weight matrix of the query image. Representing candidate images Self-attention output, , and The cross-attention parameters represent the candidate images. , and This represents the cross-attention weight matrix of the candidate images. This represents the cross-attention output of the query image. This represents the target output feature of the cross-attention phase of the query image. These represent intermediate features of the cross-attention stage of candidate images. This represents the target output feature of the cross-attention stage of the candidate image.

[0138] It should be noted that the retrieval device can use a bi-attention module to further mine and aggregate key information within images and between image pairs, such as... Figure 5 As shown, Figure 5 This is a schematic diagram of the bidirectional attention module. For the self-attention layer, a global dependency relationship is established through the self-attention mechanism to capture complex structural information within the image. Different regions in an image may contain different semantic information; through the self-attention mechanism, the model can learn the interdependencies between these regions, thereby generating features with stronger representational capabilities. For locally extracted features… The computational steps of the self-attention mechanism are as follows:

[0139]

[0140] in, It is a learnable weight matrix.

[0141] Calculate self-attention using scaled dot product attention:

[0142]

[0143] Self-attention is extended to multi-head self-attention (MSA). The multi-head self-attention mechanism (MSA) enables the model to pay attention to information from multiple representation subspaces simultaneously, thus extending the standard self-attention mechanism.

[0144] The feature tokens generated by the MSA are then refined through a feedforward network (FFN) consisting of two multilayer perceptron (MLP) layers, using the GELU activation function. Layer normalization (LN) is performed after both the MSA and FFN, and residual connections are used.

[0145] In image retrieval tasks, cross-attention mechanisms can help models establish stronger associations between query images and candidate images, focusing on the parts of candidate images that are relevant to the query image, mitigating matching errors caused by occlusion, and thus improving retrieval accuracy. For example... Figure 5 As shown, the input features of the cross-attention layer are derived from the image pair features processed by the self-attention module. and ) or ( and ), and a bidirectional feature interaction is implemented.

[0146] The BiAttention module is configured to be executed N times, wherein the input of self-attention in the first execution is , and the final output is .

[0147] Further, in order to improve the recognition efficiency of fine-grained features and make up for the information loss caused by occlusion, in an embodiment, the local feature matching process of the local feature block matching module comprises:

[0148] calculating the spatial position dot product between the bidirectional attention output features of the query image and the candidate image in the bidirectional attention feature pair output by the bidirectional attention module, and generating an initial confidence matrix based on the spatial position dot product;

[0149] performing Softmax normalization on the initial confidence matrix along the row and column respectively to obtain a bidirectional confidence matrix;

[0150] performing local feature matching on the local feature blocks of the query image and the candidate image based on a preset similarity threshold and the bidirectional confidence matrix to obtain a local feature matching result, the local feature matching result is used to calculate the reordering similarity score of the image pair, and the local feature matching result includes a matching set and a matching confidence of each matching pair in the matching set, the bidirectional confidence of the feature blocks matched with each other in each matching pair in the matching set is higher than the preset similarity threshold, and the matching pair is composed of the local feature blocks matched with each other between the query image and the candidate image.

[0151] It should be noted that the feature mapping processed based on the attention mechanism is A local feature block matching mechanism is established. In the clothing image retrieval scene, due to the difficulty in distinguishing fine-grained features, serious occlusion caused by differences in shooting angles, and other problems, relying only on global features may cause similar images to be ranked last. Local feature block matching performs threshold matching on the local feature blocks of the query image and the candidate image, reduces the similarity of local features of different images, and enhances the class distinction ability. For seriously occluded similar images, since there are still matching areas in the local, after local feature matching, the local similarity of similar images is increased, thereby making up for the information loss caused by occlusion, and promoting the ranking of this kind of image in the final retrieval result to be more forward. The specific local feature matching mechanism is as follows:

[0152] First, the confidence matrix between the two feature vectors and is calculated:

[0153]

[0154] wherein, is the feature map after stretching the spatial dimension respectively in the stretching dimension, i.e. the position size of the feature block.

[0155] As Figure 6 shown, Figure 6 is a schematic diagram of a dual-softmax calculation process, and the dual-softmax is applied on the two stretching dimensions to obtain a bidirectional confidence matrix :

[0156]

[0157] By setting a similarity threshold , the matching blocks with a confidence higher than the threshold are screened out, and the spatial positions and matching confidences of the matching blocks are recorded. Meanwhile, a mutual nearest neighbor (MNN) criterion is used to filter out possible abnormal matching blocks. This requires that the feature blocks of the query image and the candidate image simultaneously satisfy the maximum similarity in both directions, so as to prevent a feature block from matching multiple feature blocks of another image. As Figure 6 shown, the third feature block of the query image only matches the third feature block of the candidate image, and will not simultaneously match the second feature block of the candidate image. The matching feature block set The calculation process is as follows:

[0158]

[0159] wherein, the similarity threshold, represents a feature block matching pair;

[0160] Further, in order to accurately score the similarity of the image pair, in an embodiment, the reordering network further comprises a similarity scoring module;

[0161] The similarity scoring module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, and determine a reordering similarity score of the image pair.

[0162] The similarity scoring process of the similarity scoring module comprises:

[0163] Based on the local feature matching result output by the local feature block matching module, the matching confidence of each matching pair in the matching set is aggregated to obtain an aggregated confidence.

[0164] determine a feature similarity of the image pair based on the aggregated confidence, and calculate a re-ranking similarity score of the image pair based on the feature similarity.

[0165] It should be noted that, in order to improve the retrieval performance by local feature matching, the re-ranking stage needs to establish accurate local correspondence between image pairs. Since the re-ranking is performed by mining similar fine-grained features of the same class of images and distinguishing features of different classes of images to improve the performance of global retrieval, by setting the feature blocks satisfying the mutual nearest neighbor search criterion as similar feature blocks and setting the similarity of the unmatched feature blocks as 0, the similarity between two feature blocks can be set as:

[0166]

[0167] wherein, denotes the feature block similarity;

[0168] The sum of the confidence values of all matched feature blocks is taken as the similarity between the image pairs. Thus, the similarity of the entire image pair can be obtained as:

[0169]

[0170] wherein, denotes the image similarity between the query image and the candidate image , denotes the matching confidence, denotes the set of matched feature blocks.

[0171] Further, in order to solve the problem of local region importance difference in clothing image retrieval and accurately configure the importance of each position so as to improve the similarity score accuracy, in an embodiment, the local feature matching result output by the local feature block matching module aggregates the matching confidence of each matching pair in the matching set to obtain an aggregated confidence, including:

[0172] generating a spatial weight matrix based on the local feature matching result output by the local feature block matching module;

[0173] positionally weighting the feature blocks of each matching pair in the matching set based on the spatial weight matrix to obtain a spatial position weight of each feature block;

[0174] weight-aggregating the matching confidence of each matching pair in the matching set based on the spatial position weight to obtain an aggregated confidence.

[0175] It should be noted that in the process of reordering, the matching of the center position of the clothing image is more important than the matching of the edge position. In view of the problem of local area importance difference in clothing image retrieval, an adaptive center weight distribution mechanism is designed to realize position importance weighting. By introducing a two-dimensional Gaussian kernel function to generate a spatial weight matrix, the position of the feature block is weighted, and the Gaussian function formula is as follows:

[0176]

[0177] wherein, is the position coordinate, , is the center position coordinate. is the standard deviation of the Gaussian function, which controls the speed of weight attenuation.

[0178] As Figure 7 shown, Figure 7 is an example of position importance weight, A and B are the weights of different positions of the image, and B is closer to the center than A and is assigned a more important weight. After calculating the position importance weight of each position, the weight is assigned to the clothing image. In this embodiment, the weight matrix is stretched to one dimension to obtain the weight of the position , wherein , the confidence of the matched feature block is weighted to obtain the final image similarity:

[0179]

[0180] Step S60: Reordering the first image sequence based on the reordered similarity score to obtain a second image sequence, and determining a target clothing image matching the query image based on the second image sequence.

[0181] In some embodiments, in view of the problems of fine-grained difference discrimination difficulty and occlusion interference existing in clothing image retrieval, an effective clothing image retrieval reordering model is proposed. The retrieval device inputs the query image into the clothing image retrieval reordering model for clothing image retrieval by pre-constructing the clothing image retrieval reordering model, and outputs a target clothing image matching the query image.

[0182] Referring to Figure 8 , Figure 8A schematic diagram of a processing procedure of a clothing image retrieval and re-ranking model, the clothing image retrieval and re-ranking model comprising a global retrieval module and a re-ranking network, wherein the re-ranking network comprises a BiAttention module, a local feature block matching module, a spatial relative attention mechanism module, a position importance weighting module, and a loss function (Scale-invariant Loss).

[0183] The processing procedure of the clothing image retrieval and re-ranking model comprises: firstly, extracting image global features based on a Swin Transformer for global retrieval to obtain a preliminary candidate image set. On this basis, a re-ranking network composed of multiple layers of BiAttention is introduced to mine more discriminative local features from query-candidate image pairs. BiAttention uses self-attention to capture semantic information within the image and enhances the feature's ability to represent fine-grained differences; uses bidirectional cross-attention to capture the association information between image pairs, effectively enhancing the model's robustness to the occlusion problem. In addition, the re-ranking network uses spatial relative attention to strengthen spatial position constraints, and performs position importance weighting on the matched feature blocks to focus the final image pair similarity evaluation on key feature blocks.

[0184] Since the similarity of the two images is composed of the sum of the feature block confidence exceeding the threshold, at this time the similarity of the image has not been normalized, so it cannot set the similarity of positive and negative samples to a fixed boundary as the contrast loss function. To this end, the embodiment proposes a scale-invariant loss function to avoid the scale sensitivity problem caused by unnormalized features. By introducing a dynamic scaling factor, the optimization target is independent of the absolute value, and the relative relationship of the similarity is concerned:

[0185]

[0186] wherein, represents the similarity of the positive sample, represents the similarity of the negative sample, is a dynamic scaling factor, represents a tolerable proportion threshold, when , it means that the similarity of the negative sample is allowed to be at most 50 of the similarity of the positive sample, is a specified minimum value (such as 1e-6) to prevent division by zero error, and Relu is an activation function.

[0187] The embodiment extracts the global feature of the query image and performs global retrieval to obtain a first image sequence preliminarily matched with the query image, performs local feature block matching through a reordering network, and outputs a reordering similarity score, thereby fully mining the local features in the clothing image, enhancing the perception ability of local differences in the clothing image, reducing inter-class false matching, improving the local feature retrieval ability of the occluded image, and achieving accurate and effective retrieval in a large-scale clothing image dataset.

[0188] In some embodiments, in order to verify the technical effect of the present application, based on a certain technology company clothing dataset, a large-scale clothing image dataset is established, 52120 images of 3436 commodity ids are randomly selected for training, 52306 images of 3398 commodity ids are used for testing, among which 1000 images are randomly selected as a query set, a total of 812 commodity ids, and 51306 candidate images. The dataset data includes original images, maximum detection box positions, and cropped pictures. Among them, the query image and the candidate image use the content of the maximum detection box for comparison, search for similar pictures in all candidate images and the query image, and hope that the pictures of the same commodity id are ranked in the front, and the similar pictures are ranked in the back. Since the size of the cropped image is not uniform, the bilinear interpolation method is used to set the image to the same resolution (448, 448). Here, recall@K and mAP@100 are used to evaluate the performance of the model:

[0189] (1). recall@K

[0190] For each query image, return K images, whether there is a correct (same class) return image. If there is one or more, the recall score is 1. Conversely, the recall score is 0. Then calculate the average recall rate of all query images:

[0191]

[0192] Where r represents the number of all query images, q represents the current query image, and score represents the score of the returned image.

[0193] (2). mAP@K

[0194] Since there may be the same or similar images in different classes, this paper uses the average precision (mAP) score as an indicator to evaluate the performance of similarity retrieval. First, calculate the average precision (AP) score of each query, and then calculate their average value as the mAP score.

[0195] The formula for calculating mAP is:

[0196]

[0197]

[0198] Where Q represents the number of images to be retrieved, r represents the total number of relevant images, K represents the ranking position, is the accuracy of the first K results, and represents the image score at position K, 1 for relevant and 0 for irrelevant.

[0199] This paper conducts experiments based on Python3.10 and Pytorch1.12.1 framework, uses Adam optimizer, epoch is set to 40, and learning rate is 3e-5. The input image is enhanced by horizontal flip, random rotation and aspect ratio distortion, and the resolution is adjusted to (224, 224).

[0200] The global retrieval module uses Swin Transformer as the basic model, uses the pre-trained model swin-small on ImageNet-1K to train and fine-tune on the clothing dataset, and the final output feature dimension is 784. The loss function uses the contrastive loss function, and the positive and negative sample boundaries are set to 0.5 and 1 respectively.

[0201] The re-ranking network uses Swin Transformer as the basic model to extract the second stage features, and sets the parameters of this part of the network to be frozen. The number of BiAttention modules is set to 2, and the threshold for feature block matching is set to 0.2, and the multi-head attention is set to have 8 heads. The Gaussian kernel sigma in PIW is set to 5 to make the weight decay relatively smooth, and the dynamic scaling factor in the proportion loss function is set to 0.7. In the re-ranking module, the final ranking result is determined by the final similarity score , where is the global similarity score, is the local similarity score. Because the image re-ranking is performed on images that look very similar, it must be robust to hard samples. The image pairs input to the re-ranking network during the training process are composed of positive sample pairs and hard negative sample pairs, where the most similar negative sample mined by the global feature is taken as the hard negative sample, as shown in Figure 9 , and Figure 9 is a schematic diagram of a hard negative sample pair.

[0202] The experimental results are shown in Tables 1 and 2. The re-ranking network proposed in this paper has the best retrieval performance on the clothing dataset. First, the retrieval results of the model in this paper are compared with three global retrieval benchmark models, where GeM and Arcfac are based on Restnet50 and Resnet101 respectively. GeM uses a trainable generalized mean pooling layer and has an advantage in overall image representation. Arcface loss shows good results in global feature learning by inducing a smaller intra-class variance. IRT uses DeiT as the backbone network to extract features, fine-tunes the transformer using metric learning, and additionally uses a differential entropy regularizer to enhance the contrast loss. However, these methods are difficult to distinguish the fine-grained differences in the clothing image retrieval scene and are sensitive to background and pose changes. Compared with the models R50-GeM, R01-GeM, R50-Arcface, R101-Arcface, and IRT, the model in this embodiment improves the Recall@1 index by 4.7%, 4.0%, 1.7%, 1.5%, and 1.3% respectively, and improves the mAP@100 index by 15.43%, 14.78%, 8.46%, 7.93%, and 3.58% respectively.

[0203] Table 1. Comparison of different methods in clothing image retrieval results Recall@K

[0204]

[0205] Table 2. Comparison of different methods in clothing image retrieval results mAP@100

[0206]

[0207] At the same time, under the condition of the same global feature retrieval, the text reordering model and four reordering methods are compared, among which Global is the global feature retrieval method proposed in this paper. aQE uses the retrieval results to average the feature expansion, but does not consider the local details and the structure information between images. DL-ENDR optimizes the image similarity using the diffusion-based graph structure, but when the initial retrieval result has errors, the diffusion path is easily disturbed by noise. SuperGlobal improves the overall representation ability through feature enhancement, but lacks explicit alignment ability, and has certain misjudgment when dealing with images with local similarity and global difference. CAS emphasizes cross-scale information fusion, but lacks explicit matching mechanism at the regional level. Compared with the above methods, the method in this paper introduces a local feature block matching strategy, which improves the fine-grained similarity modeling ability and the robustness of occlusion. Compared with the above reordering models aQE, DL-ENDR, SuperGlobal, CAS, the model in this paper improves the Recall@1 index by 3.2%, 1.3%, 1.6%, 1.1% respectively, and the mAP@100 index by 5.26%, 3.08%, 0.27%, 0.65% respectively. Referring to Figure 10 , Figure 10 is a comparison diagram of different methods in clothing image retrieval results.

[0208] In order to verify the effectiveness of the reordering model based on local feature block matching, three variant experiments are designed for analysis: 1) w / o local: without reordering network, only through global feature retrieval. 2) w / o PIW: the image similarity of the reordering network is not weighted by the position importance. 3) w / o SRA: the image similarity of the reordering network is not subjected to spatial relative attention. The experimental results are shown in Tables 3 and 4. After adding the reordering network, the Recall@1, Recall@10 and Recall@20 indexes are increased by 2.9%, 0.4% and 0.2% respectively, and the mAP@100 index is increased by 1.07%. It can be found that after local feature block matching, the retrieval performance is better, the spatial relative attention mechanism can enhance the spatial scale constraint of the feature block, and it is also necessary to weight the importance of the matched blocks, because the edge features matched are not required for retrieval, and the weight needs to be reduced.

[0209] Table 3, ablation experiment results of clothing image retrieval Recall@K comparison

[0210]

[0211] Table 4, ablation experiment results of clothing image retrieval mAP@100 comparison

[0212]

[0213] To verify the effect of reordering, 100, 200, 300, 400 pictures and global image retrieval effect are reordered for comparison, and the experimental results are shown in Tables 5 and 6, which increase by 2.9%, 2.8%, 3.1%, 3.5% respectively in Recall@1, and increase by 1.07%, 0.94%, 0.92%, 0.89% respectively in mAP@100, and the results show that the performance of global image retrieval can be improved after reordering.

[0214] Table 5, experimental results of Recall@K comparison of different reordering numbers in clothing image retrieval

[0215]

[0216] Table 6, experimental results of mAP@100 comparison of different reordering numbers in clothing image retrieval

[0217]

[0218] Since the image pair may match many feature blocks, the local similarity may be very large, but when no or few matching blocks exceeding the threshold are found, the value is 0 or close to 0, and using the value alone for reordering is not reliable, and using only local features cannot reflect global information. Therefore, the global feature similarity is used for retrieval, and the similarity calculated by local feature block matching is used to optimize the global retrieval similarity. For a query image, in the top_k reordering process, k times of local feature block matching will be performed, and the similarity obtained by the k times of matching will be respectively subjected to sigmoid function, regularization, softmax function, and then combined with the global similarity for reordering. As shown in Table 7, the mAP@100 effect of global similarity reaches the optimum.

[0219] Table 7, combination method of local similarity and global similarity

[0220]

[0221] In addition, the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores a clothing image retrieval program based on local feature block matching. The clothing image retrieval program based on local feature block matching is executed by a processor to realize the steps of the clothing image retrieval method based on local feature block matching as described above.

[0222] The computer readable storage medium provided in the application may be, for example, a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination of the above. More specific examples of the computer readable storage medium may include, but are not limited to, an electrical connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electrical wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination of the above.

[0223] The computer readable storage medium described above may be included in the garment image retrieval device based on local feature block matching, or may exist independently without being assembled into the garment image retrieval device based on local feature block matching.

[0224] In addition, the embodiment of the application also provides a computer program product, including a garment image retrieval program based on local feature block matching, and the garment image retrieval program based on local feature block matching realizes the steps of the garment image retrieval method based on local feature block matching when executed by a processor.

[0225] The computer program product embodiment of the application is basically the same as the above-mentioned garment image retrieval method based on local feature block matching, and will not be repeated here.

[0226] Reference Figure 11 , Figure 11 The structure block diagram of the first embodiment of the garment image retrieval device based on local feature block matching of the application is shown in the figure.

[0227] As Figure 11 shown, the garment image retrieval device based on local feature block matching provided by the embodiment of the application includes:

[0228] The feature extraction module 10 is used for inputting the query image into the global retrieval module for global feature extraction to obtain the first global feature.

[0229] a global matching module 20, configured to perform similarity matching between the first global feature and second global features of initial images in a database, to obtain a global matching result, the second global features being obtained by performing global feature extraction on the initial images by the global retrieval module;

[0230] an image ranking module 30, configured to filter a plurality of candidate images from the initial images based on the global matching result, and rank the candidate images based on global similarity between each candidate image and the query image, to obtain a first image sequence;

[0231] an image pairing module 40, configured to pair the query image with each candidate image in the first image sequence respectively, to generate a plurality of image pairs;

[0232] a re-ranking scoring module 50, configured to input the plurality of image pairs into a re-ranking network to obtain a re-ranking similarity score of each image pair, the re-ranking network comprising a local feature block matching module, the local feature block matching module being configured to perform local feature matching between the query image and the candidate image in each image pair;

[0233] a target image retrieval module 60, configured to reorder the first image sequence based on the re-ranking similarity score to obtain a second image sequence, and determine a target clothing image matching the query image based on the second image sequence.

[0234] The embodiment extracts the global feature of the query image and performs global retrieval to obtain the first image sequence preliminarily matched with the query image, performs local feature block matching through the reordering network, and outputs the reordering similarity score, thereby fully mining the local features in the clothing image, enhancing the perception ability of the local differences in the clothing image, reducing the inter-class false matching, improving the local feature retrieval ability of the occluded image, and achieving accurate and effective retrieval in a large-scale clothing image dataset.

[0235] The clothing image retrieval device based on local feature block matching provided in the application adopts the clothing image retrieval method based on local feature block matching in the above embodiment, and can solve the technical problem of clothing image retrieval based on local feature block matching. Compared with the prior art, the clothing image retrieval device based on local feature block matching provided in the application has the same beneficial effects as the clothing image retrieval method based on local feature block matching provided in the above embodiment, and other technical features in the clothing image retrieval device based on local feature block matching are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0236] It should be understood that the above is only an example, and the technical solutions of the application do not constitute any limitation. In specific applications, those skilled in the art can set it up as needed, and the application does not limit it.

[0237] It should be noted that the above-described workflow is merely illustrative and does not limit the scope of protection of the present application. In actual applications, a person skilled in the art can select part or all of the above-described workflow to achieve the purpose of the embodiment according to actual needs, which is not limited herein.

[0238] In addition, technical details not described in detail in the embodiment can be found in the garment image retrieval method based on local feature block matching provided by any embodiment of the present application, which will not be described here.

[0239] It should be noted that in this document, the terms "comprise", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or system. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article or system that includes the element.

[0240] The above-mentioned embodiment numbers of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0241] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, an optical disk) and includes a number of instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present application.

[0242] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation, or direct or indirect application in other related technical fields, is also included in the patent protection scope of the present application.

Claims

1. A garment image retrieval method based on local feature block matching, characterized in that, The method comprises: inputting a query image into a global retrieval module for global feature extraction to obtain a first global feature; performing similarity matching between the first global feature and second global features of each initial image in a database to obtain a global matching result, the second global features being obtained by pre-extracting global features of the initial images by the global retrieval module; selecting multiple candidate images from the initial images based on the global matching result, and sorting the candidate images based on global similarity between each candidate image and the query image to obtain a first image sequence; pairing the query image with each candidate image in the first image sequence to generate multiple image pairs; inputting the multiple image pairs into a re-ranking network to obtain a re-ranking similarity score of each image pair, the re-ranking network comprising a local feature block matching module configured to perform local feature matching between the query image and the candidate image in each image pair; the re-ranking network further comprises a spatial relative attention module and a bidirectional attention module; the spatial relative attention module is configured to perform spatial enhancement processing on the image pairs input into the re-ranking network to output spatial enhancement features; the spatial enhancement processing process of the spatial relative attention module comprises: inputting an original feature into a convolution layer, generating a first initial feature and a second initial feature based on an original feature dimension of the original feature, the original feature being a local image feature of the image pairs input into the re-ranking network; calculating spatial attention based on the first initial feature and the second initial feature to obtain spatial position attention weights of the image pairs; performing spatial enhancement processing on the image pairs based on the spatial position attention weights to output spatial enhancement features; the bidirectional attention module is configured to perform feature iterative optimization processing on the spatial enhancement features output by the spatial relative attention module, and output a bidirectional attention feature pair when the number of iterations reaches an optimization number threshold, the bidirectional attention feature pair comprising bidirectional attention output features of the query image and the candidate image; re-ranking the first image sequence based on the re-ranking similarity score to obtain a second image sequence, and determining a target clothing image matching the query image based on the second image sequence.

2. The garment image retrieval method based on local feature block matching according to claim 1, wherein, The processing process of the bidirectional attention module comprises a self-attention phase and a cross-attention phase; the processing process of the self-attention phase comprises: wherein, denotes an intermediate feature of a self-attention stage, denotes layer normalization, denotes a multi-head self-attention unit, , and denote a query matrix, a key matrix and a value matrix, respectively, which are linearly transformed from an input feature of a self-attention stage, denotes concatenating the feature with a residual connection to an output result of the layer normalization, denotes a target output feature of the self-attention stage, denotes a feed-forward network, which consists of a multi-layer perceptron, denotes concatenating the intermediate feature with a residual connection to an output result of the feed-forward network, denotes a feature concatenation operation, denotes an output of the -th attention head, denotes a number of attention heads, denotes an output projection weight matrix, , and denote a query matrix, a key matrix and a value matrix, respectively, of the -th attention head, denotes a scaled dot-product attention function; the processing process of the cross-attention phase comprises: wherein, denotes a self-attention output of the query image, , and denote cross-attention parameters of the query image, the cross-attention parameters comprising a query matrix, a key matrix and a value matrix, , and denote cross-attention weight matrices of the query image, denotes a self-attention output of the candidate image , , and denote cross-attention parameters of the candidate image, , and denote cross-attention weight matrices of the candidate image, denotes a cross-attention output of the query image, denotes a target output feature of the cross-attention stage of the query image, denotes an intermediate feature of the cross-attention stage of the candidate image, denotes a target output feature of the cross-attention stage of the candidate image.​ 3. The garment image retrieval method based on local feature block matching according to claim 2, wherein, the local feature matching process of the local feature block matching module comprises: calculating a spatial position dot product between the bidirectional attention output features of the query image and the candidate image in the bidirectional attention feature pair output by the bidirectional attention module, and generating an initial confidence matrix based on the spatial position dot product; performing Softmax normalization on the initial confidence matrix along the row and column respectively to obtain a bidirectional confidence matrix; The local feature matching result is used for calculating a reordering similarity score of the image pair, and the local feature matching result includes a matching set and a matching confidence of each matching pair in the matching set, a bidirectional confidence of feature blocks matched with each other in each matching pair in the matching set is higher than the preset similarity threshold, and the matching pair is composed of the local feature blocks matched with each other between the query image and the candidate image.

4. The garment image retrieval method based on local feature block matching according to claim 3, wherein, The reordering network further includes a similarity score module; The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, and determine the reordering similarity score of the image pair. The similarity scoring process of the similarity score module includes: The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, and determine the reordering similarity score of the image pair. The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, and determine the reordering similarity score of the image pair.

5. The garment image retrieval method based on local feature block matching according to claim 4, wherein, The similarity scoring process of the similarity score module includes: The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, and determine the reordering similarity score of the image pair. The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, and determine the reordering similarity score of the image pair. The similarity score module is configured to perform similarity scoring based on the local feature matching result output by the local feature block matching module, and determine the reordering similarity score of the image pair.

6. The garment image retrieval method based on local feature block matching according to any one of claims 1 to 5, wherein, The global retrieval module is constructed based on a Swin-Transformer model. The loss function of the global retrieval module is a contrastive loss function, and a mathematical expression of the contrastive loss function is: wherein, represents a contrastive loss function, is a batch size of samples, is a sample class label, and respectively represent a sample index, is a cosine similarity between feature vectors of a positive sample image pair, is a cosine similarity between feature vectors of a negative sample image pair, and are boundaries of a positive sample pair and a negative sample pair, respectively.

7. A garment image retrieval apparatus based on local feature block matching, characterized by, The clothing image retrieval device based on local feature block matching includes: A feature extraction module is configured to input a query image into a global retrieval module to perform global feature extraction, and obtain a first global feature. A global matching module is configured to perform similarity matching on the first global feature and a second global feature of each initial image in a database, and obtain a global matching result, the second global feature being obtained by performing global feature extraction on the initial image by the global retrieval module in advance. An image ordering module is configured to filter a plurality of candidate images from the initial images based on the global matching result, and order the candidate images based on a global similarity between each candidate image and the query image, and obtain a first image sequence. An image pairing module is configured to pair the query image with each candidate image in the first image sequence respectively, and generate a plurality of image pairs. The reordering scoring module is configured to input the plurality of image pairs into a reordering network to obtain a reordering similarity score of each image pair, the reordering network comprising a local feature block matching module configured to perform local feature matching on a query image and a candidate image in each image pair; The reordering network further comprises a spatial relative attention module and a bidirectional attention module; The spatial relative attention module is configured to perform spatial enhancement processing on the image pairs input into the reordering network to output spatially enhanced features; The spatial enhancement processing of the spatial relative attention module comprises: inputting an original feature into a convolutional layer, generating a first initial feature and a second initial feature based on an original feature dimension of the original feature, the original feature being a local image feature of the image pairs input into the reordering network; calculating spatial attention based on the first initial feature and the second initial feature to obtain spatial position attention weights of the image pairs; performing spatial enhancement processing on the image pairs based on the spatial position attention weights to output spatially enhanced features; The bidirectional attention module is configured to perform feature iterative optimization processing on the spatially enhanced features output by the spatial relative attention module, and output a pair of bidirectional attention features when the number of iterations reaches an optimization threshold, the pair of bidirectional attention features comprising bidirectional attention output features of the query image and the candidate image. The target image retrieval module is configured to reorder the first image sequence based on the reordering similarity scores to obtain a second image sequence, and determine a target clothing image matching the query image based on the second image sequence.

8. A garment image retrieval device based on local feature block matching, characterized by, The clothing image retrieval device based on local feature block matching comprises a memory, a processor, and a clothing image retrieval program based on local feature block matching stored on the memory, the processor being configured to run the clothing image retrieval program based on local feature block matching, the clothing image retrieval program based on local feature block matching being configured to implement the clothing image retrieval method based on local feature block matching according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a clothing image retrieval program based on local feature block matching, and the clothing image retrieval program based on local feature block matching is executed by the processor to implement the clothing image retrieval method based on local feature block matching according to any one of claims 1 to 6.

Images