Two-way one-to-many embedding alignment method for text-based person retrieval

Abstract

The application relates to a two-way one-to-many embedding alignment method for text-based person retrieval. 1) A pre-trained ResNet-50 backbone network is used to extract picture features, and a pre-trained BERT and a trainable Bi-LSTM model are used to extract global visual and text features; 2) the visual features V are uniformly divided into K non-overlapping parts, a word attention module is used to obtain K different text features, and local visual and text features are obtained; 3) a non-local module is established, and non-local visual and language features are obtained based on cosine similarity; 4) a new two-way one-to-many embedding paradigm is established, which is specifically divided into two blocks of functions, namely two-way embedding and one-to-many embedding; 5) identity loss and composite ranking loss are adopted, and the two are added to construct an overall loss function for optimization. The model can determine the optimization direction and simultaneously solve the one-to-many matching problem of the picture-text pair.

Description

Technical Field

[0001] This invention relates to a method for retrieving people, and more particularly to a bidirectional one-to-many embedding alignment method for cross-modal text-based people retrieval. Background Technology

[0002] Text-based person retrieval (TPR) (Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person search with natural language description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1970–1979, 2017.1,2,5,6) is a critical task that uses natural language description queries to find target persons from large image databases. Compared to traditional image-based or attribute-based person retrieval queries, language descriptions are more accessible than images and more comprehensive than attributes. However, due to the significant modal differences between vision and language, TPR is more challenging than image-based and attribute-based person retrieval.

[0003] To mitigate the significant modality gap, most literature (Xiao Han, Sen He, Li Zhang, and Tao Xiang. Text-based person search with limited data. arXiv preprint arXiv:2110.10807,2021.1,6; Aichun Zhu, Zijie Wang, Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, and Gang Hua. DSSL: Deep surroundings-person separation learning for text-based person retrieval. In Proceedings of the 29th ACM International Conference on Multimedia, pages 209–217,2021.1,2,4,5,6) employs a modality-sharing one-to-one embedding paradigm. The retrieval model embeds image and text descriptions into a joint space using a one-to-one projection function, and then studies the semantic alignment between image and text pairs within that space. While this paradigm is theoretically elegant and has been widely used for decades, it still has limitations in terms of optimization and alignment. Summary of the Invention

[0004] This invention aims to address the optimization difficulties, intermodal alignment challenges, and neglect of one-to-many image-text pair relationships inherent in traditional modality-sharing one-to-one embedding paradigms. To this end, the invention proposes a bidirectional one-to-many embedding alignment method for text-based person retrieval, enabling the model to determine the optimization direction and solve the one-to-many image-text pair matching problem. The new model uses a projection function to embed visual features into the text space and aligns them with text features based on cosine similarity. The same operation is applied to text features. Simultaneously, to alleviate the difficulty of intermodal alignment, a residual embedding module group is designed to mitigate the challenges of one-to-many alignment.

[0005] This invention includes the following steps:

[0006] 1) Use the pre-trained ResNet-50 backbone network to extract image features, use the pre-trained fixed BERT and trainable Bi-LSTM models to extract text features, and obtain global visual and text features through global max pooling and fully connected layers.

[0007] 2) In order to obtain local features, the visual feature V is uniformly divided into K non-overlapping parts, and a word attention module is used to obtain K different text features. The visual local features and text local features are extracted through global max pooling and fully connected layers.

[0008] 3) Establish a non-local module (NLM) and obtain non-local visual and linguistic features through cosine similarity;

[0009] 4) Establish a novel bidirectional one-to-many embedding paradigm, specifically including bidirectional embedding and one-to-many embedding. The bidirectional embedding is used for visual features and linguistic features, simultaneously projecting one feature into the feature space of the other through an embedding function. The one-to-many embedding is designed for the one-to-many relationship between image and text pairs, establishing an embedding module group with a residual structure. Each module group consists of M residual embedding modules, and each residual embedding module can embed visual features into the text feature space or text features into the visual feature space. Through the bidirectional one-to-many embedding module, global visual features and global linguistic features, local visual features and local linguistic features, and non-local visual features and non-local linguistic features can all generate multiple embedded features in the text and image spaces for matching.

[0010] 5) Two common loss functions, ID loss and CR loss, are used. The two are added together to construct the overall loss function and then optimized.

[0011] In step 1), the specific steps for obtaining global visual and textual features include: extracting image features V∈R using a pre-trained ResNet-50 backbone network. H×W×C H, W, and C represent the height, width, and number of channels of the image features, respectively. A pre-trained, fixed BERT model and a trainable Bi-LSTM model are used to extract text features T∈R. N×C N and C represent the length and number of channels of the text representation, respectively; global visual and text features are obtained through global max pooling (GMP) and fully connected layers;

[0012]

[0013]

[0014] in, These are the learnable parameters of the fully connected layer, C g The channel dimension of the global feature, v g ,t g These represent global visual features and global text features, respectively.

[0015] In step 2), the specific steps for uniformly segmenting the visual feature V into K non-overlapping parts, simultaneously using a word attention module to obtain K different text features, and extracting the visual local features and text local features through global max pooling and fully connected layers can be as follows:

[0016] First, the visual features are evenly divided into K blocks, specifically represented as follows:

[0017]

[0018] For text features, a Word Attention (WAM) module is used to extract K local text features, specifically represented as follows:

[0019] Then, global max pooling and fully connected layers are used to obtain visual local features and textual local features. The specific formula is shown below, where...

[0020]

[0021]

[0022] In step 3), the specific steps for establishing nonlocal modules and obtaining nonlocal visual and linguistic features through cosine similarity can be as follows:

[0023] First, calculate the linear embedding. Cosine similarity score between ki The specific calculation formula is as follows:

[0024]

[0025] in, Then, local visual features are aggregated based on the similarity score, as shown below:

[0026]

[0027] in, Finally, the specific calculation formula for nonlocal visual features is as follows:

[0028]

[0029] in, Similarly, non-local text features are also obtained in this way.

[0030] In step 4), the specific method of bidirectional embedding can be as follows: The model first keeps the text features unchanged and only projects the visual features into the text feature space for matching; since the text features in the text space are fixed, and the visual features have a clear optimization direction, it is beneficial to improve the optimization stability; similarly, the model also keeps the visual features unchanged and only projects the text features into the image space for alignment; finally, the formula for constructing global features by the bidirectional embedding module is as follows:

[0031] v g-txt =f v2t (v g )

[0032] v g-img =f t2v (t g )

[0033] Among them, v g - txt It is a global visual feature in the text space, v g - img It is a global text feature in the image space; f v2t (·) is the projection function that projects visual features onto the text space, f t2v (·) is a projection function that projects text features onto the image space.

[0034] The specific method employs a group of residual embedding modules (REM-G) for one-to-many embedding; each REM-G consists of M residual embedding modules (REMs). For each residual embedding module, the model uses linear embedding based on residuals. Specifically, the m-th residual embedding module processing global features is shown below:

[0035]

[0036]

[0037] in, These are the learnable parameters of the m-th residual embedding module. These are global visual features in text space and global text features in image space, respectively.

[0038] In step 5), the ID loss formula is as follows:

[0039]

[0040] Where v, t, and y represent the visual feature, text feature, and corresponding ID number, respectively. These are learnable parameters; C inQ represents the dimension of the input feature and the number of person IDs, respectively.

[0041] The formula for the Composite Ranking (CR) loss is as follows:

[0042]

[0043]

[0044] Where (v,t) and (v,t') represent the correctly matched image-text pairs. The text-image pair represents a mismatch, α1 and α2 represent the boundaries, β represents the weight of the weak supervision term, S(·,·) represents the cosine similarity between the two samples, the model aligns the two modality-specific spaces, and each sample generates M features after projection, the similarity scores calculated on multiple embedded features in these two spaces are added together to obtain the final similarity score, the formula is:

[0045]

[0046] The final loss function is the sum of the ID loss and the CR loss:

[0047] L = L ID +L CR .

[0048] Text-based person retrieval (TPR) is a fine-grained image-text retrieval task. Due to the significant formal differences between visual and linguistic representations, establishing connections between the two is extremely challenging. Traditional methods ignore the uncertainty of optimization direction when both modal features are movable, and also overlook the one-to-many relationships frequently present in image-text pairs. To address these issues, this invention proposes a bidirectional one-to-many embedding alignment method, enabling the model to determine the optimization direction while simultaneously resolving the one-to-many matching problem between image-text pairs. Compared to traditional methods, this invention offers the following significant advantages:

[0049] 1) This invention is the first attempt in multimodal retrieval to explore a bidirectional embedding paradigm that aligns visual and text samples, effectively alleviating the problem of uncertain optimization direction in traditional paradigms.

[0050] 2) This invention models the one-to-many relationship in image-text pairs and further proposes a residual embedding module group for one-to-many embedding, which effectively improves the one-to-many alignment effect in image-text retrieval.

[0051] 3) The model proposed in this invention not only achieves state-of-the-art performance on three TPR datasets, but also achieves good results on other general and fine-grained image-text retrieval tasks, indicating that it has potential for expansion. Attached Figure Description

[0052] Figure 1 To explain the difficulty of bidirectional optimization between visual and textual modalities in text-based person retrieval, as well as one-to-many retrieval scenarios.

[0053] Figure 2 This is a schematic diagram of the main network structure of the model.

[0054] Figure 3 A comparison of the detection performance of the basic model and the proposed BEAT model.

[0055] Figure 4 The results show the experimental results of the model performing fine-grained retrieval on two image-text retrieval datasets. Detailed Implementation

[0056] The following embodiments will further illustrate the present invention with reference to the accompanying drawings.

[0057] like Figure 1 Traditional one-to-one embedding paradigms for modality sharing suffer from optimization difficulties, including the challenge of aligning modalities and neglecting one-to-many image-text pairs. This invention's novel model uses a projection function to embed visual features into the text space and aligns them with text features based on cosine similarity. The same operation is applied to text features. Furthermore, to address the one-to-many image-text pair scenario, a residual embedding module group is designed to alleviate the difficulty of one-to-many alignment.

[0058] like Figure 2 Each module in this embodiment of the invention is as follows:

[0059] 1. Multi-granularity feature extraction module

[0060] First, the image features V∈R are extracted using a pre-trained ResNet-50 backbone network. H×W×C H, W, and C represent the height, width, and number of channels of the image features, respectively. A pre-trained, fixed BERT model and a trainable Bi-LSTM model are used to extract text features T∈R. N×C N and C represent the length and number of channels of the text representation, respectively. Global max pooling (GMP) and fully connected layers are then used to obtain global visual and text features.

[0061]

[0062]

[0063] in These are the learnable parameters of the fully connected layer, C g The channel dimension of the global feature, v g ,t g These represent global visual features and global text features, respectively.

[0064] Local feature extraction module:

[0065] To obtain local visual features, the visual features are first divided into K blocks evenly, specifically as follows:

[0066]

[0067] For text features, a Word Attention (WAM) module is used to extract K local text features, specifically represented as follows:

[0068] Then, global max pooling and fully connected layers are used to obtain visual local features and textual local features. The specific formula is shown in the figure below.

[0069]

[0070]

[0071] Non-local feature extraction module:

[0072] First, calculate the linear embedding. Cosine similarity score between ki The specific calculation formula is as follows:

[0073]

[0074] in, Then, local visual features are aggregated based on the similarity score, as shown below:

[0075]

[0076] in, Finally, the specific calculation formula for nonlocal visual features is as follows:

[0077]

[0078] in, Similarly, non-local text features are also obtained in this way.

[0079] 2. Bidirectional one-to-many embedded module

[0080] Bidirectional Embedded Module:

[0081] Specifically, the model first keeps the text features unchanged, projecting only the visual features into the text feature space for matching. This way, since the text features in the text space are fixed, and the visual features have a clear optimization direction, it helps improve optimization stability. Similarly, the model also keeps the visual features unchanged, projecting only the text features into the image space for alignment. Finally, the formula for constructing global features by the bidirectional embedding module is as follows:

[0082] v g-txt =f v2t (v g )

[0083] v g-img =f t2v (t g )

[0084] Among them, v g - txt It is a global visual feature in the text space, v g - img It is a global text feature in the image space. v2t (·) is the projection function that projects visual features onto the text space, f t2v (·) is a projection function that projects text features onto the image space.

[0085] One-to-many embedded modules:

[0086] As is well known, there is a one-to-many relationship between image and text pairs, so traditional one-to-one projection may lead to suboptimal results. Since multiple positive samples of the query may be slightly different, it is difficult to match multiple positive samples with the unique embedding features of the query if only one-to-one projection is applied to the query samples. To accommodate this one-to-many relationship in image-text pairs, this invention proposes a Residual Embedding Module Group (REM-G) to perform one-to-many embedding. Specifically, each REM-G consists of M residual embedding modules (REMs). For each residual embedding module, the model employs linear embedding based on the residuals. The m-th residual embedding module processing the global features is shown below:

[0087]

[0088]

[0089] in, These are the learnable parameters of the m-th residual embedding module. These are global visual features in text space and global text features in image space, respectively.

[0090] 3. Model Optimization

[0091] The model employs two common loss functions: identity loss and composite ranking (CR) loss. The two are added together to construct the overall loss function and then optimized.

[0092] The formula for identity loss is as follows:

[0093]

[0094] Where v, t, and y are the visual features, text features, and their corresponding ID numbers, respectively. These are learnable parameters. C in Q represents the dimension of the input feature and the number of person ID numbers, respectively.

[0095] The formula for the composite ranking (CR) loss is as follows:

[0096]

[0097]

[0098] Where (v,t) and (v,t') represent correctly matched image-text pairs. The text-image pair represents a mismatch, α1 and α2 represent the boundaries, and β represents the weight of the weak supervision term. S(·,·) represents the cosine similarity between the two samples. The model aligns the two modality-specific spaces, and each sample generates M features after projection; the similarity scores calculated on multiple embedded features in these two spaces are summed to obtain the final similarity score, with the formula as follows:

[0099]

[0100] The final loss function is the sum of the ID loss and the CR loss:

[0101] L = L ID +L CR

[0102] Text-based people retrieval (TPR) is a crucial task that uses natural language descriptive queries to find target people from large image databases. Compared to traditional image-based or attribute-based people retrieval queries, language descriptions are more accessible than images and more comprehensive than attributes. However, due to the significant modal differences between visual and language-based methods, TPR is more challenging than both image-based and attribute-based people retrieval.

[0103] To mitigate the significant modality gap, most traditional methods employ a modality-sharing one-to-one embedding paradigm. The retrieval model embeds image and text descriptions into a joint space using a one-to-one projection function, and then studies the semantic alignment between image and text pairs within this space. While this paradigm is theoretically elegant and has been widely used for decades, it still suffers from two limitations in terms of optimization and alignment. This invention addresses the optimization difficulties, inter-modality alignment challenges, and neglect of one-to-many image-text pair relationships inherent in the traditional modality-sharing one-to-one embedding paradigm by proposing a novel one-to-many modality embedding alignment method that simultaneously improves upon both of these issues.

[0104] To evaluate the effectiveness of the proposed BEAT model, experiments were conducted on three widely used TRP datasets: CUHK-PEDES, ICFG-PEDES, and RSTPReID.

[0105] Table 1. Comparison with the SOTA method on the CUHK-PEDES dataset

[0106] Model R@1 R@5 R@10 CNN-RNN 8.07 - 32.47 GNA-RNN 19.05 - 53.64 PWM-ATH 27.14 49.45 61.02 GLA 43.58 66.93 76.26 MIA 53.10 75.00 82.90 A-GANet 53.14 74.03 81.95 ViTAA 55.97 75.84 83.52 IMG-Net 56.48 76.89 85.01 CMAAM 56.68 77.18 84.86 HGAN 59.00 79.49 86.60 DSSL 59.98 80.41 87.56 MGEL 60.27 80.01 86.74 SSAN 61.37 80.15 86.73 NAFS 61.50 81.19 87.51 TBPS 61.65 80.98 86.78 LapsCore 63.40 - 87.80 BEAT 64.23 82.91 88.65

[0107] Table 2. Comparison with the SOTA method on the ICFG-PEDES dataset.

[0108] Mode1 R@1 R@5 R@10 DualPath 38.99 59.44 68.41 CMPM+CMPC 43.51 65.44 74.26 MIA 46.49 67.14 75.18 SCAN 50.05 69.65 77.21 ViTAA 50.98 68.79 75.78 SSAN 54.23 72.63 79.53 BEAT 58.16 75.91 82.04

[0109] Table 3. Comparison with the SOTA method on the RSTPReID dataset

[0110] Model R@1 R@5 R@10 IMG-Net 37.60 61.15 73.55 AMEN 38.45 62.40 73.80 DSSL 32.43 55.08 63.19 ASPD-Net 39.90 65.15 74.40 SUM 41.38 67.48 76.48 BEAT 46.90 70.90 79.35

[0111] Compared to the previous state-of-the-art (SOTA) algorithms, BEAT achieves a relative performance improvement of +1.31% on the CUHKPEDES dataset (see Table 1 for details); +7.25% on the ICFG-PEDES dataset (see Table 2 for details); and +13.34% on the RSTPReID dataset (see Table 3 for details). Figure 3 The detection performance of the basic model is compared with that of the proposed BEAT model. Furthermore, to verify the generality of BEAT, experiments were conducted on three general and fine-grained image-text retrieval datasets: MS-COCO, Caltech-UCSD Birds (CUB), and Oxford-102 Flowers (Flowers). Specific results of the fine-grained retrieval experiments are shown below. Figure 4 As shown, by Figure 3 and 4As can be seen, compared with the original Base model, the BEAT model has improved recall under the condition of different recall criteria in the two datasets, thus demonstrating the effectiveness of the model algorithm.

Claims

1. A two-way one-to-many embedding alignment method for text-based person retrieval, characterized in that The method comprises the following steps: 1) extracting picture features using a pre-trained ResNet-50 backbone network, extracting text features using a pre-trained fixed BERT and a trainable Bi-LSTM model, and obtaining global visual and text features through global max pooling and a fully connected layer; The specific steps of obtaining the global visual and text features include: extracting picture features V ∈ R H×W×C using a pre-trained ResNet-50 backbone network N×C , H, W, and C represent the height, width, and channel number of the picture features respectively, using a pre-trained fixed BERT and a trainable Bi-LSTM model to extract text features T ∈ R H×W×C , N, and C represent the length and channel number of the text representation respectively; obtaining the global visual and text features through global maximum pooling GMP and a fully connected layer; wherein, are fully connected layer learnable parameters, C g denotes the channel dimension of global feature, v g ,t g respectively represent global visual feature and global text feature; 2) in order to obtain local features, the visual features V are uniformly divided into K non-overlapping parts, and K different text features are obtained by using a word attention module, and the visual local features and the text local features are extracted through global max pooling and a fully connected layer; 3) a non-local module is established, and non-local visual features and language features are obtained through cosine similarity; 4) a new bidirectional one-to-many embedding paradigm is established, which specifically comprises bidirectional embedding and one-to-many embedding; the bidirectional embedding is used for simultaneously projecting one feature to the feature space of the other feature with respect to the visual features and the language features; the one-to-many embedding is designed for the one-to-many relationship between the image and the text pair, an embedding module group with a residual structure is established, and then each module group is composed of M residual embedding modules, and each residual embedding module can embed the visual features into the text feature space or embed the text features into the visual feature space; through the bidirectional one-to-many embedding module, the global visual features and the global language features, the local visual features and the local language features, and the non-local visual features and the non-local language features generate multiple embedding features in the text and image spaces for matching; 5) two common loss functions, identity loss and composite ranking loss, are used, and the two are added to construct an overall loss function and are optimized.

2. The two-way one-to-many embedding alignment method for text-based character retrieval of claim 1, wherein In step 2), the specific steps of uniformly dividing the visual features V into K non-overlapping parts, and obtaining K different text features by using a word attention module, and extracting the visual local features and the text local features through global max pooling and a fully connected layer are as follows: The visual features are uniformly divided into K blocks, and the specific expression is as follows: For text features, a word attention mechanism module (WAM) is used to extract K local text features, which is specifically represented as Then, global max pooling and a fully connected layer are used to obtain visual local features and text local features, which are specifically represented as follows: wherein, 3. The two-way one-to-many embedding alignment method for text-based character retrieval of claim 2, wherein In step 3), the specific steps of establishing a non-local module and obtaining non-local visual features and language features through cosine similarity are as follows: After computing the linear embedding the cosine similarity score s between ki , which is computed as follows: wherein, The local visual features are then aggregated based on the similarity scores, as follows: wherein, Finally, the non-local visual feature is specifically calculated as follows: wherein Similarly, non-local text features are also obtained in this way.

4. The two-way one-to-many embedding alignment method for text-based character retrieval of claim 3, wherein In step 4), the specific method of the bidirectional embedding is as follows: the model keeps the text features unchanged and only projects the visual features into the text feature space for matching; since the text features in the text space are fixed, the visual features have a clear optimization direction, which is conducive to improving the optimization stability; then the model keeps the visual features unchanged and only projects the text features into the image space for alignment; finally, the construction formula of the bidirectional embedding module for the global features is as follows: v g-txt = f v2t (v g ) v g-img = f t2v (t g ) where v g-txt is the global visual feature in the text space, v g-img is the global text feature in the image space; f v2t (·) is the projection function from visual feature to text space, f t2v (·) is the projection function from text feature to image space.

5. The two-way one-to-many embedding alignment method for text-based character retrieval according to claim 4, characterized in that In step 4), the one-to-many embedding is realized by using a residual embedding module group REM-G; each REM-G is composed of M residual embedding modules REMs; for each residual embedding module, the model uses a residual-based linear embedding to process the global features; The processing process of the mth residual embedding module is as follows: wherein, are learnable parameters of the mth residual embedding module, are global visual features in the text space and global text features in the image space, respectively.

6. The two-way one-to-many embedding alignment method for text-based character retrieval of claim 5, wherein In step 5), the formula of the identity loss is as follows: where v, t, y are visual features, text features and corresponding ID numbers, respectively, are learnable parameters; C in Q represents the dimension of input features and the number of ID numbers of people, respectively.

7. The two-way one-to-many embedding alignment method for text-based character retrieval of claim 6, wherein In step 5), the loss formula of the composite ranking is as follows: where (v, t) and (v, t') represent the matched correct image-text pair, where (v, t) and (v, t') represent the matched correct image-text pair, The model aligns the two modality-specific spaces, and each sample produces M features after projection. The similarity scores calculated on multiple embedding features in these two spaces are added as the final similarity score, and the formula is: The final loss function L is the sum of the ID loss and the CR loss: L = L ID + L CR .

Images