A BERT-based text-to-image cross-modal person re-identification method

Abstract

The present invention discloses a BERT-based text-to-image cross-modal pedestrian re-identification method. The method comprises the following steps: first, using SR-ResNet50 to extract image features; and using BERT and ResNet10 cascade to extract text features. Secondly, in order to make the model focus on important words in the text and significant areas in the image, RCBAM modules are added to the image branch, text branch and CRG network branch. In order to eliminate the modal differences between text and image, the text and image features are spliced and input into the CRG network to learn the modality-invariant feature representation. Finally, the cross-modal projection matching loss, cross-modal projection classification loss and cross-modal sampling triplet loss are combined to train the optimization model, and finally, the accuracy of cross-modal pedestrian re-identification is improved. The present invention is mainly used in video surveillance intelligent analysis application systems, and has broad application prospects in the fields of text-to-image retrieval, re-identification of intelligent security, etc.

Description

Technical Field

[0001] The present invention relates to a BERT-based text-to-image cross-modal pedestrian re-identification method and a new BERT-based text-to-image cross-modal pedestrian re-identification framework. It involves the problem of cross-modal pedestrian re-identification from natural language text descriptions to images in the field of video intelligent surveillance and belongs to the field of computer vision and intelligent information processing. Background Art

[0002] Image-to-image person re-identification (Re-ID) technology has been widely used in fields such as security and autonomous driving. However, in criminal investigations and many crime scenarios, surveillance cameras are usually unable to capture photos of suspects or criminals. Unlike image-to-image Re-ID methods, the Re-ID method based on natural language text description to image (Text-to-Image based Re-ID, TI-ReID) does not require any images of the query person. For example, in the security field, in many cases it is difficult to obtain query images, and we can only search for the corresponding target person in the dataset based on the natural language description of the criminal's appearance by the eyewitness. Figure 1 As shown, given a natural language text description of a specific query person, it aims to retrieve pedestrian images that best match the text description from a large-scale person image database.

[0003] The introduction of relevant image-language description datasets and deep learning techniques for TI-ReID has significantly advanced research in this area. Early TI-ReID methods mostly employed independent image encoders (e.g., CNNs) and text encoders (e.g., RNNs and Bi-LSTMs), then performed similarity matching between text and image features. In recent years, to bridge the gap between the text and image domains, researchers have proposed a series of TI-ReID methods, broadly categorized as global matching and local matching. Global matching methods focus on learning global visual and textual representations to obtain a unified feature space without considering modality differences. However, images contain many unique local details that are difficult to mine using global representation extraction. Furthermore, the presence of irrelevant regions in images introduces noise to the global information. To further mine discriminative and comprehensive information, several local alignment matching methods have been proposed, matching person images with text descriptions through local alignment. However, while local matching methods have achieved promising performance, they introduce additional models such as human pose estimation, semantic segmentation, or attribute recognition, making them difficult to train and test end-to-end. Some methods also use a multi-granularity similarity measurement strategy. This means that during the test phase, these methods need to learn multiple local representations for each image or text and repeatedly calculate local similarities. Both the additional model and the complex similarity measurement are quite time-consuming. Therefore, it is necessary to design a simple and easy-to-use framework for the TI-ReID problem.

[0004] With the emergence of Transformers and BERT, using BERT pre-trained models to extract text features for cross-modal text-to-image Re-ID has become increasingly popular. Because BERT pre-trained models are learned from large-scale corpora, they power heterogeneous downstream text-to-image tasks. For image features, we use the SR-ResNet50 network, which includes a residual network module with spatial and channel attention, to effectively extract more local details of pedestrians.

[0005] Most existing Re-ID methods based on text descriptions attempt to learn a joint mapping that projects the corresponding text-image pairs into a shared subspace. However, these methods often ignore the association between the details of the language description and the details of the image. To overcome this problem, we concatenate the text description features and image features and then learn them jointly and sharedly. As a result, the model can learn the intrinsic association between language and vision. Since the text-image matching dataset we use is annotated with identities, in order to learn more discriminative features and fully utilize the text-image matching relationship, we perform a cross-modal projection identity loss on both images and texts. In particular, to eliminate the feature differences between image and text modalities, we introduce a cross-modal projection matching loss that fuses the cross-modal feature projection into the KL divergence to associate the representations of different modalities.

[0006] The triplet loss function is a common metric loss and is widely used in the field of image retrieval. The triplet loss function not only has the characteristic of reducing the intra-class distance, but also has the characteristic of increasing the inter-class distance. This characteristic makes the triplet loss function very suitable for pedestrian Re-ID network training. In recent years, scholars have continuously improved and developed the triplet loss. The most widely used one is the difficult triplet loss. In the TI-ReID task, in addition to paying attention to the intra-modal differences, I need to pay more attention to the impact of inter-modal differences. In view of the huge differences between texts and images of different modalities in the TI-ReID dataset, we propose a cross-modal difficult sampling triplet joint loss It consists of three parts of loss: global hard-to-sample triplet loss Intra-modal hard-to-sample triplet loss and inter-modality hard-to-sample triplet loss Since there are two modalities in the TI-ReID task, this paper expands the scope of difficult sample selection to text and image modalities, and obtains the global difficult triplet loss. This loss focuses more on intra-modal variation, complementing the global triplet loss. In TI-ReID, while inter-modal variation can be greater than intra-modal variation, correctly identifying intra-modal variation is also a crucial component of validating the Re-ID algorithm. Images of the same person in different modalities, as well as text descriptions of the same person in different modalities, can vary significantly. Images of different people in the same modality, as well as text descriptions of different people in the same modality, can also be similar. These factors can affect the model's recognition performance. Therefore, the intra-modal hard-to-sample triplet loss is proposed to enhance learning of intra-modal variation. The loss focuses more on inter-modal changes. In the TI-ReID problem, the model needs to pay more attention to inter-modal changes, because inter-modal changes cannot be distinguished by color, clothing style, etc., but can only be distinguished by features such as clothing attribute description and style description. Therefore, the inter-modal difficult sampling triplet loss is proposed, which adds inter-modal change learning and uses it as The loss complements the final result and improves the accuracy of cross-modal person re-identification. Summary of the Invention

[0007] This paper proposes a BERT-based text-to-image cross-modal person re-identification method and designs a new BERT-based dual-path natural language text description-to-image cross-modal person re-identification framework, named BDNet. BDNet leverages the intrinsic connection between images and text descriptions to transfer embedding representations from one modality to the other, improving the accuracy of cross-modal Re-ID. For the image branch, the paper proposes an SR-ResNet50 backbone network to extract image features. This network can better guide the model to focus on the target object and its salient areas, providing a highly discriminative visual feature representation for the subsequent Re-ID model. For the text branch, a pre-trained BERT model is used to extract text features, and the BERT output features are further represented using ResNet10. This can better extract the more critical attribute semantic information in the text description and enhance the text feature representation capability. To mitigate the difference between text and image modalities, the paper combines a cross-modal projection matching loss, a cross-modal projection classification loss, and a cross-modal hard-to-sample triplet loss to jointly train and optimize the model. In addition, in order to strengthen the connection between key attribute phrases in the text description and local blocks of the image, and let the model learn to pay attention to more important attribute words in the text description and salient areas in the image, the RCBAM attention module is used in the image branch, text branch and shared CRG network, ultimately achieving the goal of improving the accuracy of cross-modal pedestrian re-identification.

[0008] A BERT-based text-to-image cross-modal person re-identification method includes the following steps:

[0009] (1) First, a pedestrian image of size 384×128 is input into the SR-ResNet50 network to obtain the image feature f V ; Input the text into the BERT network to obtain the feature f t , then f t Input ResNet10 network to get text feature f T ;Finally, the feature f V and f T Input into the VG module respectively to obtain features and

[0010] (2) In order to eliminate the difference between text and image modalities, the global maximum pooling is used to pool the image features f V and text features f T Process and obtain features and Then use the "Concatenate" operation to concatenate the features and Splice and get the fusion feature f V+T , and finally f V+T Send it to the CRG module to get the feature f′ V+T ;Learn two modality-invariant feature representations through CRG shared network;

[0011] (3) In order to enhance the connection between key attribute phrases in the text and local image, and let the model learn to focus on important attribute words in the text description and salient regions in the image, the RCBAM attention module is used in the image branch, text branch and shared CRG network;

[0012] (4) Use cross-modal projection matching loss, cross-modal projection classification loss and cross-modal difficult-to-sample triplet loss to jointly train and optimize the network model, ultimately effectively improving the accuracy of text-to-image cross-modal pedestrian re-identification. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 Schematic diagram of the text-to-image cross-modal re-identification process;

[0014] Figure 2 BDNet network structure;

[0015] Figure 3 Experimental comparison of global maximum pooling (GMP) and global average pooling (GAP);

[0016] Figure 4 ResNet50 structure diagram;

[0017] Figure 5 SR-ResNet50 structure diagram;

[0018] Figure 6 CBAM attention module structure diagram;

[0019] Figure 7 RCBAM attention module structure diagram;

[0020] Figure 8 Visual comparison of CBAM module and RCBAM module;

[0021] Figure 9 Performance comparison of CBAM and RCBAM attention modules;

[0022] Figure 10 Channel Attention Module (CAM) structure diagram;

[0023] Figure 11 Spatial Attention Module (SAM) structure diagram;

[0024] Figure 12 Bottleneck structure diagram;

[0025] Figure 13 ResNet10 structure diagram;

[0026] Figure 14 BERT text feature extraction diagram;

[0027] Figure 15 Rendering of text-to-image cross-modal re-identification. DETAILED DESCRIPTION

[0028] The following is combined with Figure 1 , Attachment Figure 2 , Attachment Figure 3 , Attachment Figure 4 , Attachment Figure 5 , Attachment Figure 6 , Attachment Figure 7 , Attachment Figure 8 , Attachment Figure 9 , Attachment Figure 10 , Attachment Figure 11 , Attachment Figure 12 , Attachment Figure 13 , Attachment Figure 14 and attached Figure 15 The present invention will be further described:

[0029] Figure 1 This is a flowchart of the text-to-image cross-modal re-identification (TI-ReID) process:

[0030] The specific process of the TI-ReID task is: given a text description of a specific query person, the pedestrian image that best matches the text description is retrieved from a large-scale person image database.

[0031] (1) BDNet working principle

[0032] Figure 2 The BDNet network structure diagram is shown in Figure 2. The BDNet network model framework is trained and tested in an end-to-end manner. The BDNet network mainly consists of three parts: image feature extraction module, text feature extraction module, feature embedding module and its joint loss constraint module. Assume that each batch of pedestrian images and text descriptions are training data Where N represents the number of image-text pairs in each batch, and each batch of data consists of a pedestrian image V and the corresponding text description T.

[0033] For image feature extraction, the image of size 384×128 is input into the SR-ResNet50 backbone network to extract image features and obtain image features. Then, f V Input the VG module to obtain the final visible light pedestrian image features The SR-ResNet50 backbone network can better guide the model to focus on the target object and its salient areas, effectively improving the feature representation capability of the Re-ID model.

[0034] For text feature extraction, we first use the pre-trained BERT model to extract the input text description and obtain text features. Then the feature f t Send it to the ResNet10 network to get the features Then f T Input VG module to get the final text features ResNet10 is used to further learn the feature representation of the text representation output by Bert, thereby learning a more discriminative text description feature representation.

[0035] In order to eliminate the difference between text and image modalities, we concatenate image and text features, and then use a shared network to uniformly process visual features and text features, which helps the network learn more modality-invariant feature representations. Specific operation: Since the first two dimensions (N and C) of image and text features are the same, only the last two dimensions (H and W) need to be unified. Here we use global maximum pooling to unify the H and W dimensions. Perform global maximum pooling and get in Text features Perform global maximum pooling and get in We then use the "Concatenate" operation to combine the image features and text features Splice and get f V+T ,in Finally, f V+T Enter the shared CRG network module (see Figure 2 ) and then get the feature f' V+T After the "Concatenate" operation, we can uniformly process visual features and text features through a shared network, which helps the network learn more modality-invariant feature representations.

[0036] Figure 3As shown in the figure, we experimentally compared the performance of Global Max Pooling (GMP) and Global Average Pooling (GAP). The results show that GMP performs slightly better than GAP, so we ultimately chose GMP. The reasons for this choice are: if we expect all feature map information to contribute, then GAP will be more effective, such as in tasks such as image classification and semantic segmentation; if we expect to extract the part with the largest and strongest response to serve in the next stage, while also reducing the impact of useless information, then GMP will be more effective; the present invention expects to obtain significant and prominent feature representation, so it uses the GMP operation.

[0037] Figure 2 As shown, for the VG component, the x.view(b,c,-1) operation is used to modify the input image feature f V and text features f T The shape of the tensor is that the H and W dimensions are merged and then subjected to the TM operation (i.e. torch.mean(x p , dim=-1)) and then output a feature vector of size (N, 2048).

[0038] Figure 2 As shown, for the CRG shared network component, the input features (b, c, h, w) are sequentially subjected to 1×1 convolution, batch normalization, and Relu nonlinear activation; then fed into the RCBAM module; next, h and w are merged into one dimension through the x.view(b, c, -1) operation, and finally through the TM operation (i.e., torch.mean(x p , dim=-1)) and then output a feature vector of size (N,2048).

[0039] (2) SR-ResNet50 structure

[0040] In the image branch, we select SR-ResNet50 as the image feature extraction network. Figure 5 As shown in Figure 3, the structure of SR-ResNet50 is derived from the evolution of ResNet50. Figure 4As shown in the figure, the ResNet50 network contains 5 stages, namely stage0, stage1, stage2, stage3, and stage4. The structure of Stage0 is relatively simple and can be regarded as a preprocessing of the input; the structures of stages1 to 4 are relatively similar; the present invention adds an RCBAM module after stages1 to 4 of ResNet50, respectively, and names it the SR-ResNet50 network structure.

[0041] As shown in Table 1, to verify the location and number of RCBAM modules, we experimentally evaluated and compared the impact of different numbers of RCBAM modules on the experiment. The experimental results show that adding RCBAM modules after stages 1 to 4 achieves the best performance! Compared with not adding RCBAM modules, Rank1 improves by 2.18% and mAP improves by 2.95%. Compared with the original ResNet50 network, the SR-ResNet50 network can more effectively guide the model to focus on the target object and its salient areas, effectively improving the representation ability of image features.

[0042] (3)RCBAM module

[0043] Figure 6 Block diagram of the attention module for CBAM (Channel and Spatial Attention Module without Residual). Figure 7 This is a block diagram of the attention module structure of the RCBAM (Residual Channel and Spatial Attention Module). The RCBAM module primarily consists of a Channel Attention Module (CAM) and a Spatial Attention Module (SAM). In the RCBAM module, input features are first subjected to a "channel attention" operation, followed by a "spatial attention" operation. Finally, the output features are added to the input features for output. This addition operation accelerates module training and improves model training performance without adding additional parameters or computational overhead.

[0044] f c =ψ c (f)(1)

[0045] f s =ψ c (f′) (2)

[0046]

[0047] in represents element-by-element addition, represents the element-by-element product, ψ c represents the channel attention operation, ψ sDenotes the spatial attention operation. During the addition process, the corresponding attention values are propagated (copied): the channel attention values are propagated along the spatial dimension and vice versa. f″ is the final refined output. Figure 4 The computation process of each attention map is described.

[0048] Adding the RCBAM attention module to the image branch network guides the network to better focus on the target object in the image; adding the RCBAM attention module to the text branch allows the network to better focus on key attribute words in the text. We also added the RCBAM module to the shared splicing network to enable the network to better learn to focus on modality-invariant feature representations.

[0049] In order to effectively explain the role of RCBAM, we experimentally visualized the features output by the RCBAM module and the CBAM module. Figure 8 (C) It can be observed that the model trained using RCBAM pays more attention to the local details of pedestrians. Figure 9 As shown in Figure 3, we experimentally compared the performance of the CBAM and RCBAM modules. Compared with CBAM, the RCBAM module improved Rank-1 by 2.53% and mAP by 1.86%. The experiment proves that the RCBAM module with residual channel and spatial attention performs better.

[0050] Figure 10 Figure 1 shows the structure of the Channel Attention Module (CAM). We exploit the channel relationship between features to generate a channel attention map. To efficiently compute channel attention, we compress the spatial dimensions of the input feature map. Global average pooling is currently widely used to aggregate spatial information. Since max pooling collects the largest and most responsive parts of features—that is, we expect to capture salient features—we can achieve better channel attention. Therefore, we use both global average pooling and global max pooling.

[0051] First, we use global average pooling and global max pooling operations to aggregate the spatial information of feature maps and generate two different spatial context descriptors. and Denote the average pooling feature and the maximum pooling feature respectively. Then, these two descriptors are input into a shared network MLP. We generate our channel attention feature ψ c The shared network consists of a multilayer perceptron (MLP) and one hidden layer. To reduce parameter overhead, the hidden activation size is set to where R is the reduction ratio. After applying the shared network to each descriptor, we sum the elements using operation to merge the output feature vectors. In short, channel attention is calculated as:

[0052]

[0053] Where σ represents the sigmoid function, and in Note that the MLP weights W0 and W1 are shared for both inputs, and the ReLU activation function follows W0.

[0054] Figure 11 For the spatial attention module structure diagram, input features Two 2D feature maps are obtained by using average pooling and maximum pooling operations at the channel level: and They are then concatenated and convolved using a standard convolutional layer to generate our 2D spatial attention map. In short, the spatial attention is calculated as:

[0055]

[0056] Where σ represents the sigmoid function, f 3×3 Represents a convolution operation with a filter size of 3×3.

[0057] (4) Text feature extraction module

[0058] The advantage of pre-trained text models is that they don't require extensive training data for specific scenarios, saving time and increasing efficiency. We selected the BERT pre-trained model, known for its strong generalization capabilities, to represent text descriptions. BERT's model architecture is a multi-layer bidirectional transformer encoder based on the original implementation. Because transformers are widely used and our implementation is nearly identical to the original, we omit a detailed background description of the model architecture.

[0059] Figure 14 As shown in Figure 1, for a given token, the input representation is constructed by adding the corresponding token, segment, and embedding position. Given a sentence, we apply basic tokenization and split it into words, which are then processed sequentially using BERT. In addition, to further enhance the text feature representation capability of BERT's output, we add a ResNet10 network module after the BERT module. Figure 13 This is the ResNet10 network structure diagram, which includes a 1×1 convolution, 3 Bottleneck modules and 1 RSCAM module. Figure 12 This is the Bottleneck module structure diagram.

[0060] (5) Loss function

[0061] A. Cross-modal Projection Matching Loss Function

[0062] Since there is a huge modality difference between text and images, in order to effectively alleviate the impact of cross-modal differences, we introduce the cross-modal projection matching loss L CMPM To constrain the model. CMPM The loss integrates cross-modal projections into the KL divergence to correlate representations of different modalities, ultimately improving recognition performance.

[0063] Given a mini-batch containing N image and text pairs, for each image v i , the image and text pairs are constructed as where y i,j =1 means (v i ,t j ) is a matching image pair, and y i,j = 0 means the image pair does not match. i and t j The probability of is defined as:

[0064]

[0065] where p i,j It can be seen as all pairs in a small batch (v i ,t j ) is the percentage of the scalar projection of t j Represents a normalized text feature. Represents the scalar projection image feature v i To text feature t j Therefore, the greater the similarity between image features and text features, the larger the scalar projection between image features and text features. Consider the fact that in each mini-batch, the image sample v i There may be multiple matching text samples, (v i ,t j ) is normalized to the true matching probability:

[0066]

[0067] where q i,j yes and The normalized true matching probability between , since there may be multiple matching text descriptions in one batch.

[0068] V i The association with each batch of correctly matched text samples is defined as:

[0069]

[0070] where ε is a small number to avoid numerical problems.

[0071] is the constraint between each image representation and its matching text representation. In turn, we process the class, and the constraint between each text representation and its matching image representation is

[0072]

[0073] B. Cross-modal projection classification loss function

[0074] The commonly used recognition loss in face recognition applications is the norm-softmax cross entropy loss, which performs L2 normalization on the weights of the output layer. Note that the norm-softmax loss is modified on top of the softmax loss. The improved softmax loss algorithm can obtain more compact intra-class discriminant features and more sparse inter-class discriminant features. Assume that the image features we extract are Text features The label set is In order to improve the classification process of image features v i The discriminative ability of the image classification norm-softmax loss (L norm-softmax ) can be calculated as:

[0075]

[0076] where y i represents y i Tags, W yj and W j y represents the weight matrix W i -th and j-th columns. L norm-softmax The loss normalizes all weight vectors to the same length to reduce the influence of weight size on distinguishing different samples.

[0077] Here, we do not classify the original features, but instead project the image features onto the projections of the corresponding text features for classification. Cross-modal projection integrates the similarity between image and text into the classification, thereby strengthening the association between matching pairs. We also incorporate cross-modal projection into norm-softmax to encourage features to be more compactly distributed along the weight vector. The cross-modal projection classification loss function is defined as follows:

[0078]

[0079] in Represents the image feature v iNormalized text features t i Intuitively, all matching text samples need to be located within the image feature v i Projection to w yi direction to promote correct classification. Text classification features The loss is calculated in a similar way:

[0080]

[0081] The final cross-modal projection classification loss can be calculated as:

[0082]

[0083] As shown in Table 2, compared with strategies 2 and 3, Rank-1 improved by 0.67% and mAP improved by 0.09%. Compared with strategies 5 and 6, Rank-1 improved by 0.25%. By adding the cross-modal projection classification loss constraint, our model performance has been improved to a certain extent.

[0084] C. Joint loss function for cross-modal difficult sample triplets

[0085] The triplet loss function not only has the characteristic of reducing the intra-class distance, but also has the characteristic of increasing the inter-class distance. This characteristic makes the triplet loss function very suitable for pedestrian Re-ID network training. In addition to paying attention to the intra-modal differences, the TI-ReID task also needs to pay special attention to the impact of the differences between text and image modalities. Therefore, we propose a cross-modal difficult sample triplet joint loss L CMHTF To overcome the above problems. CMHTF By global hard sample triplet loss Intra-modal hard sample triplet loss and inter-modality hard sample triplet loss It consists of three parts.

[0086] In the TI-ReID task, we expand the scope of difficult sample selection to two modalities (text and image) and modify the mini-batch. Each mini-batch randomly samples P pedestrian categories, and then randomly selects K pictures and K text descriptions from each pedestrian category, resulting in a total of 2PK picture and text sample pairs. Global difficult triplet loss The definition is as follows:

[0087]

[0088] Where D(·) represents the Euclidean distance, the anchor point The selection range is a set of two modes (picture and text), is a sample of the same modality as the anchor category, are samples of the same modality but different from the anchor category.

[0089] Compared with the global triplet loss, the intra-modal triplet loss focuses more on intra-modal changes. It is a supplement to the global triplet loss. In TI-ReID, although the inter-modal differences are greater than the intra-modal differences, correctly identifying the intra-modal differences is also an important part of testing the pedestrian re-identification algorithm. There are large differences between different pictures of the same pedestrian in the same modality, and pictures of different pedestrians in the same modality are very similar, which will affect the recognition effect of the model. Therefore, we propose the intra-modal triplet loss to increase the learning of intra-modal changes. Inter-modal difficult sample triplet loss The definition is as follows:

[0090]

[0091] Among them, when a≤K, E={1,2,···,K}; when a>K, E={k+1,k+2,···,2K}. represents samples of the same modality with the same category as the anchor point, Represents samples of the same modality with different categories from the anchor.

[0092] Compared with the global triplet loss, the inter-modal triplet loss focuses more on inter-modal changes. In the TI-ReID problem, the model needs to pay more attention to inter-modal changes, because inter-modal changes cannot be distinguished by color, clothing style, etc., but can only be distinguished by features such as clothing attribute description and style description. Therefore, this paper proposes the inter-modal triplet loss, which adds inter-modal change learning and uses it as a supplement to the global triplet loss. Intra-modal difficult sample triplet loss is defined as follows:

[0093]

[0094] Where, when a≤K, E={k+1,k+2,···,2K}; when a>K, E={1,2,···,K}. Image Here, a∈{1,2,···,K} represents the anchor image sample, and a∈{k+1,k+2,···,2K} represents the anchor sample is text. represents samples of the same modality with the same category as the anchor point, Represents cross-modal samples that are different from the anchor category.

[0095] The joint loss of cross-modal difficult sample triplets is defined as follows:

[0096]

[0097] As shown in Table 2, compared with strategies 3 and 4, Rank-1 improved by 1.22% and mAP improved by 0.98%. Compared with strategies 6 and 7, Rank-1 improved by 0.55% and mAP improved by 0.09%. Compared with strategies 6 and 7, Rank-1 improved by 0.34% and mAP improved by 0.37%. Compared with strategies 8 and 9, Rank-1 improved by 0.65% and mAP improved by 0.38%. Experimental results show that the joint loss of cross-modal difficult sample triplets also plays a role in improving experimental performance.

[0098] D. Total loss function

[0099] In order to enable the model to learn more discriminative image and text features, we set L CMPM , L CMPC and L CMHTF The three loss functions are jointly trained to optimize the model. The experimental results in Table 2 show that the joint loss constraint effectively improves the cross-modal re-identification performance. The total loss is defined as follows:

[0100] L=λ1L CMPM +λ2L CMPC +λ3L CMHTF (19)

[0101] Among them, the hyperparameters λ1 is 1, λ2 is 1, and λ3 is 0.05.

[0102] The present invention conducts a network structure ablation study on the CUHK-PEDES dataset, where strategy 1 in Table 2 represents the baseline network, RE represents random erasure, RCBAM represents the spatial and channel attention module with residual, and f V+T Indicates that image and text features are concatenated in the batch dimension; L CMPM represents the cross-modal projection matching loss; L CMPC represents the cross-modal projection classification loss; L CMHTF Denotes triplet loss. We also compared this method with several mainstream algorithms using a single query setting and using Rank-1, Rank-10, Rank-20, and mAP as evaluation metrics. The experimental results are shown in Tables 2 and 3, demonstrating significant improvements in accuracy compared to the baseline network and other comparison algorithms.

[0103] Table 1 RCBAM ablation experiments at different stages of Resnet50

[0104]

[0105] Table 2 Ablation study on CUHK-PEDES data

[0106]

[0107] Table 3 Comparison of the results of the proposed method with those of mainstream algorithms on the CUHK-PEDES dataset

[0108]

Claims

1. A BERT-based text-to-image cross-modal person re-identification method, It is characterized by the following steps: (1) First, input the pedestrian image of size 384×128 into the SR-ResNet50 network to obtain the image feature f V , the network adds an RCBAM module after stage 1 to 4 of ResNet50 respectively; input the text into the BERT network to obtain the feature f t , then f t Input ResNet10 network to get text feature f T ;Finally, the feature f V and f T Input into the VG module respectively to obtain features and (2) In order to eliminate the difference between text and image modalities, the global maximum pooling is used to pool the image features f V and text features f T Process and obtain features and Then use the "Concatenate" operation to concatenate the features and Splice and get the fusion feature f V+T , and finally f V+T Send it to the CRG module to get the feature f′ V+T ;Learn two modality-invariant feature representations through CRG shared network; (3) To enhance the connection between key attribute phrases in the text and local image parts, and to enable the model to learn to focus on important attribute words in the text description and salient regions in the image, the RCBAM attention module is used in the image branch, text branch, and shared CRG network. (4) Using cross-modal projection matching loss, cross-modal projection classification loss, and cross-modal hard-to-sample triplet loss to jointly train and optimize the network model, ultimately effectively improving the accuracy of text-to-image cross-modal person re-identification; The RCBAM module consists of two modules: the channel attention module (CAM) and the spatial attention module (SAM). In the RCBAM module, the input features are first subjected to the "channel attention" operation, then the "spatial attention" operation, and finally the output features are added to the input features and output. The addition operation speeds up module training and improves model training results without adding extra parameters or computational complexity. f c =ψ c (f) (1) f s =ψ c (f′) (2) in represents element-by-element addition, represents the element-by-element product, ψ c represents the channel attention operation, ψ s represents the spatial attention operation; during the addition process, the corresponding propagation attention value: the channel attention value is propagated along the spatial dimension and vice versa; f″ is the final refined output; Adding the RCBAM attention module to the image branch network can guide the network to better focus on the target object in the image; adding the RCBAM attention module to the text branch can enable the network to better focus on key attribute words in the text; at the same time, we also added the RCBAM module to the shared CRG network part to enable the network to better learn to focus on modality-invariant feature representations; We use the channel relationship between features to extract channel attention maps. To effectively calculate channel attention, we compress the spatial dimensions of the input feature map. For the aggregation of spatial information, the global average pooling method is currently widely used. Since maximum pooling can collect the largest and most responsive parts of the features, that is, we hope to obtain significant features, thus achieving better channel attention, we use both global average pooling and global maximum pooling features. First, we use global average pooling and global maximum pooling operations to aggregate the spatial information of feature maps and generate two different spatial context descriptors; and Represent the average pooling features and the maximum pooling features respectively; Then, these two descriptors are fed into a shared network MLP, where We generate our channel attention feature ψ c ; The shared network consists of a multilayer perceptron (MLP) and one hidden layer; To reduce parameter overhead, the hidden activation size is set to where R is the reduction ratio; after applying the shared network to each descriptor, we use the element-wise summation operation to merge the output feature vectors; in short, channel attention is calculated as: Where σ represents the sigmoid function, and in Note that the MLP weights W0 and W1 are shared for both inputs, and the ReLU activation function is followed by W0; Input features Two 2D feature maps are obtained by using average pooling and maximum pooling operations at the channel level: and They are then concatenated and convolved using a standard convolutional layer to generate our 2D spatial attention map; in short, the spatial attention is calculated as: Where σ represents the sigmoid function, f 3×3 Represents a convolution operation with a filter size of 3×3.

2. The method according to claim 1, wherein in step (1), it is assumed that the training data pair is Where N represents the number of image-text pairs in each batch, and each batch of data consists of an image V and a corresponding text description T. For each pedestrian image feature extraction, the pedestrian image of size 384×128 is first input into the SR-ResNet50 network to obtain the image features. Then f V Input VG module to obtain the final pedestrian image features For the text feature extraction of each natural language text description, first use the text description to input the BERT pre-training model to obtain the text features Then the feature f t Send it to the ResNet10 network to get text features Finally, f T Input VG module to get the final text features Given a sentence, we apply basic tokenization and split it into words, then process them sequentially using BERT. Furthermore, to further enhance the text feature representation capabilities of BERT's output, we cascade a ResNet10 network module after the BERT module. ResNet10 is used to further extract features from the text representation output by BERT, resulting in a more discriminative text feature representation. For the VG component, the x.view(b,c,-1) operation is used to modify the input image feature f V and text features f T The shape of the tensor is that the H and W dimensions are merged, and then after the TM operation, the output is a feature vector of size (N, 2048); For the CRG shared network component, the input features (b, c, h, w) are sequentially subjected to 1×1 convolution, BatchNormalization, and Relu nonlinear activation, and then fed into the RCBAM module. Next, h and w are merged into one dimension through the x.view(b,c,-1) operation, and finally a TM operation is performed to output a feature vector of size (N,2048).

3. According to the method of claim 1, in step (2), in order to eliminate the difference between text and image modalities, we splice the image and text features, and then unify the visual features and text features through a shared network, so as to facilitate the network to learn more modality-invariant feature representations; since the first two dimensions N and C of image and text features are the same, only the last two dimensions H and W need to be unified. Here, we use global maximum pooling to unify the H and W dimensions; for image features, Perform global maximum pooling and get in Text features Perform global maximum pooling and get in Then we use the "Concatenate" operation to combine the image features and text features Splice and get f V+T ,in Finally, f V+T After being fed into the CRG module, the feature f′ is obtained V+T After the "Concatenate" operation, we can uniformly process visual features and text features through a shared network, which helps the network learn more modality-invariant feature representations.

4. The method according to claim 1, characterized in that In step (4), a cross-modal joint loss is used to train and optimize the network model. The joint loss includes cross-modal projection matching loss, cross-modal projection classification loss, and cross-modal difficult-to-sample triplet joint loss. First, the cross-modal projection matching loss function; Due to the huge modality difference between text and image, in order to effectively reduce the impact of cross-modal differences, we introduce the cross-modal projection matching loss L CMPM To constrain the model; L CMPM The loss improves the recognition performance by integrating cross-modal projection into KL divergence to associate representations of different modalities. Given a mini-batch containing N image and text pairs, for each image v i , the image and text pairs are constructed as where y i,j =1 means (v i ,t j ) is a matching image pair, and y i,j =0 means unmatched image pair; matching v i and t j The probability of is defined as: where p i,j It can be seen as all pairs in a small batch (v i ,t j )'s scalar projection percentage; t j Represents normalized text features; Represents the scalar projection image feature v i To text feature t j Therefore, the greater the similarity between image features and text features, the larger the scalar projection between image features and text features; Consider the fact that in each mini-batch, the image sample v i There may be multiple matching text samples, (v i ,t j ) is normalized to the true matching probability: where q i,j yes and The normalized true matching probability between , since there may be multiple matching text descriptions in a batch; V i The association with each batch of correctly matched text samples is defined as: in ε is a small number to avoid numerical problems; is the constraint between each image representation and its matching text representation. In turn, we process the class, and the constraint between each text representation and its matching image representation is Second, cross-modal projection classification loss function: The commonly used recognition loss in face recognition applications is norm-softmax cross entropy loss, which performs L2 normalization on the weights of the output layer; Note that norm-softmax loss is modified on top of softmax loss; The improved softmax loss algorithm can obtain more compact intra-class discriminant features and more sparse inter-class discriminant features; Assuming that the image features we extracted Text features The label set is In order to improve the classification process of image features v i The discriminative ability of the image is the norm-softmax loss L norm-softmax It can be calculated as: where y i represents y i Tags, W yj and W j Representation weight matrix W of y i -th and j-th columns; L norm-softmax The loss normalizes all weight vectors to the same length to reduce the impact of weight size on distinguishing different samples; The SR-ResNet50 network does not classify the original features, but instead projects the image features onto the projections of the corresponding text features for classification. Cross-modal projection integrates the similarity between image and text into the classification, thereby strengthening the association between matching pairs. In addition, cross-modal projection is incorporated into norm-softmax to encourage features to be more compactly distributed along the weight vector. The cross-modal projection classification loss function is defined as follows: in Represents the image feature v i Normalized text features t i vector projection of the image feature v; intuitively, all matching text samples need to be located at the image feature v i Projection to w yi direction, to facilitate correct classification; Text classification features The loss is calculated in a similar way: The final cross-modal projection classification loss can be calculated as: Third, the cross-modal difficult sample triplet joint loss function: The triplet loss function not only has the characteristics of reducing the intra-class distance, but also increases the inter-class distance; this characteristic makes the triplet loss function very suitable for pedestrian Re-ID network training; the TI-ReID task needs to pay attention to the influence of the difference between text and image modalities in addition to the intra-modal difference; therefore, the cross-modal difficult sample triplet joint loss L is proposed CMHTF To solve the above problems; L CMHTF By global hard sample triplet loss Intra-modal hard sample triplet loss and inter-modality hard sample triplet loss It consists of three parts; In the TI-ReID task, we expand the scope of difficult sample selection to two modalities, namely text and image, and modify the mini-batch. Each mini-batch randomly samples P pedestrian categories, and then randomly selects K pictures and K text descriptions from each pedestrian category, resulting in a total of 2PK picture and text sample pairs; the global difficult triplet loss The definition is as follows: Where D(·) represents the Euclidean distance, the anchor point The selection range is two modal sets, namely pictures and texts. is a sample of the same modality as the anchor category, is a sample of the same modality that is different from the anchor category; Compared to the global triplet loss, the intra-modality triplet loss focuses more on intra-modality variation and complements the global triplet loss. In TI-ReID, although inter-modality variation can be greater than intra-modality variation, correctly identifying intra-modality variation is also an important part of testing the pedestrian re-identification algorithm. Different images of the same person in the same modality can vary significantly, and images of different people in the same modality can be very similar, which can affect the recognition performance of the model. Therefore, the intra-modality triplet loss is proposed to enhance the learning of intra-modality variation. Inter-modal hard sample triplet loss The definition is as follows: Wherein, when a≤K, E={1,2,···,K}; when a>K, E={k+1,k+2,···,2K}; represents samples of the same modality with the same category as the anchor point, Represents samples of the same modality that are different from the anchor category; Compared with the global triplet loss, the inter-modal triplet loss focuses more on inter-modal changes. In the TI-ReID problem, the model needs to pay more attention to inter-modal changes, because inter-modal changes cannot be distinguished by color or clothing style, but can only be distinguished by clothing attribute description and style description features. Therefore, this paper proposes the inter-modal triplet loss, which adds inter-modal change learning and uses it as a supplement to the global triplet loss. The intra-modal difficult sample triplet loss is defined as follows: Among them, when a≤K, E={k+1,k+2,···,2K}; when a>K, E={1,2,···,K}; image where a∈{1,2,···,K} represents the anchor image sample, and a∈{k+1,k+2,···,2K} represents the anchor sample is text; represents samples of the same modality with the same category as the anchor point, Represents cross-modal samples that are different from the anchor category; The joint loss of cross-modal difficult sample triplets is defined as follows: In order to enable the model to learn more discriminative image and text features, we set L CMPM , L CMPC and L CMHTF The combined loss constraint is used for supervised training. The experimental results in Table 2 show that the combined loss constraint greatly improves the cross-modal re-identification performance. The total loss is defined as follows: L=λ1L CMPM +λ2L CMPC +λ3L CMHTF (19) Among them, the hyperparameters λ1 is 1, λ2 is 1, and λ3 is 0.05.

Images