Text pedestrian search method based on self-supervised mask model and cross-modal codebook

Abstract

A text pedestrian search method based on a self-supervised mask model and a cross-modal codebook, comprising: first, covering the input text and picture, and then inputting into a feature extraction backbone network to obtain visual features F V and text features F T Then input the visual features F V and text features F T Into the mapping layer to obtain picture global features F V1 and text global features F T1 Then align them. At the same time, input the visual features F V and text features F T Into the cross-modal codebook, replace the visual features F V and text features F T With the closest features in the codebook, and then input the replaced features into the picture decoder and the text classification network respectively, and finally compare the results with the original input. The present application can not only improve the feature learning ability of the model, but also improve the alignment ability of the model for two modal features.

Description

TECHNICAL FIELD

[0001] The present application is based on the field of cross-modal retrieval, and particularly relates to a method using a feature alignment method based on a mask and a cross-modal codebook. BACKGROUND

[0002] Text-based pedestrian search aims to match text description queries with correct pedestrian images, which has great potential in surveillance systems, event analysis and intelligent albums. Compared with image query pedestrian re-identification (also known as image-based pedestrian re-identification), text description is easier to access in most cases, which makes text-based pedestrian search popular in recent years. Methods for solving cross-modal retrieval are mainly divided into two categories, one is to learn feature representation, and the other is to extract features of two modalities and then perform feature alignment.

[0003] In order to better learn suitable features from pictures and texts, there are methods of using generative adversarial networks and text descriptions to color (previously grayscale) human images; there are also methods of using self-supervised learning to obtain prior knowledge through CLIP, and then transferring to a cross-modal momentum contrast learning framework. At the same time, in order to solve the difference between the two modalities, there are also many works using attention mechanism to help achieve alignment between text and image features. This will require the use of a pre-trained object detection model or manually set the region to obtain the picture information of each position of the person in the picture, and then input the corresponding picture information and the corresponding text information into the attention module to achieve feature alignment. This will undoubtedly bring great computational pressure to training and testing. In order to better achieve the accuracy of cross-modal pedestrian retrieval, it is very important to solve the problem of feature alignment and feature learning between the two modalities. SUMMARY

[0004] In order to overcome the deficiencies of the prior art in cross-modal feature learning and alignment, the present application provides a method of combining a mask model and a cross-modal codebook to enhance the model's feature learning and alignment capabilities, and further improve the accuracy of cross-modal pedestrian retrieval.

[0005] In order to achieve the above purpose, the technical scheme of the text pedestrian search method based on the self-supervised mask model and the cross-modal codebook of the present application is as follows:

[0006] Step 1, read the data set, and input each pair of matched text description and picture as data of the model;

[0007] Step 1.1, first scale the picture to a predetermined size, and perform data enhancement such as horizontal flip, random addition of Gaussian noise, etc., then divide the picture into (h / p)*(w / p) square small blocks, p is the edge length of each small block, and h and w are the length and width dimensions of the picture respectively;

[0008] Step 1.2, then randomly select part of the picture block, and cover it with a unified mask token;

[0009] Step 1.3, input the text description into the word segmentation tool to convert words and phrases into corresponding numbers; at the same time, randomly select part of the text block and cover it with a unified mask token;

[0010] Step 2, input the processed picture and description text covered by the mask into the feature encoder of the two modalities; specifically including:

[0011] Step 2.1, the visual backbone network E V , and load the pre-trained model parameters on the dataset ImageNet to process the image input to obtain the visual feature F V ;

[0012] Step 2.2, the text backbone network E T , also load the pre-trained model parameters, process the text input to obtain the text feature F T ;

[0013] Step 2.3, then input the two features into the mapping layer respectively to obtain the global features of the two modalities;

[0014] Step 2.4, for the global features of the two modalities obtained, calculate the CMPC loss function and the CMPM loss function to measure the distance between the matched text pictures and the distance between the unmatched text pictures;

[0015] The CMPC loss function is expressed as follows:

[0016]

[0017]

[0018]

[0019] L cmpc =L tpi +L ipt (4) The CMPM loss function is expressed as follows:

[0020]

[0021]

[0022]

[0023]

[0024]

[0025] L cmpm = L i2t + L t2i (10)

[0026] where x i is visual feature, z i is text feature, W j is weight matrix, y i,j represents whether the input is a matched image-text pair, ∈ is a very small positive number to prevent division by zero;

[0027] Step 3, input the visual feature F V and the text feature F T obtained by the feature extraction backbone network in step 2 into the cross-modal codebook, the dimension size of the visual feature F V is (h / p)*(w / p)*D, the dimension size of the text feature F T is L*D, L is the length of the text, D is the channel number of the visual feature or the text feature, and the channel numbers of the visual feature and the text feature are the same; specifically including: further processing the features obtained in step 2, and the specific operation is implemented according to the following steps;

[0028] Step 3.1, the visual feature F V and the text feature F T have ((h / p)*(w / p)+L) feature vectors in total, the channel numbers of these feature vectors are the same as the channel numbers of the feature vectors in the codebook, then the distances between the feature vectors in the codebook and all text features and visual features are calculated, the visual feature F V and the text feature F T are replaced by the corresponding feature vectors with the closest distance in the codebook, and the searching method is as follows:

[0029]

[0030]

[0031] where z i represents the visual feature F V and the text feature F T , c i represents the feature vector in the codebook, and K represents the number of feature vectors in the codebook;

[0032] Step 3.2, after the feature vectors in the codebook are used to replace the original vectors in the visual feature F V and the text feature F T , the new visual feature F V2and new text feature F T2 ; Because the replaced feature vector is discrete, and the replacement process is not differentiable, gradient estimation straight-through is needed to backpropagate the gradient to the previous module, as shown in the following formula:

[0033]

[0034] Where sg(·) represents the stop gradient propagation, and l2 represents the normalization operation.

[0035] Step 3.3, after replacing the input feature vector, the momentum of the codebook feature is updated simultaneously, using the updated formula as follows:

[0036]

[0037] Where λ mom is the weight of updating the codebook, c h is the feature vector in the codebook.

[0038] Step 4, reconstruct the input picture and text:

[0039] Step 4.1, the image decoder uses a single-layer deconvolution network to restore the picture to the input size and channel number, then compares it with the original picture and calculates the reconstruction loss function.

[0040]

[0041] Step 4.2, the text selects the text encoder E T pre-trained text classifier (fine-tuned in the training stage), classifies the features through the linear layer Linear of the last layer of the text classifier, predicts the difference between the text and the input, and calculates the classification loss function.

[0042]

[0043] Where Ω T is a function that calculates how many tokens in x T , x T is the visual feature, and y T is the correct label of the text.

[0044] Step 5, according to the three loss functions in steps 2, 3, and 4, use the backpropagation algorithm and gradient descent algorithm to optimize the model; specifically including:

[0045] Step 5.1, according to the actual input and expected output, the overall error formula is obtained, which is:

[0046] L total= L align + λ1L recon + λ2L cosebook (17)

[0047] where L align is the CMPC and CMPM loss function that calculates the degree of alignment of the two modalities, L recon is the loss function that calculates the difference between the reconstructed input text and picture and the original text and picture that was not covered; L codebook is the difference between the replaced feature segments and the input feature segments in order to optimize the cross-modal codebook; λ1 and λ2 are the weights of the two loss functions L recon and L codebook in the entire loss function.

[0048] Step 5.2, use the back propagation algorithm and gradient descent algorithm to optimize the model parameters; set the batch size to 64, use the Adam optimizer, and set the initial learning rate of the network to 4*10 -5 , but in the first ten training rounds, linearly increase the learning rate, let the learning rate increase from 4*10 -6 to 4*10 -5 , and the learning rate decreases to 0.1 of the original at the 50th and 80th epoch, and a total of 120 rounds of training.

[0049] Step 6, when testing the model, select the features after the backbone network and the mapping layer, and use the features of the two modalities as the input and query set respectively, calculate the cosine similarity, then sort to get the corresponding query results.

[0050] Preferably, in step 1.1, the picture is first scaled to 384px*128px, and data augmentation such as horizontal flip and random addition of Gaussian noise is performed, and then the picture is divided into 192 small blocks, each with a size of 16px*16px.

[0051] Preferably, the proportion of the random selection of part of the picture blocks for covering in step 1.2 is 5%, 10%, 15%.

[0052] Preferably, the proportion of the random selection of part of the text blocks for covering in step 1.3 is 20%, 25%, 30%, 35%.

[0053] Preferably, the visual encoder E V in step 2.1 is a Resnet network or a Vision Transformer network.

[0054] Preferably, the text encoder E TBert network, lstm network, bi-lstm network.

[0055] Preferably, the number of feature vectors in the codebook in step 3.1 is 512, 1024, 2048.

[0056] Preferably, the weight λ of the codebook update in step 3.3 mom is set to 0.8.

[0057] Preferably, the loss function L in step 5.1 recon and the weight λ1 and λ2 of L codebook are both set to 0.2.

[0058] Preferably, step 5.2 specifically includes: setting the batch size to 64, using the optimizer Adam, setting the initial learning rate of the network to 4*10 -5 , linearly increasing the learning rate in the first ten training rounds, and letting the learning rate increase from 4*10 -6 to 4*10 -5 in the first ten training rounds.

[0059] The working principle of the present application is: the present application mainly improves the matching accuracy through two modules: first, the mask module is used to enable the model to extract features while obtaining the ability to reconstruct the original image and the original text, so that the learned feature vectors have corresponding high-level semantic information. Secondly, through the discrete cross-modal codebook, the features of the two modalities are found in the codebook at the same time, so as to realize the alignment of text features and picture features, and then realize the accuracy of cross-modal retrieval and matching on pedestrian data.

[0060] The simple mask strategy proposed by the present application can be applied to any network, and through the reconstruction of the original image and the original text, the learning ability of the features can be improved, so that the features extracted by the feature extraction network have rich semantic information. Secondly, compared with the complex feature alignment method used in other methods, the solution of the present application is to establish a discrete codebook for storing visual tokens, and through the discrete codebook, the features of the text and the picture are aligned in the semantic level, thereby helping to improve the ability of the model to align cross-modal features. Furthermore, compared with the complex multi-modal model (time complexity O(mn)), the present application uses a double-flow feature extraction network, which only needs to use a text feature extraction network and a picture feature extraction network to extract text and picture features respectively, and the time complexity is O(m+n), so the present application can improve the feature extraction ability and the feature alignment ability while achieving the fastest matching and retrieval effect under the premise of the same backbone network.

[0061] The advantages of the present application are: 1. The model can learn semantic information in the text and visual modalities, providing a theoretical basis for cross-modal alignment. 2. By establishing a cross-modal discrete codebook, the alignment of features of the two modalities is facilitated, which is simpler and easier to understand than using a complex attention mechanism to achieve modal alignment. 3. Using a simple dual-flow network, the time complexity is minimized, and for m pictures and n sentences, the time complexity is O(m+n). 4. On the dataset CUHK-PEDES, the accuracy rank@1 is improved by 1.98%, and the accuracy rank@10 is improved by 1.31%; on the dataset ICFG-PEDES, the accuracy rank@1 is improved by 1.78%, and the accuracy rank@10 is improved by 2.4%. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The overall flowchart of the cross-modal pedestrian retrieval method of the present application is shown in the figure. DETAILED DESCRIPTION

[0063] The present application will be further described below in conjunction with the drawings and implementation methods.

[0064] A cross-modal pedestrian retrieval method based on a mask model and a cross-modal codebook, as shown in the figure, is implemented according to the following steps: Figure 1

[0065] 1) Read the dataset, and input each pair of matched text description and picture as data input of the model.

[0066] 11) First, scale the picture to 384px*128px, and perform data enhancement such as horizontal flipping, random addition of Gaussian noise, etc.

[0067] 12) Then divide the picture into 192 small blocks according to the size of 16px*16px. Then randomly select 5%, 10%, and 15% of the small blocks to cover them with uniform mask tokens.

[0068] 13) Input the text description into a tokenizer to convert words and phrases into corresponding numbers. At the same time, randomly select 20%, 25%, 30%, and 35% of the numbers and cover them with uniform mask tokens.

[0069] 2) As shown in the figure, input the processed picture and description text covered by the mask into the feature encoders of the two modalities. Figure 1

[0070] ​​21) The visual feature extraction backbone network uses the visual encoder Vision Transformer-base, which is mainly composed of normalization Norm layer, multi-head self-attention mechanism layer, and fully connected layer MLP layer, and the obtained model parameters are loaded on the pre-trained data set ImageNet. The text feature extraction backbone network uses the text encoder Bert-base, which has a similar structure to the visual encoder Vision Transformer, and also loads the pre-trained model to initialize the model. When the text encoder is Resnet, it is mainly composed of convolutional layer CNN and residual block. When the text encoder is lstm or bi-lstm, it is mainly composed of recurrent neural network RNN.

[0071] 22) After the feature extraction network of the two modalities, the visual feature F V and the text feature F T are obtained. In order to align the global features of the two modalities, the visual feature F V and the text feature F T are input into two mapping layers. The visual mapping layer is only composed of a single linear layer Linear layer, and then the global maximum pooling layer is used to obtain the visual global feature F V1 . For the text feature, three stacked residual blocks are used. For each input feature in the residual block, on the one hand, it is input into three 1*1 convolution and ReLU activation function, and on the other hand, it is only input into a 1*1 convolution and ReLU activation function. Then the two inputs are added, and finally the obtained feature is input into the average pooling layer to obtain the text global feature F T1 .

[0072] 23) For the obtained global features of the two modalities, the CMPC loss function and the CMPM loss function are calculated to measure the distance relationship between the matched text pictures and the distance relationship between the unmatched text pictures. The CMPC loss function is expressed as follows:

[0073]

[0074]

[0075]

[0076] L cmpc = L tpi + L ipt (4)

[0077] The CMPM loss function is expressed as follows:

[0078]

[0079]

[0080]

[0081]

[0082]

[0083] L cmpm = L i2t + L t2i (10)

[0084] where x i is the visual feature, z i is the text feature, W j is the weight matrix, y i,j represents whether the input is a matched image-text pair, ∈ is a very small positive number to prevent division by zero.

[0085] 24) The overall loss function for measuring the difference between two modal features, which helps the modal pair, is represented as follows:

[0086] L align = L cmpc + L cmpm (18)

[0087] 3) The visual feature F V and the text feature F T obtained by the feature extraction backbone network in step 2 are input into the cross-modal codebook (the dimension size of the visual feature F V is 24*8*768, and the dimension size of the text feature F T is 100*768).

[0088] 31) The visual feature F V and the text feature F T have a total of (24*8+100) feature vectors with a channel number of 768, the channel number of the feature vectors in the codebook is also 768, and the codebook has 512, 1024, 2048 feature vectors. Then, the distances between the feature vectors in the codebook and all text features and visual features are calculated, and the visual feature F V and the text feature F T are replaced by the corresponding feature vectors with the closest distance found in the codebook. The search method is as follows:

[0089]

[0090]

[0091] where z i represents the visual feature F V and the text feature F T , c i represents the feature vector in the codebook, K represents the number of feature vectors in the codebook, and the value is selected as 512, 1024, 2048 in the present application.

[0092] 32) After the feature vectors in the codebook are used to replace the original vectors in the visual feature F V and the text feature F T , the new visual feature F V2 and the text feature F T2 are obtained. Because the replaced features are discrete and the replacement process is not differentiable, the gradient estimation straight-through is needed to backpropagate the gradient to the previous module, and the specific method is shown in the following formula:

[0093]

[0094] where sg(·) represents the stop gradient, and l2 represents the normalization operation.

[0095] 33) After the input feature vectors are replaced, the features in the codebook are updated synchronously, and the updated formula is as follows:

[0096]

[0097] where λ mom is the weight for updating the codebook, and c h is the feature vector in the codebook.

[0098] 4) In step 4, the visual feature F V2 and the text feature F T2 obtained through the cross-modal codebook are input into the decoder, and the self-supervised learning method is used to make the model reconstruct the input picture and text.

[0099] 41) In step 4.1, the visual feature F V2 is input into the visual decoder, and the visual encoder used in the present application is a single-layer deconvolution layer, which restores the length and width, the number of channels of the visual feature F V2 to be consistent with the input image. The model is guided to perform self-supervised learning through the reconstruction loss, and the difference between the picture recovered by the decoder and the input picture is calculated, and the loss function calculation method is as follows:

[0100]

[0101] 42) The text features are also input into the text classifier. The text classifier selected by the application is a text classifier pre-trained by the text encoder Bert. The input features are classified by the linear layer Linear of the last layer of the text classifier. The purpose is to calculate whether the model can correctly restore the words covered in the input stage. The loss function of the text mask model is calculated as follows:

[0102]

[0103] where Ω T is a function that calculates how many tokens in x T is the visual feature, y T is the correct text label; T

[0104] 5) According to the above three loss functions, we use the back propagation algorithm and the gradient descent algorithm to optimize the model.

[0105] 51) According to the actual input and the expected output, we get the overall error formula, which is:

[0106] L total = L align + λ1L recon + λ2L codebook (17)

[0107] where L align is the CMPC / M loss function that calculates the alignment degree of the two modalities, L recon is the loss function that calculates the difference between the reconstructed input text and picture and the original text and picture that was not covered at the beginning. L codebook is the difference between the replaced feature segment and the input feature segment in order to optimize the cross-modal codebook. λ1 and λ2 are the weights of L recon and L codebook in the entire loss function, both of which are set to 0.2 during training; in L codebook , λ mom is set to 0.8 to update the codebook.

[0108] 52) Use the back propagation algorithm and the gradient descent algorithm to optimize the model parameters. Set the batch size to 64, use the optimizer Adam, and set the initial learning rate of the network to 4*10 -5 In the first ten training rounds, the learning rate is linearly increased from 4*10 -6 to 4*10 -5 , and the learning rate is reduced to 10% of the original at the 50th and 80th epoch, for a total of 120 training rounds. ​

[0109] 6) When testing on the validation set, only the visual backbone network and the text backbone network are used to extract the input of the two modalities. For a given text description and a query set consisting of M images, the two backbone networks extract N text features and M image features, respectively.

[0110] 61) Then, the cosine similarity is used to calculate the similarity matrix between the N descriptions and M images, and the size of the matrix is N*M. Then, according to the similarity size, the top ten images with high matching rates corresponding to each text input are obtained. Then, the ten images and the true answer are compared to calculate the accuracy-related indicators.

[0111] 62) In order to verify the accuracy and effectiveness of the actual application of the method of the present application, the indicator CMC is used to represent the retrieval results, and the accuracy rank@1, rank@5, rank@10 values on the data set CUHK-PEDES and the data set ICFG-PEDES are calculated in detail to evaluate the algorithm performance and the experimental results are shown in Tables 1 and 2. According to the experimental results, the following conclusions can be drawn: (1) The accuracy is significantly improved. After using the mask-based model of the present application, there is a great improvement on both the data set CUHK-PEDES and the more challenging data set ICFG-PEDES. On the data set CUHK-PEDES, the accuracy rank@1 is improved by 1.98%, and the accuracy rank@10 is improved by 1.31%; on the data set ICFG-PEDES, the accuracy rank@1 is improved by 1.78%, and the accuracy rank@10 is improved by 2.4%. (2) While the model improves the retrieval effect, it can also reconstruct the input text and image, which shows that the model of the present application indeed improves the retrieval accuracy by improving the ability of feature extraction and alignment of the model.

[0112] Table 1: Retrieval results on the CUHK-PEDES data set

[0113]

[0114] Table 2: Retrieval results on the ICFG-PEDES data set

[0115]

Claims

1. A text-based person search method based on a self-supervised masking model and cross-modal codebook, characterized in that, The process involves partially obscuring image blocks and text blocks, and creating a cross-modal codebook, including the following steps: Step 1: Read the dataset and use the matching text description and image for each pair as the data input to the model; Step 1.1: First, scale the image to the predetermined size, then perform horizontal flipping and random Gaussian noise addition for data augmentation. Then, divide the image into (h / p)*(w / p) square blocks, where p is the side length of each block, and h and w are the length and width of the image, respectively. Step 1.2: Then, randomly select a portion of the image blocks and cover them using a uniform mask token; Step 1.3: Input the text description into the word segmenter to convert words and phrases into corresponding numbers; at the same time, randomly select a portion of text blocks and cover them with a uniform mask token; Step 2 involves inputting the processed masked image and descriptive text into the two-modal feature encoders; specifically, this includes: Step 2.1, Visual Encoder E V It loads the model parameters pre-trained on the ImageNet dataset and processes the image input to obtain the visual features F. V ; Step 2.2, Text Backbone Network E T Similarly, by loading the pre-trained model parameters and processing the text input, the text features F are obtained. T ; Step 2.3: Then input the two features into the mapping layer to obtain the global features of the two modalities; Step 2.4: For the global features of the two obtained modalities, calculate the CMPC loss function and the CMPM loss function to measure the distance between matching text images and the distance between non-matching text images; The CMPC loss function is expressed as follows: L cmpc L tpi +L ipt (4) The CMPM loss function is expressed as follows: L cmpm L i2t +L t2i (10) Where x i For visual features, z i For text features, W j Let y be the weight matrix. i,j Indicates whether the input is a matching image-text pair; ∈ is a very small positive number to prevent division by zero; Step 3, extract the visual features F obtained from the feature extraction backbone network in Step 2. V and text features F T Input into the cross-modal codebook, visual feature F V The dimension size is (h / p)*(w / p)*D, and the text feature F T The dimension is L*D, where L is the length of the text and D is the number of channels for visual features or text features. The number of channels for visual features and text features is the same. Specifically, it includes further processing of the features obtained in step 2. The specific operations are carried out according to the following steps. Step 3.1, Visual Feature F V and text features F T There are a total of ((h / p)*(w / p)+L) feature vectors. The number of channels in these feature vectors is the same as the number of channels in the feature vectors in the codebook. Then, the distances between the feature vectors in the codebook and all text features and visual features are calculated, and the visual feature F is... V and text features F T Each feature vector is replaced by the closest corresponding feature vector found in the codebook. The search method is as follows: Where z i Representing visual feature F V and text features F T c i,k K represents the feature vectors in the codebook, and K represents the number of feature vectors in the codebook. Step 3.2: Replace the original visual feature F with the feature vector from the codebook. V and text features F T After processing the vectors in the vector, we obtain the new visual feature F. V2 and new text features F T2 Because the replaced feature vectors are discrete and the replacement process is non-differentiable, gradient estimation straight-through is needed to backpropagate the gradients to the preceding modules. The specific method is shown in the following formula: Where sg(·) represents stopping gradient propagation and l2 represents normalization operation; Step 3.3: After replacing the input feature vector, the features in the codebook need to be updated synchronously using momentum. The update formula is as follows: Where λ mom It is the weight of updating the codebook, c h These are the feature vectors in the codebook; Step 4, reconstruct the input image and text: Step 4.1: The image decoder uses a single-layer deconvolutional network to restore the image to the input size and number of channels, then compares it with the original image and calculates the reconstruction loss function; Step 4.2, the text selected is text encoded by E. T A pre-trained text classifier classifies features by passing them through the last linear layer of the text classifier, predicts the difference between the text and the input, and calculates the classification loss function. Where Ω T To calculate x T The function that determines how many tokens are in x T For visual features, y T Correctly label the text; Step 5: Based on the three loss functions from Steps 2, 3, and 4, optimize the model using the backpropagation algorithm and the gradient descent algorithm; specifically including: Step 5.1: Based on the actual input and the expected output, obtain the overall error formula, which is: L total =L align +λ1L recon +λ2L codebook (17) In the formula L align These are the CMPC and CMPM loss functions used to calculate the degree of alignment between two modes, L. recon It is the loss function that calculates the difference between the reconstructed input text and images and the original, unmasked text and images; L codebook To optimize cross-modal codebooks, the difference between the replaced feature fragment and the input feature fragment is calculated; λ1 and λ2 are two loss functions L recon and L codebook The weight it occupies in the entire loss function; Step 5.2: Optimize model parameters using backpropagation and gradient descent algorithms; Step 6: When testing the model, the features after passing through the backbone network and the mapping layer are selected. The features of the two modalities are used as the input and query sets, respectively. The cosine similarity is calculated and then sorted to obtain the corresponding query results.

2. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: In step 1.1, the image is first scaled to 384px*128px, and data augmentation is performed by horizontal flipping and randomly adding Gaussian noise. Then, the image is divided into 192 small blocks, each with a size of 16px*16px.

3. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: The proportion of randomly selected image blocks to be covered in step 1.2 is 5%, 10%, and 15%.

4. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: The proportion of randomly selected text blocks to be covered in step 1.3 is 20%, 25%, 30%, and 35%.

5. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: The visual encoder E described in step 2.1 V For ResNet networks and Vision Transformer networks.

6. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: The text encoder E described in step 2.2 T For Bert networks, LSTM networks, and bi-LSTM networks.

7. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: The number of eigenvectors in the codebook described in step 3.1 is 512, 1024, and 2048.

8. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: The loss function L described in step 5.1 recon and L codebook The weights λ1 and λ2 are both set to 0.

2.

9. The text-based person search method based on a self-supervised mask model and cross-modal codebook as described in claim 1, characterized in that: Step 5.2 specifically includes: setting the batch size to 64, using the Adam optimizer, and setting the initial learning rate of the network to 4*10. -5 In the first ten training rounds, the learning rate is increased linearly, starting from 4*10. -6 First grow to 4*10 -5 The learning rate drops to 10% of its original value in the 50th and 80th epochs, for a total of 120 training epochs.

Images