A cross-modal pedestrian search implementation method based on large-scale pre-trained models

Abstract

The present invention discloses a method for implementing cross-modal pedestrian search based on a large-scale pre-trained model, which relates to the fields of computer vision, natural language processing and machine learning technology. First, the images in a certain cross-modal pedestrian search dataset are divided into a training dataset and a verification dataset in proportion, and the pedestrian image attributes in the training dataset and the verification dataset are filled in according to a custom sentence template to obtain two corresponding texts. The attributes in the text data of the training dataset are randomly masked to obtain masked text. Then, a cross-modal pedestrian search algorithm is constructed based on the large-scale pre-trained model, and the images, texts and masked texts of the training dataset are imported into the cross-modal pedestrian search algorithm, and a cross-modal pedestrian search algorithm model is generated through training iterations. Finally, the search accuracy of the trained cross-modal pedestrian search algorithm model is tested using a test data set.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, natural language processing and machine learning technologies, and in particular to a method for implementing cross-modal pedestrian search based on a large-scale pre-trained model. Background Art

[0002] With the continued expansion of the video surveillance market and the advancement of deep learning technology, video surveillance is becoming increasingly intelligent. Pedestrian search within surveillance scenarios plays a crucial role in many applications, including smart security and smart cities. Pedestrian search involves searching for a specific pedestrian from multiple non-overlapping cameras, given relevant information about that pedestrian. Existing pedestrian search methods require an image of the pedestrian, but these methods are ineffective when only a verbal description from an eyewitness or insider is available.

[0003] Attribute-based cross-modal pedestrian search can quickly search for specified pedestrian images through pedestrian attributes (such as age, gender, clothing, etc.), and has great application potential in the field of intelligent security. Specifically, this search method is to input the query attributes, vectorize them, calculate the cross-modal similarity with the image vectors in the image database, and sort them to achieve image search. Since cross-modal pedestrian search is achieved through deep learning technology, it greatly solves the heavy labor of manual search and effectively improves the intelligent processing capabilities of massive surveillance videos. At the same time, this search method broadens the modal restrictions of query input, solves the limitation of only using image input in pedestrian retrieval, and meets the diverse application needs of users.

[0004] Existing research on cross-modal pedestrian search can be divided into two categories. The first type of method uses attribute recognition to identify pedestrian images, obtains the corresponding attributes of the image, and saves them in a database. When applied, the image is searched by comparing the similarity between the attributes of the given query input and the attributes in the database. However, this type of method is limited by the low attribute recognition accuracy, resulting in low search accuracy. The second type of method represents the attributes and images as single feature vectors through feature extraction and aligns them in a common embedding space. When applied, the cosine similarity between the attribute and image vectors is directly calculated to search the image. However, this type of method only uses the prior knowledge of single-modal pre-training and lacks multi-modal corresponding information. Due to the semantic gap between attribute and image modalities, the performance of this type of method is limited. Summary of the Invention

[0005] In view of the fact that existing cross-modal pedestrian search methods lack multimodal corresponding information, resulting in a semantic gap between attribute and image modalities, which limits the accuracy of the search, the present invention introduces the prior knowledge of a large-scale multimodal pre-trained model to bridge the semantic gap between modalities and provides a cross-modal pedestrian search method based on a large-scale pre-trained model, so as to realize a cross-modal pedestrian search technology that combines multimodal corresponding information and achieves high accuracy.

[0006] A method for implementing cross-modal pedestrian search based on a large-scale pre-trained model includes the following steps:

[0007] Step 1: Based on a cross-modal pedestrian search dataset, the images are divided into a training dataset and a validation dataset in proportion, and paired annotation information of pedestrian categories and pedestrian image attributes is obtained respectively.

[0008] Step 2: Fill the pedestrian image attributes in the training dataset and the verification dataset according to the customized sentence template to obtain the corresponding two texts.

[0009] Using a custom sentence template P, fill the pedestrian image attribute A into the corresponding position of the sentence template and convert the attribute A into text T.

[0010] Step 3: Randomly mask the attributes in the text data of the training dataset to obtain masked text

[0011] The random masking process is:

[0012] Use the tokenizer to segment the text T, randomly select attribute words in the text with a probability of 15% for masking, and change the attribute words to be masked to "<|mask|>", and finally get the masked text

[0013] Step 4: Build a cross-modal pedestrian search algorithm based on a large-scale pre-trained model, import the images, texts, and masked texts of the training dataset into the cross-modal pedestrian search algorithm, and generate a cross-modal pedestrian search algorithm model through training iterations.

[0014] The training iteration process of the cross-modal pedestrian search algorithm model is:

[0015] Step 401: using a large-scale pre-trained model to extract the global features and the final hidden state features of the image, the global features of the text, and the final hidden state features of the masked text;

[0016] Specifically:

[0017] Use the image encoder of the large-scale pre-trained model to extract the final hidden state features of image I and global features

[0018] Extract global features of text T using a text encoder based on a large-scale pre-trained model Masked Text The final hidden state features

[0019] Step 402 : Calculate a loss function based on the features extracted from the image, text, and masked text, and train the model based on the loss function.

[0020] The loss function is the sum of the image-text pair matching loss function, the text-text pair matching loss function, the mask attribute prediction loss function, and the image and text classification loss function. The specific calculation process is:

[0021] ①Image-text pair matching loss function

[0022] First, the cosine similarity is calculated using the global features of the image and text And further calculate the probability p of matching between image and text pairs i,j ;

[0023] Cosine similarity

[0024]

[0025] Matching probability p i,j :

[0026]

[0027] Where τ represents the temperature coefficient, which is used to control the peak value of the probability distribution;

[0028] Then, the image-text pair matching loss function is calculated based on the matching probability between the image and text pair and the true matching probability;

[0029] The image-text matching loss function is:

[0030]

[0031] The text-image matching loss function is:

[0032]

[0033] Among them, ε is a decimal that keeps the denominator numerically stable, and q i,j Represents the true matching probability of the image text pair, p j,i Represents the matching probability of text-image;

[0034] ②Text and text pair matching loss function

[0035] First, the cosine similarity between the text and the global features of the text is used to calculate the probability of matching between the text and the text pair.

[0036]

[0037] in, Represents the cosine similarity between text and global features of text;

[0038] Then, the intersection over union (IoU) between attributes is calculated based on the number of identical words between the attributes. i,j , and the corresponding true attribute matching probability

[0039]

[0040] Among them, A i and A j are the i-th and j-th attribute sets respectively.

[0041] Finally, the text and text pair matching loss function is calculated based on the matching probability between the text and the text pair and the corresponding true attribute matching probability.

[0042]

[0043] ③Mask attribute prediction loss function

[0044] First, mask the text The final hidden state features As query (q), the last hidden state feature of image I As key (k) and value (v), input the predefined multimodal encoder MultimodalEncoder (ME) to obtain the final fusion feature for attribute prediction

[0045]

[0046] Then, the fusion features Through the multi-layer perceptron MLP, the probability distribution of the corresponding attribute prediction results is obtained Where |Υ| represents the size of the vocabulary;

[0047] Finally, construct a loss function with mask attribute prediction as the target

[0048]

[0049] Among them, yi Represents the true vocabulary distribution, M represents the set of masked attribute words, and this loss function can implicitly learn the connection between the fine-grained features of the two modalities of image and attribute.

[0050] ④Image and text classification loss function

[0051] The global features of the image and global features of text The image category probability is obtained by classifier respectively and text category probability and the true label y i Compare and construct loss functions with image classification and text classification as optimization targets

[0052]

[0053] ⑤ The final loss function formula obtained by adding the above loss functions is:

[0054]

[0055] In step 403, the accuracy of the results of the trained cross-modal pedestrian search algorithm model is determined using a verification data set containing text to determine whether the model has performance gain. If not, the model training is terminated and the corresponding model file is saved; otherwise, return to step 3 and perform random masking again.

[0056] Step 5: Read the test data set one by one, load the trained cross-modal pedestrian search algorithm model file, test the search accuracy, and obtain the search accuracy result.

[0057] The advantages and beneficial effects of this application are:

[0058] (1) The implementation method of cross-modal pedestrian search based on a large-scale pre-trained model proposed in the present invention bridges the semantic gap between modalities and improves the accuracy of cross-modal pedestrian search by introducing the rich multimodal knowledge of a large-scale multimodal pre-trained model and enabling the multimodal pre-trained model to better learn the local features of different modalities and their corresponding relationships through the mask attribute prediction task.

[0059] (2) This paper proposes a new matching loss function for text and text pairs. This loss aligns the feature distribution of text pairs in the embedding space with the distribution of the intersection-over-union ratio of their original attributes. By training the matching loss function between the image and text pairs, texts corresponding to different attributes can be made more discriminative in the embedding space, improving the model's ability to discriminate different attributes, thereby improving the accuracy of the final cross-modal pedestrian search. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a flowchart of a method for implementing cross-modal pedestrian search based on a large-scale pre-trained model in the present invention;

[0061] Figure 2 This is a training flowchart of a cross-modal pedestrian search algorithm model based on a large-scale pre-training model in the present invention. DETAILED DESCRIPTION

[0062] In order to more clearly understand the technical principles of the present invention, the embodiments of the present invention are specifically and in detail described below with reference to the accompanying drawings.

[0063] In this embodiment, the cross-modal pedestrian search dataset Market-1501 is taken as an example. Figure 1 As shown, the specific implementation steps are as follows:

[0064] Step S0, obtaining a training data set T, a verification data set V, and a test data set U containing paired annotation information of image attributes;

[0065] Specifically, in step S0, the Market-1501 dataset contains a total of 29,419 images, 992 types of pedestrians, and 27 attributes describing pedestrian images. The training data set T contains 12,936 images, 508 types of pedestrians, and the verification data set V contains 16,483 images, 484 types of pedestrians.

[0066] Step S1: Fill the attributes A in the training data set T, the verification data set V, and the test data set U according to a custom sentence template and convert them into text T.

[0067] For example, if the attribute is {woman, teenager, front, short sleeve, bag, shorts}, after filling in according to the template, the corresponding text is "This teenager is a woman and in front. She is carrying a shoulder bag. Her upper clothing has short sleeves. She is wearing shorts."

[0068] Step S2, randomly masking the attributes in the text data in the training data set to obtain masked text;

[0069] Specifically, the text T is segmented using a tokenizer, and attribute words in the text are randomly selected with a probability of 15% for masking. The attribute words to be masked are changed to "<|mask|>", and finally the masked text is obtained.

[0070] Step S3: construct a cross-modal pedestrian search algorithm, set and adjust the algorithm parameter set, import the image, text, and mask text in the above steps into the cross-modal pedestrian search algorithm, iteratively generate and save the cross-modal pedestrian search algorithm model;

[0071] Preferably, if Figure 2 As shown, step S3 further includes:

[0072] Step S300: Use the image encoder of the pre-trained model to extract the final hidden state features of image I Global Features Use the text encoder of the pre-trained model to extract the global features of the text T Masked Text The final hidden state features

[0073] Step S301: Calculate cosine similarity using global features of image and text

[0074]

[0075] Step S302: Calculate the matching probability p between the image and text pair using the cosine similarity between global features. i,j :

[0076]

[0077] Where τ represents the temperature coefficient, which is used to control the peak value of the probability distribution and has a value of 0.02;

[0078] Step S303, constructing a loss function for matching the matching probability between the image-text pair and the true matching probability;

[0079] The image-text matching loss function is:

[0080]

[0081] The text-image matching loss function is:

[0082]

[0083] Among them, ε is a decimal that keeps the denominator numerically stable, and q i,jRepresents the true matching probability of image-text pairs. This loss function can align the matching probability distribution between image and text pairs with the true matching probability distribution, enabling the model to effectively increase the correlation between matching pairs.

[0084] Step S304: Calculate the probability of matching between text and text pairs using the cosine similarity between the text and the text global feature.

[0085]

[0086] Step S305: In order to measure the difference between attributes, the intersection over union (IoU) between attributes is calculated based on the number of identical words between the attributes. i,j , and the corresponding true attribute matching probability

[0087]

[0088] Step S306: Construct a loss function to match the matching probability between text and text pairs with the matching probability of their corresponding true attributes.

[0089]

[0090] Among them, ε is a decimal that keeps the denominator numerically stable. Through this loss function, texts corresponding to attributes with different differences can be made more distinguishable in the embedding space, thereby improving the model's ability to identify different attributes.

[0091] Step S307: Mask the text The final hidden state features As query (q), the last hidden state feature of image I As key (k) and value (v), input the predefined multimodal encoder MultimodalEncoder (ME) to obtain the final fusion feature for attribute prediction

[0092]

[0093] Step S308: Fusion features Through the multi-layer perceptron MLP, the probability distribution of the corresponding attribute prediction results is obtained Where |Υ| represents the size of the vocabulary, which is 49408;

[0094] Step S309: Construct a loss function with mask attribute prediction as the goal

[0095]

[0096] Among them, y i Represents the true vocabulary distribution, M represents the set of masked attribute words, and this loss function can implicitly learn the connection between the fine-grained features of the two modalities of image and attribute.

[0097] Step S310: Construct a loss function with image classification and text classification as optimization targets The global features of the image and global features of text The image category probability is obtained by classifier respectively and text category probability and the true label y i For comparison:

[0098]

[0099] Step S311: Match the image-text pair Text and text pair matching loss Mask attribute prediction loss Image and text classification losses Add up to get the final loss function for model training:

[0100]

[0101] Preferably, in step S3, the verification data set containing text organized in step S1 is read to determine the accuracy of the results of the cross-modal pedestrian search algorithm. If there is no performance gain, the model training is terminated and the corresponding model file is saved, otherwise the process returns to execute S2.

[0102] Preferably, after step S3, the following steps are further included:

[0103] In step S4, the test data set compiled in step S1 is read one by one in sequence, the cross-modal pedestrian search model file saved in step S3 is loaded, and the search accuracy is tested to obtain the search accuracy result.

[0104] The proposed method is applied to cross-modal pedestrian search. Compared with previous algorithms, on the Market-1501Attribute dataset, the Rank-1 accuracy is improved by 13.1%, and the mAP is improved by 31.0%; on the PETA dataset, the Rank-1 accuracy is improved by 38.3%, and the mAP is improved by 44.9%; on the PA100K dataset, the Rank-1 accuracy is improved by 27.6%, and the mAP is improved by 53.8%.

Claims

1. A method for implementing cross-modal pedestrian search based on a large-scale pre-trained model, characterized in that: The steps include: Step 1: Based on a cross-modal pedestrian search dataset, the images are divided into a training dataset and a validation dataset in proportion, and paired annotation information of pedestrian categories and pedestrian image attributes is obtained respectively; Step 2: Fill the attributes of pedestrian images in the training dataset and the validation dataset according to the customized sentence template to obtain the corresponding two texts; Using a custom sentence template P, fill the pedestrian image attribute A into the corresponding position of the sentence template and convert the attribute A into text T; Step 3: Randomly mask the attributes in the text data of the training dataset to obtain masked text Step 4: Build a cross-modal pedestrian search algorithm based on a large-scale pre-trained model. Import the images, text, and masked text of the training dataset into the cross-modal pedestrian search algorithm, and generate a cross-modal pedestrian search algorithm model through training iterations. The training iteration process of the cross-modal pedestrian search algorithm model is: Step 401: using a large-scale pre-trained model to extract the global features and the final hidden state features of the image, the global features of the text, and the final hidden state features of the masked text; Specifically: Use the image encoder of the large-scale pre-trained model to extract the final hidden state features of image I and global features Extract global features of text T using a text encoder based on a large-scale pre-trained model Masked Text The final hidden state features Step 402: Calculate a loss function based on the features extracted from the image, text, and masked text, and train the model based on the loss function; The loss function is the sum of the image-text pair matching loss function, the text-text pair matching loss function, the mask attribute prediction loss function, and the image and text classification loss function. The formula is: Among them, the image-text matching loss function is: The text-image matching loss function is: Text and text pair matching loss function Mask attribute prediction loss function Image and text classification loss functions p i,j is the image-text matching probability, ε is a decimal that keeps the denominator numerically stable, and q i,j Represents the true matching probability of the image text pair, p j,i Represents the matching probability of text-image; represents the probability of matching between text and text pairs, The probability of true attribute matching between text and text pairs; |Υ| represents the size of the vocabulary, y i represents the true vocabulary distribution, M represents the set of masked attribute words; y i is the real label, is the image category probability, is the text category probability; Step 403: Use the verification data set containing text to determine the accuracy of the trained cross-modal pedestrian search algorithm model to determine whether there is a performance gain. If not, the model training ends and the corresponding model file is saved. Otherwise, return to step 3 and perform random masking again. Step 5: Read the test data set one by one, load the trained cross-modal pedestrian search algorithm model file, test the search accuracy, and obtain the search accuracy result.

2. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1, characterized in that: The random masking process is as follows: Use the word segmenter to segment the text T, randomly select attribute words in the text with a probability of 15% for masking, and change the attribute words to be masked to "<|mask|>", and finally get the masked text 3. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1 is characterized in that: The calculation process of the image-text pair matching loss function is: First, the cosine similarity is calculated using the global features of the image and text And further calculate the probability p of matching between image and text pairs i,j ; Cosine similarity Matching probability p i,j : Where τ represents the temperature coefficient, which is used to control the peak value of the probability distribution; Then, the image-text pair matching loss function is calculated based on the matching probability between the image and text pair and the true matching probability; The image-text matching loss function is: The text-image matching loss function is: Among them, ε is a decimal that keeps the denominator numerically stable, and q i,j Represents the true matching probability of the image text pair, p j,i Represents the text-image matching probability.

4. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1 is characterized in that: The calculation process of the text and text pair matching loss function is: First, the cosine similarity between the text and the global features of the text is used to calculate the probability of matching between the text and the text pair. in, Represents the cosine similarity between text and global features of text; Then, the intersection over union (IoU) between attributes is calculated based on the number of identical words between the attributes. i,j , and the corresponding true attribute matching probability Among them, A i and A j are the i-th and j-th attribute sets respectively; Finally, the text and text pair matching loss function is calculated based on the matching probability between the text and the text pair and the corresponding true attribute matching probability.

5. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1 is characterized in that: The calculation process of the mask attribute prediction loss function is: First, mask the text The final hidden state features As query (q), the last hidden state feature of image I As key (k) and value (v), input the predefined multimodal encoder Multimodal Encoder (ME) to obtain the final fusion feature for attribute prediction Then, the fusion features Through the multi-layer perceptron MLP, the probability distribution of the corresponding attribute prediction results is obtained Where |Υ| represents the size of the vocabulary; Finally, construct a loss function with mask attribute prediction as the target Among them, y i Represents the true vocabulary distribution, M represents the set of masked attribute words, and this loss function can implicitly learn the connection between the fine-grained features of the two modalities of image and attribute.

6. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1, characterized in that: The calculation process of the image and text classification loss function is: The global features of the image and global features of text The image category probability is obtained by classifier respectively and text category probability and the true label y i Compare and construct loss functions with image classification and text classification as optimization targets

Images