Method for realizing cross-modal pedestrian search based on large-scale pre-training model

Abstract

The invention discloses a cross-modal pedestrian search implementation method based on a large-scale pre-training model, and relates to the technical field of computer vision, natural language processing and machine learning. The method comprises the following steps: firstly, dividing images in a certain cross-modal pedestrian search data set into a training data set and a verification data set in proportion, and filling pedestrian image attributes in the training data set and the verification data set according to self-defined sentence templates to obtain two corresponding texts; and randomly masking attributes in the text data of the training data set to obtain a masked text. And then constructing a cross-modal pedestrian search algorithm based on the large-scale pre-training model, importing the image, the text and the mask text of the training data set into the cross-modal pedestrian search algorithm, and generating a cross-modal pedestrian search algorithm model through training iteration. And finally, testing the search accuracy of the trained cross-modal pedestrian search algorithm model through the test data set.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, natural language processing and machine learning, and in particular to a method for implementing cross-modal pedestrian search based on a large-scale pre-training model. Background Art

[0002] With the continuous expansion of the video surveillance market and the continuous development of deep learning technology, video surveillance has gradually become intelligent. Pedestrian search in surveillance scenarios plays an important role in many scenarios such as smart security and smart cities. Pedestrian search is to search for a specified pedestrian from multiple non-overlapping cameras given the relevant information of a pedestrian. Existing pedestrian search methods require an image of a pedestrian, but when the image of the target pedestrian cannot be obtained, such methods cannot be used when there is only a verbal description from an eyewitness or an insider.

[0003] Attribute-based cross-modal pedestrian search can quickly search for specified pedestrian images through pedestrian attributes (such as age, gender, clothing, etc.), and has great application potential in the field of intelligent security. Specifically, this search method inputs the query attributes, vectorizes them, calculates the cross-modal similarity with the image vectors in the image database and sorts them to achieve image search. Since cross-modal pedestrian search is achieved through deep learning technology, it greatly solves the heavy labor of manual search and effectively improves the intelligent processing capabilities of massive surveillance videos. At the same time, this search method broadens the modal restrictions of query input, solves the limitation that only image input can be used in pedestrian retrieval, and meets the diverse application needs of users.

[0004] Existing research on cross-modal pedestrian search can be divided into two categories. The first method uses attribute recognition to obtain the attributes corresponding to the image and saves them in a database. When applied, the image is searched by comparing the similarity between the attributes of a given query input and the attributes in the database. However, this type of method is limited by the low attribute recognition accuracy, resulting in low accuracy during search. The second type of method represents the attributes and images as a single feature vector through feature extraction and aligns them in a common embedding space. When applied, the cosine similarity between the attribute and image vectors is directly calculated to search the image. However, this type of method only uses the prior knowledge of single-modal pre-training and lacks multi-modal corresponding information. Due to the semantic gap between the attribute and image modalities, the performance of this type of method is limited. Summary of the invention

[0005] In view of the fact that existing cross-modal pedestrian search methods lack multimodal corresponding information, resulting in a semantic gap between attribute and image modalities, which limits the accuracy of the search, the present invention introduces prior knowledge of a large-scale multimodal pre-trained model to bridge the semantic gap between modalities and provides a cross-modal pedestrian search method based on a large-scale pre-trained model, so as to realize a cross-modal pedestrian search technology that combines multimodal corresponding information and achieves high accuracy.

[0006] A method for implementing cross-modal pedestrian search based on a large-scale pre-trained model includes the following steps:

[0007] Step 1: Based on a cross-modal pedestrian search dataset, the images are divided into a training dataset and a validation dataset in proportion, and paired annotation information of pedestrian categories and pedestrian image attributes are obtained respectively.

[0008] Step 2: Fill the attributes of pedestrian images in the training dataset and the validation dataset according to the customized sentence templates to obtain two corresponding texts.

[0009] Using a custom sentence template P, fill the pedestrian image attribute A into the corresponding position of the sentence template and convert the attribute A into text T.

[0010] Step 3: Randomly mask the attributes in the text data of the training dataset to obtain masked text

[0011] The random masking process is:

[0012] Use the tokenizer to tokenize the text T, randomly select attribute words in the text with a probability of 15% for masking, and change the attribute words to be masked to "<|mask|>", and finally get the masked text

[0013] Step 4: Build a cross-modal pedestrian search algorithm based on a large-scale pre-trained model, import the images, texts, and masked texts of the training dataset into the cross-modal pedestrian search algorithm, and generate a cross-modal pedestrian search algorithm model through training iterations.

[0014] The training iteration process of the cross-modal pedestrian search algorithm model is:

[0015] Step 401, using a large-scale pre-trained model to extract global features and final hidden state features of the image, global features of the text, and final hidden state features of the masked text;

[0016] Specifically:

[0017] The image encoder of the large-scale pre-trained model is used to extract the final hidden state features of image I. and global features

[0018] Extract global features of text T using a text encoder based on a large-scale pre-trained model Mask Text The final hidden state features

[0019] Step 402, calculating a loss function based on features extracted from the image, text, and masked text, and training the model based on the loss function.

[0020] The loss function is the sum of the image-text pair matching loss function, the text-text pair matching loss function, the mask attribute prediction loss function, and the image-text classification loss function. The specific calculation process is:

[0021] ① Image-text pair matching loss function

[0022] First, the cosine similarity is calculated using the global features of the image and text And further calculate the probability p of matching between image and text pairs i,j ;

[0023] Cosine similarity

[0024]

[0025] Matching probability p i,j :

[0026]

[0027] Among them, τ represents the temperature coefficient, which is used to control the peak value of the probability distribution;

[0028] Then, the image-text pair matching loss function is calculated based on the matching probability between the image and text pair and the true matching probability;

[0029] The image-text matching loss function is:

[0030]

[0031] The text-image matching loss function is:

[0032]

[0033] Here, ε is a decimal that keeps the denominator numerically stable, and q i,j represents the true matching probability of the image text pair, p j,i Represents the matching probability of text-image;

[0034] ②Text and text pair matching loss function

[0035] First, the cosine similarity between the text and the global features of the text is used to calculate the probability of matching between the text and the text pair.

[0036]

[0037] in, Represents the cosine similarity between text and global features of the text;

[0038] Then, the intersection over union (IoU) between attributes is calculated based on the number of identical words between attributes. i,j , and the corresponding true attribute matching probability

[0039]

[0040] Among them, A i and A j are the i-th and j-th attribute sets respectively.

[0041] Finally, the text and text pair matching loss function is calculated based on the matching probability between the text and the text pair and the matching probability of their corresponding true attributes.

[0042]

[0043] ③Mask attribute prediction loss function

[0044] First, mask the text The final hidden state features As query (q), the last hidden state feature of image I As key (k) and value (v), input the predefined multimodal encoder MultimodalEncoder (ME) to obtain the final fusion feature for attribute prediction

[0045]

[0046] Then, the fusion features Through the multi-layer perceptron MLP, the probability distribution of the corresponding attribute prediction results is obtained Where |Υ| represents the size of the vocabulary;

[0047] Finally, construct a loss function with mask attribute prediction as the target

[0048]

[0049] Among them, yi Represents the true vocabulary distribution, M represents the set of masked attribute words, and this loss function can implicitly learn the connection between the fine-grained features of the two modalities of image and attribute.

[0050] ④Image and text classification loss function

[0051] The global features of the image and global features of the text The image category probabilities are obtained through classifiers respectively and text category probability and the true label y i Compare and construct loss functions with image classification and text classification as optimization objectives

[0052]

[0053] ⑤The final loss function formula obtained by adding the above loss functions is:

[0054]

[0055] Step 403, using a verification data set containing text to determine the accuracy of the results of the trained cross-modal pedestrian search algorithm model, to determine whether the model has performance gain. If not, the model training is terminated and the corresponding model file is saved; otherwise, return to step three and perform random masking again.

[0056] Step 5: Read the test data set one by one, load the trained cross-modal pedestrian search algorithm model file, test the search accuracy, and obtain the search accuracy result.

[0057] The advantages and beneficial effects of this application are:

[0058] (1) The implementation method of cross-modal pedestrian search based on a large-scale pre-trained model proposed in the present invention introduces the rich multimodal knowledge of the large-scale multimodal pre-trained model and enables the multimodal pre-trained model to better learn the local features of different modalities and their corresponding relationships through the mask attribute prediction task, thereby bridging the semantic gap between modalities and improving the accuracy of cross-modal pedestrian search.

[0059] (2) The present invention proposes a new matching loss function for text and text pairs, which aligns the feature distribution of text pairs in the embedding space with the distribution of the intersection and union of their original attributes. By training the matching loss function between the joint image and text pairs, the texts corresponding to different attributes can be made more distinguishable in the embedding space, improving the model's discriminability for different attributes, thereby improving the accuracy of the final cross-modal pedestrian search. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A flowchart of a method for implementing cross-modal pedestrian search based on a large-scale pre-trained model of the present invention;

[0061] Figure 2 This is a training flowchart of a cross-modal pedestrian search algorithm model based on a large-scale pre-training model of the present invention. DETAILED DESCRIPTION

[0062] In order to more clearly understand the technical principle of the present invention, the embodiments of the present invention are specifically and in detail described below with reference to the accompanying drawings.

[0063] In this embodiment, the cross-modal pedestrian search dataset Market-1501 is taken as an example. Figure 1 As shown, the specific implementation steps are as follows:

[0064] Step S0, obtaining a training data set T, a verification data set V and a test data set U containing paired annotation information of image attributes;

[0065] Specifically, in step S0, the Market-1501 dataset contains a total of 29,419 images, 992 types of pedestrians, and 27 attributes describing pedestrian images. The training data set T contains 12,936 images, 508 types of pedestrians, and the verification data set V contains 16,483 images, 484 types of pedestrians.

[0066] Step S1, filling the attribute A in the training data set T, the verification data set V and the test data set U according to the custom sentence template, and converting them into text T.

[0067] For example, if the attribute is {woman, teenager, front, short sleeve, bag, shorts}, after filling in according to the template, the corresponding text is "This teenager is a woman and in front. She is carrying a shoulderbag. Her upper clothing has short sleeves. She is wearing shorts.".

[0068] Step S2, randomly masking the attributes in the text data in the training data set to obtain masked text;

[0069] Specifically, the text T is segmented by a tokenizer, and attribute words in the text are randomly selected with a probability of 15% for masking. The attribute words to be masked are changed to "<|mask|>", and finally the masked text is obtained.

[0070] Step S3, constructing a cross-modal pedestrian search algorithm, setting and adjusting an algorithm parameter set, importing the image, text, and mask text in the above steps into the cross-modal pedestrian search algorithm, iteratively generating a cross-modal pedestrian search algorithm model and saving it;

[0071] Preferably, if Figure 2 As shown, step S3 further includes:

[0072] Step S300, using the image encoder of the pre-trained model to extract the final hidden state features of image I Global Features Use the text encoder of the pre-trained model to extract the global features of the text T Mask Text The final hidden state features

[0073] Step S301, calculating cosine similarity using global features of image and text

[0074]

[0075] Step S302, using the cosine similarity between global features to calculate the probability p of matching between the image and text pair i,j :

[0076]

[0077] Among them, τ represents the temperature coefficient, which is used to control the peak value of the probability distribution and its value is 0.02;

[0078] Step S303, constructing a loss function for matching the matching probability between the image-text pair and the true matching probability;

[0079] The image-text matching loss function is:

[0080]

[0081] The text-image matching loss function is:

[0082]

[0083] Here, ε is a decimal that keeps the denominator numerically stable, and q i,jRepresents the true matching probability of the image-text pair. This loss function can align the matching probability distribution between the image and text pair with the true matching probability distribution, so that the model can effectively increase the correlation between matching pairs.

[0084] Step S304, using the cosine similarity between the text and the text global feature to calculate the probability of matching between the text and the text pair

[0085]

[0086] Step S305: In order to measure the difference between attributes, the intersection over union (IoU) between the attributes is calculated according to the number of identical words between the attributes. i,j , and the corresponding true attribute matching probability

[0087]

[0088] Step S306: construct a loss function for matching the matching probability between text and text pairs with their corresponding true attribute matching probability.

[0089]

[0090] Among them, ε is a decimal that keeps the denominator numerically stable. Through this loss function, texts corresponding to attributes with different differences can be more distinguished in the embedding space, thereby improving the model's ability to identify different attributes.

[0091] Step S307: mask the text The final hidden state features As query (q), the last hidden state feature of image I As key (k) and value (v), input the predefined multimodal encoder MultimodalEncoder (ME) to obtain the final fusion feature for attribute prediction

[0092]

[0093] Step S308: fusion features Through the multi-layer perceptron MLP, the probability distribution of the corresponding attribute prediction results is obtained Where |Υ| represents the size of the vocabulary, which is 49408;

[0094] Step S309: construct a loss function with mask attribute prediction as the goal

[0095]

[0096] Among them, y i Represents the true vocabulary distribution, M represents the set of masked attribute words, and this loss function can implicitly learn the connection between the fine-grained features of the two modalities of image and attribute.

[0097] Step S310: construct a loss function with image classification and text classification as optimization targets The global features of the image and global features of the text The image category probabilities are obtained through classifiers respectively and text category probability and the true label y i For comparison:

[0098]

[0099] Step S311: Matching loss of image-text pair Text and text pair matching loss Mask attribute prediction loss Image and text classification losses Add together the final loss function for model training:

[0100]

[0101] Preferably, in step S3, the verification data set containing text collated in step S1 is read to determine the accuracy of the result of the cross-modal pedestrian search algorithm. If there is no performance gain, the model training is terminated and the corresponding model file is saved, otherwise, the process returns to execute S2.

[0102] Preferably, after step S3, the following steps are also included:

[0103] Step S4, read the test data set organized in step S1 one by one in sequence, load the cross-modal pedestrian search model file saved in step S3, test the search accuracy, and obtain the search accuracy result.

[0104] The method proposed in the present invention is applied to cross-modal pedestrian search. Compared with previous algorithms, on the Market-1501Attribute dataset, the Rank-1 accuracy is improved by 13.1%, and the mAP is improved by 31.0%; on the PETA dataset, the Rank-1 accuracy is improved by 38.3%, and the mAP is improved by 44.9%; on the PA100K dataset, the Rank-1 accuracy is improved by 27.6%, and the mAP is improved by 53.8%.

Claims

1. A method for implementing cross-modal pedestrian search based on a large-scale pre-trained model, characterized in that: The steps include: Step 1: Based on a cross-modal pedestrian search dataset, the images are divided into a training dataset and a validation dataset in proportion, and paired annotation information of pedestrian categories and pedestrian image attributes is obtained respectively; Step 2: Fill the attributes of pedestrian images in the training dataset and the validation dataset according to the custom sentence templates to obtain two corresponding texts; Using the custom sentence template P, fill the pedestrian image attribute A into the corresponding position of the sentence template and convert the attribute A into text T; Step 3: Randomly mask the attributes in the text data of the training dataset to obtain masked text Step 4: Build a cross-modal pedestrian search algorithm based on a large-scale pre-trained model, import the images, texts, and masked texts of the training dataset into the cross-modal pedestrian search algorithm, and generate a cross-modal pedestrian search algorithm model through training iterations; The training iteration process of the cross-modal pedestrian search algorithm model is: Step 401, using a large-scale pre-trained model to extract global features and final hidden state features of the image, global features of the text, and final hidden state features of the masked text; Specifically: The image encoder of the large-scale pre-trained model is used to extract the final hidden state features of image I. and global features Extract global features of text T using a text encoder based on a large-scale pre-trained model Mask Text The final hidden state features Step 402, calculating a loss function based on features extracted from the image, text, and masked text, and training the model based on the loss function; The loss function is the sum of the image-text pair matching loss function, the text-text pair matching loss function, the mask attribute prediction loss function, and the image-text classification loss function. The formula is: Among them, the image-text matching loss function is: The text-image matching loss function is: Text and text pair matching loss function Mask attribute prediction loss function Image and text classification loss functions p i,j is the image-text matching probability, ε is a decimal that keeps the denominator numerically stable, and q i,j represents the true matching probability of the image text pair, p j,i Represents the matching probability of text-image; represents the probability of matching between text and text pairs, The probability of true attribute matching between text and text pairs; |Υ| represents the size of the vocabulary, y i represents the real vocabulary distribution, M represents the set of masked attribute words; y i is the real label, is the image category probability, is the text category probability; Step 403, using the verification data set containing text to determine the accuracy of the trained cross-modal pedestrian search algorithm model, to determine whether the model has performance gain, if not, the model training ends and the corresponding model file is saved; otherwise, return to step 3 and perform random masking again; Step 5: Read the test data set one by one, load the trained cross-modal pedestrian search algorithm model file, test the search accuracy, and obtain the search accuracy result.

2. According to claim 1, a method for implementing cross-modal pedestrian search based on a large-scale pre-trained model is characterized in that: The random masking process is as follows: Use the word segmenter to segment the text T, randomly select attribute words in the text with a probability of 15% for masking, and change the attribute words to be masked to "<|mask|>", and finally get the masked text 3. According to claim 1, a method for implementing cross-modal pedestrian search based on a large-scale pre-trained model is characterized in that: The calculation process of the image-text pair matching loss function is: First, the cosine similarity is calculated using the global features of the image and text And further calculate the probability p of matching between image and text pairs i,j ; Cosine similarity Matching probability p i,j : Among them, τ represents the temperature coefficient, which is used to control the peak value of the probability distribution; Then, the image-text pair matching loss function is calculated based on the matching probability between the image and text pair and the true matching probability; The image-text matching loss function is: The text-image matching loss function is: Here, ε is a decimal that keeps the denominator numerically stable, and q i,j represents the true matching probability of the image-text pair, p j,i Represents the matching probability of text-image.

4. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1 is characterized in that: The calculation process of the text and text pair matching loss function is: First, the cosine similarity between text and text global features is used to calculate the probability of matching between text and text pairs. in, Represents the cosine similarity between text and global features of the text; Then, according to the number of identical words between attributes, the intersection over union (IoU) between attributes is calculated. i,j , and the corresponding true attribute matching probability Among them, A i and A j are the i-th and j-th attribute sets respectively; Finally, the text and text pair matching loss function is calculated based on the matching probability between the text and the text pair and the matching probability of their corresponding true attributes.

5. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1 is characterized in that: The calculation process of the mask attribute prediction loss function is: First, mask the text The final hidden state features As query (q), the last hidden state feature of image I As key (k) and value (v), input the predefined multimodal encoder Multimodal Encoder (ME) to obtain the final fusion feature for attribute prediction Then, the fusion features Through the multi-layer perceptron MLP, the probability distribution of the corresponding attribute prediction results is obtained Where |Υ| represents the size of the vocabulary; Finally, construct a loss function with mask attribute prediction as the target Among them, y i Represents the true vocabulary distribution, M represents the set of masked attribute words, and this loss function can implicitly learn the connection between the fine-grained features of the two modalities of image and attribute.

6. The method for implementing cross-modal pedestrian search based on a large-scale pre-trained model according to claim 1 is characterized in that: The calculation process of the image and text classification loss function is: The global features of the image and global features of the text The image category probabilities are obtained through classifiers respectively and text category probability and the true label y i Compare and construct loss functions with image classification and text classification as optimization objectives

Images