Text-to-person retrieval method based on bounding box extraction and semantic consistency constraints

Abstract

This paper discloses a cross-modal text-to-person retrieval method based on bounding box extraction and semantic consistency constraints. The method comprises the following steps: extracting fine-grained bounding boxes from images; extracting fine-grained noun phrases from text; generating a training set; constructing a fine-grained aggregation network; training the fine-grained aggregation network; and searching for people using text. This paper constructs a text-to-person retrieval model based on bounding box extraction and semantic consistency constraints. This method leverages the visual language knowledge from existing large-scale pre-trained models (GLIP and CLIP) and uses textual cues and GLIP to accurately extract key local features for identifying pedestrians, improving the accuracy of pedestrian retrieval. CLIP is used to extract visual and linguistic features to obtain a more comprehensive semantic representation. Finally, a constraint method is designed to maintain semantic consistency of features, reducing noise interference and improving the stability of pedestrian retrieval.

Description

Technical Field

[0001] This invention belongs to the field of multimodal technology and, more specifically, to a text-to-person retrieval method based on bounding box extraction and semantic consistency constraints in the field of cross-modal retrieval from text to person images. This invention can be used for cross-modal person retrieval, searching for target persons from an image library using descriptive text information. Background Art

[0002] The goal of person retrieval is to find a person of interest within a given set of scene images. With the proliferation of surveillance equipment and technological advancements, the amount of surveillance data generated is increasing. Using computers to analyze and extract information from this data to efficiently retrieve the target person has become a critical approach. Cross-modal person retrieval involves retrieving images of a target person from an image library using descriptive text information. Early work on person retrieval focused on image-by-image search, using existing pedestrian images as query criteria. However, in practical applications, the template images of the pedestrians used for retrieval are often not known in advance. Text descriptions provide a relatively comprehensive way to describe a person's attributes. Therefore, text-based person retrieval is more flexible and versatile, with widespread application in public security systems, social governance, and personal photo album searches. As a fusion task between person retrieval and image-text retrieval, the key to text-based cross-modal person retrieval lies in mining fine-grained attribute information from both images and text, and constructing highly resolvable local features to better establish identity correspondences between images and text.

[0003] As researchers continue to deepen their research on image-text retrieval and person re-identification tasks, many text-based cross-modal pedestrian retrieval methods have been proposed.

[0004] Northwestern Polytechnical University disclosed a cross-modal retrieval method from text to pedestrian images in its patent application "Text Pedestrian Retrieval Method Based on Text Dynamically Guided Visual Feature Extraction" (application number: 2021104547243, application publication number: CN 113221680 A). This method first divides each visual feature horizontally into multiple strip areas, and then uses a text-based filter generator to generate a filter with a specific description to indicate the importance of the mentioned image area in the text input. Then, a text-dynamically guided visual feature extractor is used to dynamically fuse partial visual features of each text description, and finally cross-modal feature matching is performed on the text feature vector and the final visual feature. This method realizes the retrieval of pedestrian images through text through the interaction between cross-modal information, further improving the accuracy of pedestrian retrieval tasks. However, this method still has two shortcomings: first, this method only cuts the image features horizontally, but horizontal cutting is based on pixels, so it may cut local areas such as bags, clothes, and pants, losing the integrity of regional recognition; second, since everyone's dressing style and walking posture are different, the position and method of horizontal cutting will also vary from person to person, resulting in some horizontal strips of the image being aligned dominated by background information, resulting in inaccurate fine-grained feature extraction.

[0005] Chen et al. published a text-based cross-modal pedestrian retrieval method in their article "A simple but effective part-based convolutional baseline for text-based person search" (Neurocomputing 2022). The method proposes a simple and effective end-to-end text-based pedestrian search learning framework. First, visual and textual local representations are extracted through a dual-path local alignment network structure. The visual local representation is extracted by horizontal segmentation, where the pedestrian image is horizontally segmented into several stripes. In the text representation learning branch, word embeddings are learned through a BERT model with pre-training and fixed parameters and further processed by a multi-branch residual network. In each branch, the model learns text representations to adaptively match the corresponding visual local representations in order to extract aligned textual local representations. In addition, a multi-stage cross-modal matching strategy is proposed, which eliminates the modal differences between low-level, local and global features, thereby gradually narrowing the feature differences between the image domain and the text domain. However, the problem is that this method does not take into account the correlation between intra-modal information. Specifically, each pedestrian image can be divided into multiple regions, each containing varying degrees of fine-grained features, such as shoes, pants, and shirts. These fine-grained features should interact and complement the global features. However, this approach still has shortcomings. Pedestrian images often have varying angles, lighting, background noise, and other background noise. Therefore, the global representation is easily corrupted by noise, resulting in semantic errors, leading to divergence from the local representation and reducing retrieval accuracy. Summary of the Invention

[0006] The present invention aims to address the shortcomings of the aforementioned prior art by providing a text-to-person retrieval method based on bounding box extraction and semantic consistency constraints. This approach aims to address the problems in existing cross-modal text-to-person image retrieval methods, which either fail to effectively utilize the critical fine-grained information in the image for true identity recognition, resulting in the loss of some fine-grained semantics; or overly focus on cross-modal matching of image and text features, ignoring the semantic consistency of different granularity information within the modality.

[0007] The idea of ​​achieving the purpose of the present invention is that the present invention introduces a more accurate and complete fine-grained information extraction method, which uses a predefined vocabulary of interest as a text prompt to obtain relevant bounding boxes in a targeted manner, such as body parts, clothing details, accessories, and other key local areas for identifying pedestrian identities, to obtain more accurate and rich fine-grained features. In this way, the problem that the current existing text-to-pedestrian image cross-modal retrieval methods do not make effective use of the key fine-grained information in the image that truly identifies the identity, resulting in the loss of some fine-grained semantics, is solved. The present invention designs a method to maintain semantic consistency constraints on image features of different granularities. By maximizing the mutual information between global features and local features, the image representation of the model at different granularities remains centered on identity representative features, avoiding ambiguity between the global semantic representation and the local semantic representation of the same identity due to interference from noise such as different angles, lighting, and background in the image, thereby solving the problem of consistency in semantic representation of information of different granularities within the existing technology modality.

[0008] The technical solution adopted by the present invention includes the following steps:

[0009] Step 1: Extract fine-grained bounding boxes for each image in the image-text pair of the dataset:

[0010] The textual hints describing the attributes of pedestrians are fed into the phrase localization model GLIP along with each image to extract the bounding box of each image in the dataset.

[0011] Step 2: extract fine-grained noun phrases for each text in the image-text pair of the dataset;

[0012] Step 3: Generate training set:

[0013] Each image and its corresponding fine-grained bounding box, as well as each text and its corresponding fine-grained noun phrase, are combined into a sample, and all samples in the dataset are combined into a training set;

[0014] Step 4: Build a fine-grained aggregation network:

[0015] Step 4.1: Build a subnetwork consisting of the CLIP image encoder and text encoder. The image encoder is CLIP ViT-B / 16, and the text encoder is CLIP X-former. Both encoders are composed of a 12-layer Transformer block connected in series with a fully connected layer, and the output vector dimension is 512.

[0016] Step 4.2: Build a fine-grained aggregation network consisting of two branches in parallel. The first branch is composed of an image encoder and a bidirectional GRU in series, and the second branch is composed of a text encoder and a bidirectional GRU in series.

[0017] Step 5: Train the fine-grained aggregation network:

[0018] The training set is input into the fine-grained aggregation network. The image encoder forward propagates to output the global features of each image. The text encoder forward propagates to output the global features of each text. All bounding boxes of each image are forward propagated through the fine-grained aggregation network to obtain a local image feature. The noun phrase sequence of each text is forward propagated through the fine-grained aggregation network to obtain a local text feature.

[0019] The global features of the image and text are used to calculate the inter-modal semantic alignment loss and identity classification loss respectively. The global features of the image and the local features of the image are used to calculate the semantic consistency constraint loss. The three loss functions are added together as the total target loss function to iteratively update the network parameters until the total target loss function converges, resulting in a trained fine-grained aggregation network.

[0020] Step 6: Use text to retrieve pedestrians:

[0021] Step 6.1: For the text to be retrieved and the pedestrian image to be retrieved, obtain the corresponding noun phrases and bounding boxes using the methods of steps 1 and 2, respectively, and input them into the trained fine-grained aggregation network to obtain the global features of the text, the local features of the text, and the global features of the image, and the local features of the image;

[0022] In step 6.2, the local similarity, global similarity, and weighted total similarity of the text to be retrieved and the pedestrian image to be retrieved are calculated respectively, the similarities of the pedestrian images to be retrieved are sorted in descending order, and the first 10 images are selected from the image sequence as the retrieval results.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] First, the present invention uses a predefined vocabulary of interest as a text prompt to obtain relevant bounding boxes in a targeted manner, overcoming the problem that the key fine-grained information for truly identifying identities in existing text-to-pedestrian image cross-modal retrieval methods is not effectively utilized, resulting in the loss of some fine-grained semantics. The present invention can extract fine-grained areas with higher identity recognition, better adapt to different pedestrian postures and image scales, and enhance the ability to recognize and distinguish pedestrian identities. At the same time, the present invention is not affected by occlusion problems in the process of extracting fine-grained areas, thereby improving the accuracy of retrieval.

[0025] Second, the present invention designs a consistency constraint method to maintain high mutual information of image features of different granularities, so that global features can better complement local detail information. It overcomes the problem that the existing technology focuses too much on cross-modal matching of image and text features and ignores the consistency of semantic representation of information of different granularities within the modality. The model of the present invention can learn the representation of identity consistency between different image views, avoid the interference of background noise and the loss of key information, and improve the robustness and stability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the present invention;

[0027] Figure 2 This is a schematic diagram of fine-grained graphic information extraction according to the present invention;

[0028] Figure 3 It is a schematic diagram of the training process of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.

[0030] Reference Figure 1 , further describing the implementation steps of the embodiment of the present invention.

[0031] Step 1: extract fine-grained bounding boxes for each image in the image-text pair of the dataset.

[0032] The textual hints describing the attributes of pedestrians are fed into the phrase localization model GLIP along with each image to extract the bounding box of each image in the dataset.

[0033] Reference Figure 2, the specific process is further described in detail: traverse the data set to collect words related to the person's physical characteristics, clothing styles, and personal belongings, such as but not limited to "male", "cap", "jacket", "short sleeve", "skirt", "pants", "high heels", "backpack", etc.; since there are multiple words describing synonymous or similar concepts, in order to reduce the redundancy of the vocabulary, these similar words can be summarized into more general categories to obtain a generalized vocabulary. The collected words need to be abstracted, such as abstracting "jeans" and "sweatpants" into "pants", and abstracting "female" and "male" into "person". The abstract general conceptual words of the present invention are as follows: person, coat, pants, bag, eyes, suitcase, head, fur collar, hat, shoes, bicycle, headphones, short sleeve, pocket, arm, car, skirt, hair, doll, hand, box The above vocabulary can guide the model to focus on the key areas in the image related to the identity features of pedestrians and guide the extraction process of fine-grained bounding boxes; the large phrase localization model GLIP is used to extract bounding boxes as accurate fine-grained information of pedestrian images. GLIP is a model for locating targets of interest in images. It can identify and locate target areas in images based on important phrases and phrase combinations in the text. GLIP is modified from a target localization model to a bounding box extraction model. Each word in the text prompt is used as a text query. By calculating the similarity between the text and image features, the corresponding vocabulary features are extracted from the image features. The coordinates of the area covered by the vocabulary features with high confidence are selected as the bounding boxes of the target of interest, capturing the fine-grained areas in the pedestrian image that contain some of the words in the vocabulary. For each image I, the 8 bounding boxes with the highest confidence are extracted as the fine-grained sequence I of the image. bbox =(b1,b2…b8).

[0034] The text prompt describing the pedestrian attributes is ['person.coat.pants.bag.glasses.luggage.head.collar.cap.shoes.bike.headphone.shirt.pocket.legs.car.dress.hair.toy.hands.box.book.mask.cup.cellphone.'].

[0035] Extracting the bounding box of each image in the dataset refers to inputting the text prompt describing the pedestrian attributes and each image into the phrase localization model GLIP at the same time, and selecting the 8 bounding boxes with the highest confidence from the detection layer output by GLIP as the fine-grained information of the image and storing them in a list sequence.

[0036] Step 2, extract fine-grained noun phrases for each text in the image-text pair of the dataset.

[0037] Reference Figure 2 , further describe the specific process in detail: use the spacy toolkit to extract noun phrases appearing in the text description, first install the spacy library and load the corresponding English model "en_core_web_sm", then use the nlp() function to pass the text to the language model for processing to obtain a document object, and then traverse each word in the document to determine whether its part-of-speech tag is a noun. If it is a noun, continue to determine whether it is the head of a noun phrase. If it is the head of a noun phrase, use the noun_chunks attribute to obtain the noun phrase p, iterate through the noun phrases in the document, store them in a list sequence, and stipulate that the number of phrases for each text is 15. For each text T, if the number of extracted noun phrases is less than 15, use a special padding field ['pad'] to fill it. If the number of extracted noun phrases exceeds 15, truncate the noun phrases at the end of the list and retain only the first 15 noun phrases. The extracted phrase fine-grained sequence T phrase =(p1,p2…p 15 ).

[0038] The extracting of fine-grained noun phrases from each text in an image-text pair of a dataset refers to generating a list sequence for each text description to store the noun phrases appearing in each text description in the dataset, repeatedly adding a "pad" field to the end of the sequence when the length of the list sequence is less than the rated length of the list sequence until the length of the sequence is equal to the rated length, and truncating the portion of the list sequence whose length is greater than the rated length. The rated length of the list sequence is the average length of all list sequences.

[0039] Step 3: Generate a training set.

[0040] Each image and its corresponding fine-grained bounding box, as well as each text and its corresponding fine-grained noun phrase, are combined into a sample, and all samples in the dataset form a training set.

[0041] Each image-text pair and its corresponding extracted fine-grained information constitute a training sample X = (I, T, I bbox ,T phrase ), N training samples and their corresponding identity label sets form a training set S = {(X n ,L n )|1≤n≤N}.

[0042] Step 4: Build a fine-grained aggregation network.

[0043] Step 4.1: Build a subnetwork consisting of the CLIP image encoder and text encoder. The image encoder is CLIP ViT-B / 16, and the text encoder is CLIP X-former. Both encoders are composed of a 12-layer Transformer block connected in series with a fully connected layer, and the output vector dimension is 512.

[0044] In step 4.2, a fine-grained aggregation network is constructed, which consists of two branches in parallel. The first branch is composed of the series connection of the image encoder and the bidirectional GRU, and the second branch is composed of the series connection of the text encoder and the bidirectional GRU.

[0045] Step 5: Train the fine-grained aggregation network.

[0046] Reference Figure 3 , give a detailed description of the training process.

[0047] The training set is input into the fine-grained aggregation network. The image encoder forward propagates and outputs the global features of each image. The text encoder forward propagates and outputs the global features of each text. All bounding boxes of each image are forward propagated through the fine-grained aggregation network to obtain an image local feature, and the noun phrase sequence of each text is forward propagated through the fine-grained aggregation network to obtain a text local feature.

[0048] Specifically, for a pedestrian image I, input it into CLIP-ViT to obtain the global feature I of the image global , its corresponding bounding box sequence I bbox The area covered by each bounding box in the image is cropped out as an image input to CLIP-ViT to obtain the regional feature sequence of the image [I l1 ,I l2 ,…I l8 ]. Use CLIP's word segmenter to describe the pedestrian text T and the corresponding noun phrase sequence T phrase Perform word segmentation, generate word-level vector representation, and use CLIP's text encoder to extract the global features T of the text description global and phrase feature sequence [T l1 ,T l2 ,…T l15]. After obtaining fine-grained local image and text features, it is usually necessary to aggregate the local features to obtain the overall fine-grained feature representation. The common practice is to use average pooling or maximum pooling for aggregation, which often only considers the statistical information of local features and ignores the correlation and interaction between each other, which may lead to the loss of feature information and the generation of information bottlenecks. The present invention uses GPO, a flexible pooling operator, to aggregate the local feature sequence of images and texts, inputs the feature sequence into the bidirectional GRU in the fine-grained aggregation network, learns the weight coefficient θ of the pooling operator of the feature vector in the sequence, and performs weighted representation on the feature sequence according to the weight coefficient to obtain local features. Local feature I of image I local =GPO([I l1 ,I l2 ,…I l8 ]), local features T of text T local =GPO([T l1 ,T l2 ,…T l15 -).

[0049] The global features of images and texts are used to calculate the inter-modal semantic alignment loss and identity classification loss respectively, and the global features of images and local features of images are used to calculate the semantic consistency constraint loss. The three loss functions are added together as the total target loss function to iteratively update the network parameters until the total target loss function converges to obtain a trained fine-grained aggregation network.

[0050] All bounding boxes of each image are forward propagated through the fine-grained aggregation network to obtain an image local feature, which means that all bounding boxes of each image are forward propagated in the image encoder to obtain a regional feature sequence. The sequence learns the trainable parameter vector θ in the bidirectional GRU as the feature weight coefficient of different positions, and uses the GPO aggregation operator to weight the feature vector of each position to obtain the aggregated image local feature I local , in, represents the kth regional feature of the regional feature sequence, θ k represents the weight of the k-th region feature, and GPO(·) represents the aggregation operator.

[0051] The noun phrase sequence of each text is passed through a fine-grained aggregation network to obtain a text local feature. The noun phrase sequence of each text is forward propagated in the text encoder to obtain a phrase feature sequence ω, which is used to learn a trainable parameter vector θ in a bidirectional GRU as a feature weight coefficient at different positions. The feature vector at each position is weighted and represented using the GPO aggregation operator to obtain the aggregated text local feature T. local , Among them, ωk represents the kth phrase feature of the phrase feature sequence, θ k represents the weight of the k-th phrase feature, and GPO(·) represents the aggregation operator.

[0052] The semantic consistency constraint loss is calculated as follows:

[0053]

[0054] Among them, L info represents the semantic consistency constraint loss, E p represents expectation, log represents logarithmic operation with the natural constant e as the base, exp represents exponential operation with the natural constant e as the base, I global Represents the global features of the image, represents the positive local features, τ represents the learnable temperature parameter, N represents the number of unmatched image-text pairs in a batch, represents the nth negative local feature, and the superscript T represents the transposition operation.

[0055] The inter-modal semantic alignment loss is calculated as follows:

[0056] S global =(I global ) T (T global )

[0057]

[0058] S local =(I local ) T (T local )

[0059]

[0060]

[0061] Among them, S global represents the global feature similarity matrix, T global Represents the global features of the text, represents the global inter-modality semantic alignment loss, τ p Indicates the temperature parameter that adjusts the slope of the gradient. represents the similarity score of the pairwise similarity of the i-th row of the similarity matrix, α represents the lower limit of the similarity of the pairwise similarity, τ n represents the temperature parameter that negatively regulates the gradient slope, represents the similarity score of the negative pair in the i-th row of the similarity matrix, β is the upper limit of the similarity of the negative pair, S local represents the local feature similarity matrix, T localRepresents the local features of the text, represents the local inter-modality semantic alignment loss, L align represents the inter-modality semantic alignment loss.

[0062] The identity classification loss is calculated as follows:

[0063] p v =softmax((W) t (I global ))

[0064] p t =softmax((W) t (T global ))

[0065] L id =(-log(p v (c))+-log(p t (c))) / 2

[0066] Among them, p v represents the identity prediction label of the image, softmax(·) represents the normalized exponential function, W represents the shared identity projection matrix, and p t Represents the identity prediction label of the text, L id represents the identity classification loss, p v (c) represents the probability of correctly predicting the identity category c through the global features of the image, p t (c) represents the probability of correctly predicting the identity category c through the global features of the text.

[0067] The semantic consistency loss maximizes the mutual information between global and regional image features, learning a consistent representation of a person's identity across different image views. Since the text corresponding to different images of the same identity may be depicted from different perspectives, no semantic consistency constraint is imposed on the text modality. The inter-modality semantic alignment loss uses cosine similarity as a measure of the correlation between visual and textual features. For positive pairs, the cosine similarity S+ is encouraged to be as large as possible. However, for negative pairs, forcing the cosine similarity S- to be as small as possible is an arbitrary constraint, as different negative pairs have different information and thus different similarity scores. Therefore, two margins β are set as the upper limit of similarity for negative pairs, and α as the lower limit of similarity for positive pairs. It is hoped that the similarity score S+ for positive pairs is no less than α, while the similarity score S- for negative pairs is no greater than β. The identity discrimination loss calculates the cross-entropy between the predicted identity label and the true label for the image or text, ensuring that the feature representations of the same identity are closely clustered in the joint embedding space.

[0068] Step 6: Use text to retrieve pedestrians.

[0069] Step 6.1: For the text to be retrieved and the pedestrian image to be retrieved, obtain the corresponding noun phrases and bounding boxes using the methods of steps 1 and 2, respectively, and input them into the trained fine-grained aggregation network to obtain the global features of the text, the local features of the text, and the global features of the image, and the local features of the image;

[0070] In step 6.2, the local similarity, global similarity, and weighted total similarity of the text to be retrieved and the pedestrian image to be retrieved are calculated respectively, the similarities of the pedestrian images to be retrieved are sorted in descending order, and the first 10 images are selected from the image sequence as the retrieval results.

[0071] The weighted total similarity is calculated by the following formula:

[0072] S=α*S global +β*S local

[0073] Here, α and β represent weight coefficients, where α+β=1.

Claims

1. A text-to-person retrieval method based on bounding box extraction and semantic consistency constraints, characterized by: Extract fine-grained bounding boxes for each image in the image-text pair of the dataset and calculate the semantic consistency constraint loss; the steps of this retrieval method include the following: Step 1: Extract fine-grained bounding boxes for each image in the image-text pair of the dataset: The textual hints describing the attributes of pedestrians are fed into the phrase localization model GLIP along with each image to extract the bounding box of each image in the dataset. Step 2: extract fine-grained noun phrases for each text in the image-text pair of the dataset; Step 3: Generate training set: Each image and its corresponding fine-grained bounding box, as well as each text and its corresponding fine-grained noun phrase, are combined into a sample, and all samples in the dataset are combined into a training set; Step 4: Build a fine-grained aggregation network: Step 4.1: Build a subnetwork consisting of the CLIP image encoder and text encoder. The image encoder is CLIP ViT-B / 16, and the text encoder is CLIP X-former. Both encoders are composed of a 12-layer Transformer block connected in series with a fully connected layer, and the output vector dimension is 512. Step 4.2: Build a fine-grained aggregation network consisting of two branches in parallel. The first branch is composed of an image encoder and a bidirectional GRU in series, and the second branch is composed of a text encoder and a bidirectional GRU in series. Step 5: Train the fine-grained aggregation network: The training set is input into the fine-grained aggregation network. The image encoder forward propagates to output the global features of each image. The text encoder forward propagates to output the global features of each text. All bounding boxes of each image are forward propagated through the fine-grained aggregation network to obtain a local image feature. The noun phrase sequence of each text is forward propagated through the fine-grained aggregation network to obtain a local text feature. The global features of the image and text are used to calculate the inter-modal semantic alignment loss and identity classification loss respectively. The global features of the image and the local features of the image are used to calculate the semantic consistency constraint loss. The three loss functions are added together as the total target loss function to iteratively update the network parameters until the total target loss function converges, resulting in a trained fine-grained aggregation network. Step 6: Use text to retrieve pedestrians: Step 6.1: For the text to be retrieved and the pedestrian image to be retrieved, obtain the corresponding noun phrases and bounding boxes using the methods of steps 1 and 2, respectively, and input them into the trained fine-grained aggregation network to obtain the global features of the text, the local features of the text, and the global features of the image, and the local features of the image; In step 6.2, the local similarity, global similarity, and weighted total similarity of the text to be retrieved and the pedestrian image to be retrieved are calculated respectively, the similarities of the pedestrian images to be retrieved are sorted in descending order, and the first 10 images are selected from the image sequence as the retrieval results.

2. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: The text prompt describing the pedestrian attributes in step 1 is ['person.coat.pants.bag.glasses.luggage.head.collar.cap.shoes.bike.headphone.shirt.pocket.legs.car.dress.hair.toy.hands.box.book.mask.cup.cellphone.'].

3. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: Extracting the bounding box of each image in the dataset as described in step 1 means inputting the textual prompt describing the attributes of pedestrians into the phrase localization model GLIP simultaneously with each image, and selecting the eight bounding boxes with the highest confidence from the detection layer output by GLIP as the fine-grained information of the image and storing them in a list sequence.

4. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: Extracting fine-grained noun phrases from each text in the image-text pair of the dataset in step 2 refers to generating a list sequence for each text description to store the noun phrases that appear in each text description in the dataset. When the length of the list sequence is less than the rated length of the list sequence, a "pad" field is repeatedly added to the end of the sequence until the length of the sequence equals the rated length. The portion of the list sequence that is longer than the rated length is truncated. The rated length of the list sequence is the average length of all list sequences.

5. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: In step 5, all bounding boxes of each image are forward propagated through the fine-grained aggregation network to obtain an image local feature, which means that all bounding boxes of each image are forward propagated in the image encoder to obtain a regional feature sequence. The sequence learns the trainable parameter vector θ in the bidirectional GRU as the feature weight coefficient of different positions, and uses the GPO aggregation operator to weight the feature vector of each position to obtain the aggregated image local feature I local , in, represents the kth regional feature of the regional feature sequence, θ k represents the weight of the k-th region feature, and GPO(·) represents the aggregation operator.

6. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: The noun phrase sequence of each text in step 5 is passed through the fine-grained aggregation network to obtain a text local feature. This means that the noun phrase sequence of each text is forward propagated through the text encoder to obtain a phrase feature sequence ω. The parameter vector θ learned and trained in the bidirectional GRU is used as the feature weight coefficient of different positions. The GPO aggregation operator is used to weight the feature vector of each position to obtain the aggregated text local feature T. local , Among them, ω k represents the kth phrase feature in the phrase feature sequence, θ k represents the weight of the k-th phrase feature, and GPO(·) represents the aggregation operator.

7. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: The semantic consistency constraint loss in step 5 is calculated as follows: Among them, L info represents the semantic consistency constraint loss, E p represents expectation, log represents logarithmic operation with the natural constant e as the base, exp represents exponential operation with the natural constant e as the base, I global Represents the global features of the image, represents the positive local features, τ represents the learnable temperature parameter, N represents the number of unmatched image-text pairs in a batch, represents the nth negative local feature, and the superscript T represents the transposition operation.

8. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 7, characterized in that: The inter-modality semantic alignment loss described in step 5 is calculated as follows: S global =(I global ) T (T global ) S local =(I local ) T (T local ) Among them, S global represents the global feature similarity matrix, T global Represents the global features of the text, represents the global inter-modality semantic alignment loss, τ p Indicates the temperature parameter that adjusts the slope of the gradient. represents the similarity score of the pairwise similarity of the i-th row of the similarity matrix, α represents the lower limit of the similarity of the pairwise similarity, τ n represents the temperature parameter that negatively regulates the gradient slope, represents the similarity score of the negative pair in the i-th row of the similarity matrix, β is the upper limit of the similarity of the negative pair, S local represents the local feature similarity matrix, T local Represents the local features of the text, represents the local inter-modality semantic alignment loss, L align represents the inter-modality semantic alignment loss.

9. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: The identity classification loss described in step 5 is calculated as follows: p v =softmax((W) t (I global )) p t =softmax((W) t (T global )) L id =(-log(p v (c))+-log(p t (c))) / 2 Among them, p v represents the identity prediction label of the image, softmax(·) represents the normalized exponential function, W represents the shared identity projection matrix, and p t Represents the identity prediction label of the text, L id represents the identity classification loss, p v (c) represents the probability of correctly predicting the identity category c through the global features of the image, p t (c) represents the probability of correctly predicting the identity category c through the global features of the text.

10. The text-to-person retrieval method based on bounding box extraction and semantic consistency constraint according to claim 1, characterized in that: The weighted total similarity described in step 6.2 is calculated by the following formula: S=α*S global +β*S local Here, α and β represent weight coefficients, and α+β=1.

Images