Text pedestrian retrieval method based on automatic text alignment

Abstract

The invention provides a text pedestrian retrieval method based on text automatic alignment, which pays attention to the relation between an image and a whole text by calculating a global feature, and automatically obtains a weighted text feature for describing each part of the image of a human body through a word attention module. Through a local feature interaction module, mutual association between each part of images and between words in text description is automatically realized, that is, retrieval is started after automatic alignment of text content and the images is realized. The method is flexible in application, and is a text pedestrian retrieval method according with an actual application scene.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a text pedestrian retrieval method based on automatic text alignment. Background Art

[0002] In recent years, pedestrian retrieval has received extensive attention. The goal of this task is to retrieve the corresponding pedestrian images from an image database given a query. According to the type of data to be searched, pedestrian retrieval can be divided into image-based pedestrian retrieval and text-based pedestrian retrieval. Among them, image-based pedestrian retrieval requires at least one specific pedestrian photo for query, which is not always available in practical applications; while text-based pedestrian retrieval only requires a text description of the pedestrian to be searched, which is more in line with daily usage habits and more in line with the needs of practical application scenarios than the former method.

[0003] Existing text pedestrian retrieval methods generally use models to extract global features of text and images, and then calculate the similarity between the two features to achieve matching between the two features; however, we believe that it is not enough to focus only on these global features of images and texts, because the text describing pedestrian images will be subjective, for example, some will describe the upper body first, and some will describe the lower body first, and the focus of the description will also be different. As a result, there will be great differences between the contents of multiple texts describing the same pedestrian image, resulting in large differences in the calculated features. Therefore, it is unreasonable to only focus on the global information of text and images, and it is also necessary to consider the relationship between pedestrian images and description texts from a local perspective.

[0004] To solve the above problems, the present invention provides a text pedestrian retrieval method based on automatic text alignment, which focuses on the connection between the image and the text as a whole by calculating global features, focuses on the connection between the text and each part of the image by calculating local image features and text weighted features, and focuses on the connection between each part of the image and the connection between each word in the text by calculating feature vectors of additional associated information. The method is a text pedestrian retrieval method that conforms to actual application scenarios. Summary of the invention

[0005] A text pedestrian retrieval method based on automatic text alignment includes the following steps: S1, uses the BERT model and the ResNeSt model to extract text global features and image global features respectively; S2, perform pooling operations on the global features of the text and the global features of the image to obtain the feature vector and calculate the triplet loss LG ; S3, divide the image into blocks according to human body parts and calculate the local features of the image; use the word attention module calculation method to calculate the text weighted features corresponding to each block of the image; S4, perform pooling operation on text weighted features and image local features to obtain feature vectors and calculate triplet loss Ll ; S5: Calculate the interaction strength between each local feature and the other local features, obtain the image local feature vector and text local feature vector with associated information, and calculate the triplet loss Ln; S6, using the calculated three triplet loss functions to calculate the final loss function between the two modal data of image and text, and using it to train the network; S7, calculate the similarity between the search text and the images in the image library and sort them to obtain the search results.

[0006] The step S1 uses the BERT model and the ResNeSt model to extract text global features and image global features respectively, including: Use BERT as the backbone network and take text as input to obtain the global features of the text; Use ResNeSt as the backbone network and take the image as input to obtain the global features of the image; The BERT is a language representation model that can extract global features of text by using the relationship between the context in the text during feature extraction. The global features of text are composed of the vectors of each word in the text and have two dimensions, height and width. The ResNeSt is an improved network based on the resnet residual network, which can efficiently extract the global features of the image. The global features of the image are features with three dimensions: height, width and number of channels.

[0007] In step S2, a pooling operation is performed on the global features of the text and the global features of the image to obtain a feature vector and calculate the triplet loss LG ,include: Perform row-by-row average pooling on the global features of the text to obtain the global feature vector E of the text; Perform global average pooling on the global features of the image to obtain the global feature vector F of the image; Calculate the distance between the global feature vector of the text and the global feature vector of the image, and calculate the triplet loss LG ; The row-by-row average pooling refers to a method of adding the values ​​of each row in a two-dimensional feature and calculating the average value to replace the row to obtain a vector, which can reduce the number of parameters and the amount of calculation; The global average pooling refers to a method of adding and averaging all values ​​in a two-dimensional feature to obtain a 1×1 vector, which can reduce the number of parameters and the amount of calculation; The calculation of the triplet loss refers to a loss function that minimizes the distance between feature vectors belonging to positive samples and maximizes the distance between feature vectors belonging to negative samples during training. Positive samples are corresponding text and image pairs, and negative samples are non-corresponding text and image pairs.

[0008] The step S3 divides the image into blocks according to the body parts and calculates the local features of the image; and uses the word attention module calculation method to calculate the text weighted features corresponding to each block of the image, including: The image is divided into k non-overlapping blocks of human body parts by body part division, and features are extracted respectively to obtain k local features of the image; The word attention module calculation method is used to calculate the weighted features of the text global feature vector E describing each human body part block, and k text weighted features corresponding to the human body part blocks are obtained respectively; the vectors of all words are combined to obtain the text weighted features describing the kth part of the human body; The word attention module calculation method refers to calculating the probability that each word in the text describes the kth part of the human body, and the probability value is obtained by multiplying a vector to be trained and a given vector and normalizing the result to between 0 and 1.

[0009] In step S4, a pooling operation is performed on the text weighted features and the image local features to obtain a feature vector and calculate the triplet loss. Ll ,include: The text weighted features of the image describing the kth part of the human body are averagely pooled row by row to obtain k vectors in total. The k vectors are concatenated to obtain a text weighted feature vector, i.e., a text local feature vector. Perform global average pooling on each local feature of the image to obtain k vectors; concatenate the k vectors to obtain the local feature vector of the image; Calculate the distance between the text weighted feature vector and the image local feature vector, and calculate the triplet loss Ll； The concatenation refers to an operation of connecting multiple vectors end to end to increase the dimension of the vectors.

[0010] In step S5, the interaction strength between each local feature and the other local features is calculated to obtain the image local feature vector and the text local feature vector with associated information attached, and the triplet loss is calculated. Ln ,include: The calculation of the interaction strength between each local feature and the remaining local features refers to calculating the interaction strength between the i-th feature and the remaining k-1 features by the following formula for the divided k features: For image features, Indicates whether there is a common text description in the image block corresponding to the kth feature and the image block corresponding to the ith feature; For text features, Indicates whether the text corresponding to the k-th feature and the text corresponding to the i-th feature describe the same image area; Using the above local feature interaction module calculation method, k image local feature vectors with additional associated information and k text local feature vectors with additional associated information can be calculated; The k calculated image local feature vectors of the additional associated information are concatenated to obtain a final image local feature vector of the additional associated information; The k calculated local feature vectors of the text of the additional associated information are concatenated to obtain the final local feature vector of the text of the additional associated information; Calculate the distance between the final image local feature vector of the additional associated information and the final text local feature vector of the additional associated information, and calculate the triplet loss Ln .

[0011] The step S6, using the calculated three triplet loss functions to calculate the final loss function between the two modal data of image and text, and using it to train the network, includes: The three triplet loss functions calculated Lg, Ll, Ln The weighted sum of is used as the final loss function between the two modal data of image and text; Input a mini-batch of data, and use the Adam optimizer to optimize the model parameters based on the loss calculated by the final loss function; The Adam optimizer refers to an adaptive moment estimation algorithm, which optimizes the network by adjusting the learning rate of each parameter; After training the data set once, recalculate the three triplet losses and perform the next training until the set number of iterations is reached; The model parameters are updated by gradient updating. After the training is completed, the parameters of the entire network are saved. The dataset includes an image library and a corresponding text description library.

[0012] The step S7, calculating the similarity between the search text and the images in the image library and sorting them to obtain search results, includes: Input the search text into the trained network, calculate the similarity between the search text and the images in the image library, and obtain the sorted images; The first result whose similarity exceeds the threshold is returned as the search result; The similarity refers to the distance between the text feature vector and the image feature vector calculated in the network;

[0013] The method provided by the present invention not only pays attention to the relationship between the overall pedestrian image and the overall text description, but also automatically obtains the weighted text features describing each part of the human body image through a word attention module, and automatically realizes the mutual correlation between each part of the image and each word in the text description through a local feature interaction module, so that the method not only pays attention to the relationship between the overall pedestrian image and the overall text description, but also pays attention to the relationship between each word in the text and each part in the image. This method of automatically aligning text content with images and then performing retrieval is not only more flexible, but also more in line with the needs of actual application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is the network model structure diagram of the text pedestrian retrieval method based on automatic text alignment.

[0015] Figure 2 Flowchart of the text pedestrian retrieval method based on automatic text alignment. DETAILED DESCRIPTION

[0016] In order to more clearly demonstrate the purpose and technical solution of the present invention, the specific implementation methods and parameters of the present invention will be described in more detail below in combination with the specific implementation process and drawings.

[0017] The network model structure diagram of the text pedestrian retrieval method based on automatic text alignment, such as Figure 1 As shown, it includes the following parts.

[0018] Input modules such as Figure 1 As shown in ①: pedestrian images and pedestrian texts are used as network inputs; the network divides the complete pedestrian image into three blocks according to human body parts and then uses them as inputs for the ResNeSt model; During training, pairs of pedestrian images and pedestrian texts are trained; the pedestrian texts are pedestrian description texts; When used, the descriptive text is the input search query text, and the image is each image to be searched in the image library.

[0019] Feature extraction module: Figure 1As shown in ② and ④, it includes a ResNeSt model for extracting image features, a BERT model for extracting text features, a word attention module for extracting text weighted features, a feature interaction module for extracting image local features with additional related information, and text local features with additional related information.

[0020] Features obtained by the feature extraction module: Figure 1 As shown in ③ and ⑤, they include global image features and local image features extracted by the ResNeSt model, global text features extracted by BERT, text weighted features extracted by the word attention module, image local features with additional associated information extracted by the feature interaction module, and text local features with additional associated information.

[0021] Loss function: Figure 1 As shown in ⑤: including triple loss function Lg, Ll, Ln.

[0022] pass Figure 1 The network shown in the figure is trained on the data set to obtain the corresponding model parameters.

[0023] A flowchart of a text pedestrian retrieval method based on automatic text alignment, such as Figure 2 As shown, the following steps are included.

[0024] Step 1: Use the BERT model and the ResNeSt model to extract text global features and image global features respectively, including: Using BERT as the backbone network, we take text as input to get the global features of the text. The feature dimension is 768×N, which represents height and width respectively, and is composed of the vector representations of N words in the text. During training, the input images are preprocessed and resized to 256X256 pixels; ResNeSt is used as the backbone network, and the image is used as input to obtain the global features of the image. The feature dimension is 256×55×55, which represent the number of channels, height and width respectively.

[0025] Step 2: Pool the global features of the text and the global features of the image to obtain the feature vector and calculate the triplet loss LG ,include: Perform row-by-row average pooling on the global features of the text, and change the feature dimension of the global features of the text from 768×N to 768×1, obtaining a 768-dimensional feature vector; Then use a trainable matrix W1 with a height and width of 128 and 768 to multiply the feature vector to obtain the global feature vector of the text, and the dimension of the vector is 128×1; Perform global average pooling on the global features of the image, reducing the feature dimension of the global features of the image from 256×55×55 to 256×1×1, that is, 256×1, to obtain a 256-dimensional feature vector; Then use a trainable matrix W2 with a height and width of 128 and 256 to multiply the feature vector to obtain the global feature vector of the image, and the dimension of the vector is 128×1; Calculate the distance between the global feature vector of the text and the global feature vector of the image, and calculate the triplet loss Lg, The specific distance calculation method is as follows: LG = max(d(a, n) - d(a, p) + margin, 0); where, (a, n) refers to a pair of matching image global feature vectors and text global feature vectors, (a, p) refers to a pair of unmatched image global feature vectors and text global feature vectors, The margin is set to 0.2 to prevent the value from being 0 when d(a, n) = d(a, p). d() means calculating the distance between two vectors. The specific calculation method is: d(a, n) = a T n / ||a||||n||, where T means transposing the vector and || || means taking the modulus of the vector.

[0026] Step 3: Divide the image into blocks according to human body parts and calculate the local features of the image; use the word attention module calculation method to calculate the text weighted features corresponding to each block of the image, including: The image is divided into three non-overlapping blocks of human body parts by human body parts, and the divided human body images are adjusted to 256×256 pixels; The ResNeSt model is used to extract features for each block, and three local image features are obtained. The dimension of each local image feature is 256×55×55, which represents the number of channels, height and width respectively. The word attention module calculation method is used to calculate the weighted features of the text global feature vector E for the description of each human body part block, and three text weighted features corresponding to the human body part blocks are obtained respectively; The feature dimension of the text weighted feature is 768×N, which represents height and width respectively, where N is the length of the word in the text; The word attention module calculation method refers to calculating the probability that each word in the text describes the kth part of the human body. The probability value is obtained by multiplying a vector to be trained and a given vector and normalizing the result to between 0 and 1. During the training process, the vector to be trained is continuously adjusted to adjust the size of the probability. The probability is multiplied by the vector of the word to obtain the vector of the word with additional weight describing the kth part of the human body. The vectors of all words are combined to obtain the text weighted features describing the kth part of the human body. The specific calculation method is as follows: First calculate the probability of each word in the text describing the kth part of the human body: s k i = sigmoid(W3 k × ei), where the sigmoid function can convert the value to a value between 0 and 1. ei represents the vector of the i-th word in the text, k refers to the current calculation of the k-th block of human body images after segmentation, and W3 k is a trainable 1×768 vector, ei is a 768×1 word vector, and the value obtained by multiplying the two is input into the sigmoid function to calculate the probability s k i ; Combine N words with the corresponding s k The product combination of i can get the text weighted features describing the k-th part of the human body image, Ek = [s k 1*e1, s k 2*e2, ... ,s k n*en], the feature dimension is 768×N, representing height and width respectively, and the text weighted feature is composed of the vectors of N words.

[0027] Step 4: Pool the text weighted features and image local features to obtain feature vectors and calculate the triplet loss. Ll ,include: Perform row-by-row average pooling on each text weighted feature, reducing the feature dimension of the text weighted feature from 768×N to 768×1, converting it into a 768-dimensional feature vector; Then use a trainable matrix W4 with a height and width of 128 and 768 to multiply the feature vector to obtain the text weighted feature vector t describing the kth part of the human body l k , the dimension of the vector is 128×1; The three text weighted features are concatenated to obtain the final text weighted feature vector, the dimension of which is 384×1, expressed as t l ; Perform global average pooling on each local feature of the image, reducing the feature dimension of the local feature of the image from 256×55×55 to 256×1×1, i.e. 256×1, and converting it into a 256-dimensional feature vector; Then use a trainable matrix W5 with a height and width of 128 and 256 to multiply the feature vector to obtain the kth image local feature vector v l k , the dimension of the vector is 128×1; The local feature vectors of the three human images are concatenated to obtain the final local feature vector of the image. The dimension of the vector is 384×1, which is expressed as v l ; Calculate the distance between the text weighted feature vector and the image local feature vector, and calculate the triplet loss Ll。

[0028] Step 5: Calculate the interaction strength between each local feature and the other local features, obtain the image local feature vector and text local feature vector with associated information, and calculate the triplet loss Ln ,include: The calculation of the interaction strength between each local feature and other local features refers to calculating the interaction strength between the kth feature and the remaining k-1 features using the following formula for the divided k features: , For image features, S ki Indicates whether there is a common text description in the image block corresponding to the kth feature and the image block corresponding to the ith feature; For text features, S ki Indicates whether the text corresponding to the k-th feature and the text corresponding to the ith feature describe the same image area; The specific calculation method is: , Among them W6 k For W7 i is a trainable matrix with a height and width of 128 and 128, k, i represents the kth and ith blocks of images, v l k and v l i The sum is the local feature vector of the kth image and the local feature vector of the ith image, i is not equal to k, the dimension of the vector is 128×1, and the calculated S ki is a value; Using alpha ki Calculate the kth intermediate vector with associated information attached. The specific calculation method is: v midk =W8 k ·sum(α ki W7 i (v l i )), among which W8 k is a trainable matrix with a height and width of 128 and 128, sum() means adding from i=1 to i=3, and i≠k, for α ki W7 i (v l i ) The values ​​of the calculated feature vectors are added, and the vector dimension 128×1 remains unchanged; the kth intermediate vector v with associated information attached is calculated mid k, v mid The dimension of k is 128×1; Use the kth intermediate vector v with associated information mid k Calculate the local feature vector of the image with associated information attached. The specific calculation method is: v n k =W9 k (v l k + v mid k ), W9 k is a trainable matrix with a height and width of 128 and 128, and the kth local feature vector v of the image with associated information attached is obtained n k , v n k The dimension is 128×1. The three local feature vectors of the image with associated information attached are concatenated to obtain the final local feature vector of the image with associated information attached, with a dimension of 384×1. Using the same method, we can calculate the local feature vector of the text with associated information attached, with a dimension of 384×1.

[0029] Step 6: Use the three triplet loss functions to calculate the final loss function between the two modal data of image and text, and use it to train the network, including: The three triplet loss functions obtained previously are Lg, Ll, Ln The weighted sum of is used as the final loss function between the two modal data of image and text La , La = x× Lg + y × Ll + z × Ln , x, y, z are trainable parameters; Input a mini-batch of data, and use the Adam optimizer to optimize the model parameters based on the loss calculated by the final loss function; The learning rate during training is set to 1.0x10 -5 , the number of iterations is 120, and the exponential moving average (EMA) is used to smooth the parameter updates during the training process; In the first ten rounds of training, the learning rate is increased from 1.0x10 -6 Increases linearly to 1.0x10 -5 , from the eleventh to the twentieth round, the learning rate is kept at 1.0x10 -5 , the learning rate from the 21st to the 85th round is adjusted to 1.0x10 -6 , the learning rate for the last thirty-five rounds was adjusted to 1.0x10 -7 ; After training the data set once, recalculate the three triplet losses and retrain until the set number of iterations is reached; The model parameters are updated using the gradient update method. After the training is completed, the parameters of the entire network are saved.

[0030] Step S7, calculating the similarity between the search text and the images in the image library and sorting them to obtain search results, including: Input the search text into the trained network, calculate the similarity between the search text and the images in the image library, and obtain the sorted images; The first result whose similarity exceeds the threshold is returned as the search result.

[0031] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the above embodiments describe the present invention in detail, those skilled in the art should understand that by modifying the technical solutions described in the above embodiments, or replacing some or all of the technical features therein by equivalents, or modifying the order of each step, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention. The values ​​of various thresholds and ranges of the present invention may also change due to different specific circumstances.

Claims

1. A text pedestrian retrieval method based on automatic text alignment, characterized in that: The following steps are involved: S1, uses the BERT model and the ResNeSt model to extract text global features and image global features respectively; S2, perform pooling operations on the global features of the text and the global features of the image to obtain the feature vector and calculate the triplet loss LG ; S3, divide the image into blocks according to human body parts and calculate the local features of the image; use the word attention module calculation method to calculate the text weighted features corresponding to each block of the image; S4, perform pooling operation on text weighted features and image local features to obtain feature vectors and calculate triplet loss Ll ; S5: Calculate the interaction strength between each local feature and the other local features, obtain the image local feature vector and text local feature vector with associated information, and calculate the triplet loss Ln; S6, using the calculated three triplet loss functions to calculate the final loss function between the two modal data of image and text, and using it to train the network; S7, calculate the similarity between the search text and the images in the image library and sort them to obtain the search results.

2. The method according to claim 1, characterized in that Step S1, using the BERT model and the ResNeSt model to extract text global features and image global features respectively, including: Use BERT as the backbone network and take text as input to obtain the global features of the text; ResNeSt is used as the backbone network, and the image is used as input to obtain the global features of the image.

3. The method according to claim 1, characterized in that Step S2: Pool the global features of the text and the global features of the image to obtain feature vectors and calculate the triplet loss. LG ,include: Perform row-by-row average pooling on the global features of the text to obtain the global feature vector E of the text; Perform global average pooling on the global features of the image to obtain the global feature vector F of the image; Calculate the distance between the global feature vector of the text and the global feature vector of the image, and calculate the triplet loss Lg.

4. The method according to claim 1, characterized in that: Step S3, dividing the image into blocks according to human body parts and calculating local features of the image; Use the word attention module calculation method to calculate the text weighted features corresponding to each image, including: The image is divided into k non-overlapping blocks of human body parts by body part division, and features are extracted respectively to obtain k local features of the image; The word attention module calculation method is used to calculate the weighted features of the text global feature vector E describing each human body part block, and k text weighted features corresponding to the human body part blocks are obtained respectively.

5. The method according to claim 1, characterized in that Step S4: Pooling the text weighted features and the image local features to obtain feature vectors and calculate the triplet loss. Ll ,include: Perform row-by-row average pooling on the text weighted features of the image describing the kth part of the human body to obtain k vectors, and concatenate the k vectors to obtain a text weighted feature vector, i.e., a text local feature vector; Perform global average pooling on each local feature of the image to obtain k vectors; concatenate the k vectors to obtain the local feature vector of the image; Calculate the distance between the text weighted feature vector and the image local feature vector, and calculate the triplet loss Ll。 6. The method according to claim 1, characterized in that Step S5, calculate the interaction strength between each local feature and the other local features, obtain the image local feature vector and text local feature vector with associated information attached, and calculate the triplet loss Ln ,include: Using a local feature interaction module calculation method, k image local feature vectors with additional associated information and k text local feature vectors with additional associated information are calculated; The k calculated image local feature vectors of the additional associated information are concatenated to obtain a final image local feature vector of the additional associated information; The k calculated local feature vectors of the text of the additional associated information are concatenated to obtain the final local feature vector of the text of the additional associated information; Calculate the distance between the final image local feature vector of the additional associated information and the final text local feature vector of the additional associated information, and calculate the triplet loss Ln .

7. The method according to claim 1, characterized in that Step S6, using the calculated three triplet loss functions to calculate the final loss function between the two modal data of image and text, and using it to train the network, including: The three triplet loss functions calculated Lg, Ll, Ln The weighted sum of is used as the final loss function between the two modal data of image and text; Input a mini-batch of data, and use the Adam optimizer to optimize the model parameters based on the loss calculated by the final loss function; After training the data set once, recalculate the three triplet losses and perform the next training until the set number of iterations is reached; The model parameters are updated using the gradient update method. After the training is completed, the parameters of the entire network are saved.

8. The method according to claim 1, characterized in that: Step S7, calculating the similarity between the search text and the images in the image library and sorting them to obtain search results, including: Input the search text into the trained network, calculate the similarity between the search text and the images in the image library, and obtain the sorted images; The first result whose similarity exceeds the threshold is returned as the search result.

Images