Text-based pedestrian image retrieval method and system fusing pose information

Abstract

This invention relates to a text-based pedestrian image retrieval method and system that integrates pose information. The method includes: acquiring text and images, and dividing the text and images into several words and several image blocks; extracting global features and features of each word from the text, as well as global features, features of each image block, and features of several human key points from the images; aggregating the features of each word and the features of each image block to obtain a set of local text features and a set of local image features; fusing the set of local image features with the features of several human key points to obtain a final set of local image features; training the model used in the preceding steps; using the trained model to extract global and local features from the text and image datasets to be retrieved, respectively, calculating similarity, and selecting the images with the highest similarity as the retrieval results. This invention can improve the accuracy of retrieval.

Description

Technical Field

[0001] This invention relates to the field of pedestrian image retrieval technology, and in particular to a text-based pedestrian image retrieval method and system that integrates posture information. Background Technology

[0002] Text-based pedestrian image retrieval is a cross-modal retrieval task, requiring the retrieval of the most matching image from a large-scale pedestrian image database given a text description. This technology is widely used in intelligent video surveillance, security, and criminal investigation, and is currently a research hotspot in natural language processing and computer vision. Common text-based pedestrian image retrieval methods incorporating pose information can be divided into global matching methods and local matching methods. Global matching methods extract global representations of samples from both image and text modalities and design appropriate objective functions to explore the shared latent embedding space, where the matching score of image-text pairs can be directly calculated. Although global matching methods extract semantically aligned global representations from images and text respectively, resulting in faster computation, they inevitably introduce background noise and fail to effectively mine fine-grained alignment information, making it difficult to achieve accurate retrieval results. Given the poor retrieval performance of global matching methods, many researchers have focused on local matching methods. Local matching methods first obtain local units in images and text through pre-defined rules, and then explicitly extract local features from these local units based on these rules. Specifically, some methods obtain local units by simply dividing an image or its feature map into stripes or blocks and text into words, and then compute local features by directly extracting feature representations from these units. However, this approach introduces background noise into the features, which severely affects the subsequent alignment stage and leads to suboptimal retrieval results. Next, a cross-modal attention mechanism is used to learn the alignment relationship between the image and text. Local matching methods effectively capture fine-grained associations between images and text, and their retrieval accuracy is significantly higher than that of global matching methods.

[0003] Existing local matching methods horizontally divide a pedestrian image into multiple parts, segment the text into individual words, extract multiple local features for each word, and then learn the local alignment relationship between the image and text through a cross-modal attention mechanism. This can significantly improve the model's accuracy; however, this local segmentation introduces background noise and is difficult to handle situations where pedestrians are occluded in the image. Furthermore, the computational cost of the cross-modal attention mechanism is high, making it difficult to apply to real-world scenarios. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is to overcome the problems of background noise and pedestrian occlusion when extracting features by local matching methods in the prior art, and to overcome the problem of large computational overhead in cross-modal image learning.

[0005] To address the aforementioned technical problems, this invention provides a text-based pedestrian image retrieval method that integrates pose information, comprising:

[0006] Step S1: Obtain text and images, and divide the text into several words and the images into several image blocks;

[0007] Step S2: Extract the global features and features of each word of the text, and simultaneously extract the global features and features of each image block of the image, as well as extract several human key point features in the image;

[0008] Step S3: Aggregate the features of each word to obtain a text local feature set, and simultaneously aggregate the features of each image block to obtain an image local feature set;

[0009] Step S4: Fuse the image local feature set with the several human body key point features to obtain the final image local feature set;

[0010] Step S5: Add cross-modal local alignment loss between the final image local feature set and the text local feature set, and between the global features of the image and the global features of the text, to train the model used in steps S2-S4, and obtain the trained model;

[0011] Step S6: Obtain the text to be retrieved, use the trained model to extract the global and local features of the text to be retrieved and the image dataset respectively, calculate the similarity between each image in the image dataset and the text to be retrieved, and select the images with the highest similarity as the retrieval results of the text to be retrieved.

[0012] In one embodiment of the present invention, the method for extracting human body key point features in step S2 includes:

[0013] The image is input into the human pose estimation model to obtain 17 heat maps, and the point with the maximum response value is extracted from each heat map as a human key point.

[0014] Each heatmap is transformed into a d-dimensional heatmap vector H∈R through a fully connected layer. 17*d Then, each heatmap vector containing human key points is multiplied element-wise with the global features of the image to obtain the human key point features P∈R. 17 *d , where R represents the vector space.

[0015] In one embodiment of the present invention, step S3 involves aggregating the features of each image patch to obtain a local image feature set. The method includes:

[0016] The features of each image block are input into a first feature aggregation module, wherein the first feature aggregation module is constructed using an attention model with multiple head attention blocks;

[0017] Based on the features of each image patch, the query matrix Q of the i-th head attention block is calculated through linear projection. i Key matrix K i Sum matrix V i ;

[0018] The query matrix Q is then processed using a self-attention mechanism. i Key matrix K i Sum matrix V i We perform weighted summation to obtain the set of local features of the image. in, Let represent the local feature of the i-th image, K represent the number of head attention blocks, d represent the dimension, and R represent the vector space.

[0019] In one embodiment of the present invention, the query matrix Q is processed through a self-attention mechanism. i Key matrix K i Sum matrix V i The weighted summation formula is:

[0020]

[0021] Among them, E i Let E represent all output vectors of the i-th attention head and E i satisfy Softmax(·) represents the normalized exponential function, and d represents the dimension.

[0022] In one embodiment of the present invention, step S3 uses a second feature aggregation module to aggregate the features of each word to obtain a local feature set of the text, wherein the second feature aggregation module and the first feature aggregation module share parameters.

[0023] In one embodiment of the present invention, step S4 involves fusing the image local feature set with the plurality of human body key point features to obtain a final image local feature set. The method includes:

[0024] For each local feature in the image local feature set, find the human key point feature with the closest Euclidean distance among the several human key point features.

[0025] Calculate the outer product of each local feature with the human keypoint feature that is closest to it in Euclidean distance, and then perform average pooling on the calculated outer product to obtain the final set of local features of the image.

[0026] In one embodiment of the present invention, step S5 trains the model using a total loss, which includes global alignment loss, cross-modal local alignment loss, and diversity loss.

[0027] The formula for the total loss is:

[0028] L = L g +L part +0.2*L div

[0029] Where L is the total loss, L g For triplet loss, L part For cross-modal local alignment loss, L div For the loss of diversity;

[0030] The global alignment loss is a triplet loss, and the formula for the triplet loss is:

[0031] L g =max[0,α-(s(e g ,t g+ ),s(e g ,t g- ))]+max[0,α-(s(e g+ ,t g ),s(e g- ,t g ))]

[0032] Where s(·) is used to calculate the Euclidean distance, and t g+ and t g- The global features e of the image are respectively represented by g Positive and negative samples, e g+ and e g- The global features t of the text are represented respectively. g The positive and negative samples, where α represents the threshold set for the triplet loss;

[0033] The cross-modal local alignment loss includes cross-modal projection matching loss and cross-modal projection classification loss, and the formula is:

[0034]

[0035] Among them, L cmpm L represents the modal projection matching loss. cmpc This represents the cross-modal projection classification loss. This represents the features in the final image's local feature set. This represents the features in the local feature set of the text, k represents the k-th head attention block, and K represents the number of head attention blocks;

[0036] The formula for the diversity loss is:

[0037]

[0038] Where sim(·) is used to calculate the cosine similarity. This represents the i-th local feature of the image. This represents the j-th local feature of the image. This represents the i-th local feature of the text. Let K represent the j-th local feature of the text, and K represent the number of local features.

[0039] To address the aforementioned technical problems, this invention provides a text-based pedestrian image retrieval system that integrates pose information, comprising:

[0040] Acquisition and Segmentation Module: Used to acquire text and images, and to segment the text into several words and the images into several image blocks;

[0041] Feature extraction module: used to extract the global features and features of each word of the text, as well as the global features and features of each image block of the image, and extract several human key point features in the image;

[0042] Aggregation module: used to aggregate the features of each word to obtain a text local feature set, and at the same time aggregate the features of each image block to obtain an image local feature set;

[0043] Fusion module: used to fuse the image local feature set with the several human body key point features to obtain the final image local feature set;

[0044] Training module: used to train the model used by the feature extraction module, aggregation module and fusion module by adding cross-modal local alignment loss between the final image local feature set and the text local feature set, and between the global features of the image and the global features of the text, to obtain a trained model;

[0045] The retrieval module is used to obtain the text to be retrieved. It uses a trained model to extract global and local features from the text and image datasets, calculates the similarity between each image in the image dataset and the text to be retrieved, and selects the images with the highest similarity as the retrieval results for the text to be retrieved.

[0046] To address the aforementioned technical problems, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the text-based pedestrian image retrieval method that integrates pose information as described above.

[0047] To address the aforementioned technical problems, the present invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the text-based pedestrian image retrieval method that integrates pose information as described above.

[0048] The technical solution of the present invention has the following advantages over the prior art:

[0049] This invention integrates pose information (i.e., human key point features) into the extracted image features, which can more effectively extract local information of pedestrian images and discard background information, thereby promoting cross-modal local semantic alignment and making the image retrieval results of subsequent basic text more accurate.

[0050] This invention employs a visual-text joint embedding approach consistent with global matching to learn cross-modal local alignment, which greatly reduces the computational cost in practical applications;

[0051] This invention combines the advantages of both global matching and local matching methods, and has higher retrieval accuracy than traditional global matching and lower computational cost than traditional local matching methods. Attached Figure Description

[0052] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0053] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0055] Example 1

[0056] Reference Figure 1 As shown, this invention relates to a text-based pedestrian image retrieval method that integrates pose information, comprising:

[0057] Step S1: Obtain text and images, and divide the text into several words and the images into several image blocks;

[0058] Step S2: Extract the global features and features of each word of the text using the text extraction model, extract the global features and features of each image block of the image using the image extraction model, and extract several human key point features (i.e., pose information) of the image using the human pose estimation model.

[0059] Step S3: The features of each word are aggregated by the first feature aggregation module to obtain a local text feature set, and the features of each image block are aggregated by the second feature aggregation module to obtain a local image feature set.

[0060] Step S4: The image local feature set is fused with the several human body key point features using a fusion model to obtain the final image local feature set;

[0061] Step S5: Add cross-modal local alignment loss between the final image local feature set and the text local feature set, and between the global features of the image and the global features of the text, to train the model used in steps S2-S4, and obtain the trained model;

[0062] Step S6: Use the trained model to extract the global and local features of the text and image datasets to be retrieved, calculate the similarity between each image in the image dataset and the text to be retrieved, and select the images with the highest similarity as the retrieval results for the text to be retrieved.

[0063] This invention integrates pose information (i.e., human key point features) into the extracted image features, which can more effectively extract local information of pedestrian images and discard background information, thereby promoting cross-modal local semantic alignment and making the image retrieval results of subsequent basic text more accurate.

[0064] The following describes the invention in detail using the text-based pedestrian image retrieval dataset CUHK-PEDES as an example:

[0065] Step (1): Obtain the training set, unify the number of words in the text in the training set to M by filling and truncating, and divide the images in the training set into N small image blocks.

[0066]

[0067] Step (3) Constructs a first feature aggregation module based on a multi-head self-attention mechanism. This module is built using an attention model with multiple head attention blocks. The multi-head attention model in the first feature aggregation module takes the image features E of each image as input (note that the input can be E or e1, e2, ... e, ...). NThis process yields K image patch embeddings for each image (K is the number of head attention blocks in the multi-head attention model, and each head attention block focuses on different semantics). Each image patch embedding is a weighted sum of local image features. Specifically, given the input image features E, this embodiment first calculates three vectors in the i-th head attention block (i = 1, 2, ..., K) through linear projection: the query matrix Q. i Key matrix K i Sum matrix V i The formula is shown below:

[0068] Q i =EW i Q

[0069] K i =EW i K

[0070] V i =EW i V

[0071] Among them, W i Q W i K W i V Let the first, second, and third weight matrices of the i-th attention model be used, and then the image patch vectors are weighted and summed using the self-attention mechanism of the following formula:

[0072]

[0073] Among them, E i This represents all output vectors of the i-th attention head (Note: Let be one of the output vectors, representing a local feature obtained by an attention head. Let represent a local semantic feature obtained by the i-th head self-attention module, and Softmax(·) denotes the normalization exponential function. For text, this embodiment uses a second feature aggregation module that shares parameters with the first feature aggregation module. This yields K sets of local text features. and K sets of local image features

[0074] Step (4): Keypoint features can accurately represent local information of the human body and effectively solve the problem of human body occlusion in images. Therefore, this embodiment uses a fusion model to fuse pose information (i.e., human body keypoint features) with a set of K local image features. Specifically, for each local feature e of the image... iWe selected the feature with high confidence and closest Euclidean distance from 17 human keypoint features. The calculation formula is as follows:

[0075]

[0076] Where k represents the feature index with the nearest Euclidean distance, and arg max represents the indexing operator. P represents a feature in the set of local features of an image. j Let represent the key features of the j-th person, <,> represent calculating the Euclidean distance, and ||·|| represent the L2 norm.

[0077] Next, keypoint features and local image features are fused using bilinear pooling. The calculation formula is as follows:

[0078] First, we obtain the outer product of the two features:

[0079]

[0080] Then, average pooling is performed on the matrix to obtain the K final image local features after fusion.

[0081] Step (5): In this embodiment, when training the model composed of the above sub-models, a cross-modal local alignment loss is added to the total loss function. Semantic alignment between the image and text is achieved by using this cross-modal local alignment loss. The global alignment loss in the total loss function is the triplet loss, calculated as follows:

[0082] L g =max[0,α-(s(e g ,t g+ ),s(e g ,t g- ))]+max[0,α-(s(e g+ ,t g ),s(e g- ,t g ))]

[0083] Among them, L g The loss is the triplet loss, s(·) is the Euclidean distance calculation, and t g+ and t g- The global features e of the image are respectively represented by g Positive and negative samples of image features; similarly, e g+ and e g- The global features t of the text are represented respectively. g Positive and negative samples; α represents the threshold set for the triplet loss.

[0084] Cross-modal local alignment loss includes CMPM loss and CMPC loss, and its calculation formula is as follows:

[0085]

[0086] Among them, L cmpm L represents the modal projection matching loss. cmpc This represents the cross-modal projection classification loss. This represents the features in the final image's local feature set. The text represents the features in the local feature set, k represents the k-th head attention block, and K represents the number of head attention blocks.

[0087] L part The purpose is to learn text-based pedestrian image retrieval tasks by mapping the feature vectors of images and text into the same space. It promotes similarity between text and images of pedestrians with different identities by minimizing the KL divergence, while maximizing the similarity between text and images of pedestrians with the same identity. In this way, CMPM makes text and image features from the same identity more similar in the feature space, while text and image features from different identities more dispersed.

[0088] The CMPC loss is a classification loss function that distinguishes pedestrians of different identities by promoting greater similarity in the feature representations of pedestrian samples from the same identity. Simultaneously, it can also distinguish the feature representations of other samples from pedestrians of different identities. Thus, CMPC can learn how to differentiate pedestrians of different identities and make pedestrian samples from the same identity more similar in text-based pedestrian image retrieval.

[0089] Furthermore, this embodiment incorporates a diversity loss mechanism to prevent the local features learned by multiple head attention blocks from becoming identical. The loss function is as follows:

[0090]

[0091] Where sim(·) is used to calculate the cosine similarity. This represents the i-th local feature in the final image's local feature set. This represents the j-th local feature in the final image local feature set. This represents the i-th local feature in the set of local features of the text. Let represent the j-th local feature in the set of local features of the text, and K represent the number of local features.

[0092] The overall loss function is as follows:

[0093] L = L g +L part+0.2*L div

[0094] Step (6): To verify the accuracy of the method invented in this embodiment, this embodiment is tested on a test set, which includes a query text set and a gallery image set. The goal of this embodiment is to find images of the same pedestrian as the one in the query text set, given a query text. The testing method is to extract the features of each sample from both the query text and all gallery images using the model trained above, then compare the similarity between the features of the query text and the features of all gallery images, and select the gallery images with the highest similarity as the final retrieval results.

[0095] Example 2

[0096] This embodiment provides a text-based pedestrian image retrieval system that integrates pose information, including:

[0097] Acquisition and Segmentation Module: Used to acquire text and images, and to segment the text into several words and the images into several image blocks;

[0098] Feature extraction module: used to extract the global features and features of each word of the text, as well as the global features and features of each image block of the image, and extract several human key point features in the image;

[0099] Aggregation module: used to aggregate the features of each word to obtain a text local feature set, and at the same time aggregate the features of each image block to obtain an image local feature set;

[0100] Fusion module: used to fuse the image local feature set with the several human body key point features to obtain the final image local feature set;

[0101] Training module: used to train the model used by the feature extraction module, aggregation module and fusion module by adding cross-modal local alignment loss between the final image local feature set and the text local feature set, and between the global features of the image and the global features of the text, to obtain a trained model;

[0102] The retrieval module is used to obtain the text to be retrieved. It uses a trained model to extract global and local features from the text and image datasets, calculates the similarity between each image in the image dataset and the text to be retrieved, and selects the images with the highest similarity as the retrieval results for the text to be retrieved.

[0103] Example 3

[0104] This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the text-based pedestrian image retrieval method that fuses posture information as described in Embodiment 1.

[0105] Example 4

[0106] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the text-based pedestrian image retrieval method that integrates posture information as described in Embodiment 1.

[0107] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0108] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0109] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0110] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0111] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0112] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A text-based pedestrian image retrieval method fusing pose information, characterized in that: include: Step S1: Obtain text and images, and divide the text into several words and the images into several image blocks; Step S2: Extract the global features and features of each word of the text, and simultaneously extract the global features and features of each image block of the image, as well as extract several human key point features in the image; Step S3: Aggregate the features of each word to obtain a text local feature set, and simultaneously aggregate the features of each image block to obtain an image local feature set; Step S4: Fuse the image local feature set with the several human body key point features to obtain the final image local feature set; Step S5: Add cross-modal local alignment loss between the final image local feature set and the text local feature set, and between the global features of the image and the global features of the text, to train the model used in steps S2-S4, and obtain the trained model; Step S6: Use the trained model to extract the global and local features of the text and image datasets to be retrieved, calculate the similarity between each image in the image dataset and the text to be retrieved, and select the images with the highest similarity as the retrieval results for the text to be retrieved. 2.The text-based pedestrian image retrieval method with fused pose information according to claim 1, wherein: The method for extracting human key point features from the image in step S2 includes: The image is input into the human pose estimation model to obtain 17 heat maps, and the point with the maximum response value is extracted from each heat map as a human key point. Each of the heatmaps is transformed through a fully connected layer. 3D heatmap vector Then, each heatmap vector containing human body key points is multiplied element-wise with the global features of the image to obtain the human body key point features. ,in, Represents a vector space. 3.The text-based pedestrian image retrieval method with fused pose information according to claim 1, wherein: In step S3, the features of each image patch are aggregated to obtain a local image feature set. The method includes: The features of each image block are input into a first feature aggregation module, wherein the first feature aggregation module is constructed using an attention model with multiple head attention blocks; Based on the features of each image patch, the first image patch is calculated using linear projection. The query matrix of each header attention block Key matrix Sum matrix ; The query matrix is ​​processed using a self-attention mechanism. Key matrix Sum matrix We perform weighted summation to obtain the set of local features of the image. ,in, Indicates the first Local features of an image Indicates the number of head attention blocks. Representing dimension, Represents a vector space.

4. The text-based pedestrian image retrieval method based on fused pose information according to claim 3, characterized in that: The query matrix is ​​processed through a self-attention mechanism. Key matrix Sum matrix The weighted summation formula is: ； wherein, denotes the output vector of the i-th attention head and denotes all output vectors of the i-th satisfies , denotes the normalized exponential function, denotes the dimension. 5.The text-based pedestrian image retrieval method with fused pose information according to claim 3, characterized in that: In step S3, the second feature aggregation module aggregates the features of each word to obtain a local feature set of the text, wherein the second feature aggregation module and the first feature aggregation module share parameters. 6.The text-based pedestrian image retrieval method with fused pose information according to claim 1, wherein: In step S4, the image local feature set is fused with the several human body key point features to obtain the final image local feature set. The method includes: For each local feature in the image local feature set, find the human key point feature with the closest Euclidean distance among the several human key point features. Calculate the outer product of each local feature with the human keypoint feature that is closest to it in Euclidean distance, and then perform average pooling on the calculated outer product to obtain the final set of local features of the image. 7.The text-based pedestrian image retrieval method with fused pose information according to claim 1, wherein: Step S5 trains the model using a total loss, which includes global alignment loss, cross-modal local alignment loss, and diversity loss. The formula for the total loss is: ； wherein, is the total loss, is the triplet loss, is the cross-modal local alignment loss, is the diversity loss; The global alignment loss is a triplet loss, and the formula for the triplet loss is: ； in, To calculate the Euclidean distance, and These represent the global features of the image. Positive and negative samples, and These represent the global features of the text. Positive and negative samples, This indicates the threshold value set for the triplet loss; The cross-modal local alignment loss includes cross-modal projection matching loss and cross-modal projection classification loss, and the formula is: ； in, This represents the modal projection matching loss. This represents the cross-modal projection classification loss. This represents the features in the final image's local feature set. Features represent the features in the local feature set of the text. Indicates the first A head attention block, Indicates the number of head attention blocks; The formula for the diversity loss is: ； in, To calculate cosine similarity, The image represents the first Local features, The image represents the first Local features, The first character representing the text Local features, The first character representing the text Local features, This indicates the number of head attention blocks.

8. A text-based pedestrian image retrieval system incorporating posture information, characterized in that: include: Acquisition and Segmentation Module: Used to acquire text and images, and to segment the text into several words and the images into several image blocks; Feature extraction module: used to extract the global features and features of each word of the text, as well as the global features and features of each image block of the image, and extract several human key point features in the image; Aggregation module: used to aggregate the features of each word to obtain a text local feature set, and at the same time aggregate the features of each image block to obtain an image local feature set; Fusion module: used to fuse the image local feature set with the several human body key point features to obtain the final image local feature set; Training module: used to train the model used by the feature extraction module, aggregation module and fusion module by adding cross-modal local alignment loss between the final image local feature set and the text local feature set, and between the global features of the image and the global features of the text, to obtain a trained model; The retrieval module is used to obtain the text to be retrieved. It uses a trained model to extract global and local features from the text and image datasets, calculates the similarity between each image in the image dataset and the text to be retrieved, and selects the images with the highest similarity as the retrieval results for the text to be retrieved.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the text-based pedestrian image retrieval method that integrates pose information as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the text-based pedestrian image retrieval method that integrates pose information as described in any one of claims 1 to 7.

Images