Text-to-image pedestrian retrieval method and device based on mask denoising and medium

Abstract

The application provides a text-to-image pedestrian retrieval method and device based on mask denoising and a medium. The method comprises: performing mask and same kind word random replacement on entity words and attribute words respectively according to a set probability to generate training text after mask replacement; inputting a text feature vector and an image feature vector into a cross-modal interaction encoder to obtain a fusion feature representation; predicting original words at a masked position based on the fusion feature representation to calculate a mask prediction loss; calculating a text-image contrast learning loss based on a similarity relationship between the text feature vector and the image feature vector; extracting features of a query text and a to-be-retrieved pedestrian image library by using a pedestrian retrieval model, calculating the similarity of the query text features and each to-be-retrieved pedestrian image feature, and generating a ranking result to output a target pedestrian image matched with the query text. The application can improve the robustness of visual semantic alignment in a noise scene and significantly improve the pedestrian retrieval accuracy.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a text-to-image pedestrian retrieval method, apparatus and medium based on mask denoising. Background Technology

[0002] Text-to-image person retrieval (TIPR) is a cross-modal retrieval technology that uses natural language descriptions to locate target pedestrians in a large-scale pedestrian image database. Compared to person re-identification (Re-ID), which relies solely on image queries, TIPR allows users to directly input free and flexible text descriptions, making it highly valuable in scenarios such as public safety, smart city surveillance, and criminal investigation evidence collection.

[0003] To achieve accurate alignment between visual and linguistic modalities, existing TIPR research mainly focuses on feature extraction and matching strategies, which can be summarized into two main directions:

[0004] Global matching method: Calculates similarity between a unified visual global vector and a text global vector, and optimizes it with contrastive learning or metric learning loss. The advantage is that the network structure is simple and the inference speed is fast. The disadvantage is that it cannot characterize local semantics such as "clothing color" and "personal items" in a fine-grained manner.

[0005] Local matching method: Further subdivide image regions and text phrases, and use explicit region detection or implicit attention mechanism to achieve word-block level alignment, which can improve fine-grained retrieval performance.

[0006] Existing technologies still suffer from the following problems: 1) Reliance on the assumption of perfect image-text matching: Most TIPR methods assume that the image-text matching in the training set is completely correct, without considering incorrect matching caused by pose occlusion, ambient lighting, or incomplete descriptions in real-world scenarios. Once this assumption is broken, the model is prone to learning noisy samples as positive examples, resulting in a significant decrease in retrieval accuracy. 2) Limited transfer effect of single-modal noise resistance mechanisms: Existing sample selection and robust loss functions are mainly designed for image classification or single-modal tasks. Directly applying them to cross-modal TIPR makes it difficult to accurately identify and eliminate incorrect text-image correspondences, resulting in limited improvement in robustness. 3) Difficulty in balancing fine-grained local alignment and computational overhead: Explicit local matching requires calling external detectors, while implicit local matching, although saving extra modules, is more sensitive to noise. Both lack stable solutions that balance accuracy and real-time performance in large-scale retrieval scenarios. Summary of the Invention

[0007] In view of this, embodiments of this application provide a text-to-image pedestrian retrieval method, apparatus, and medium based on mask denoising to solve the problems of cross-modal alignment distortion and decreased retrieval accuracy caused by noise interference from incorrect text-image pairing in the prior art.

[0008] A first aspect of this application provides a text-to-image pedestrian retrieval method based on mask denoising, comprising: acquiring cross-modal training data containing pedestrian images and corresponding text descriptions; performing semantic analysis on the text descriptions to identify and label entity words and attribute words; performing masking and random replacement of similar words on entity words and attribute words according to a set probability to generate masked training text; inputting the masked training text and corresponding pedestrian images into a text encoder and an image encoder respectively to obtain text feature vectors and image feature vectors; and inputting the text feature vectors and image feature vectors into a cross-modal interactive encoder to obtain a fused feature representation. The algorithm predicts the original words at the masked positions based on the fused feature representation and calculates the mask prediction loss. It also calculates the image-text comparison learning loss based on the similarity relationship between the text feature vector and the image feature vector. The mask prediction loss and the image-text comparison learning loss are weighted and summed according to preset weights to jointly optimize the parameters of the text encoder, image encoder, and cross-modal interactive encoder, thus obtaining the pedestrian retrieval model. The pedestrian retrieval model is used to extract features from the query text and the pedestrian image database to be retrieved, calculate the similarity between the query text features and the features of each pedestrian image to be retrieved, and generate a ranking result to output the target pedestrian image that matches the query text.

[0009] A second aspect of this application provides a text-to-image pedestrian retrieval device based on mask denoising, comprising: a recognition module, configured to acquire cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words; a masking module, configured to perform masking and random replacement of similar words on entity words and attribute words according to a set probability, generating masked training text; an encoding module, configured to input the masked training text and the corresponding pedestrian image into a text encoder and an image encoder respectively, to obtain text feature vectors and image feature vectors; and a fusion module, configured to input the text feature vectors and image feature vectors into a cross-modal interactive encoder to obtain fused feature representations. The system comprises the following modules: a prediction module for predicting the original words at the masked positions based on the fused feature representation and calculating the mask prediction loss; a calculation module for calculating the image-text comparison learning loss based on the similarity relationship between the text feature vector and the image feature vector; a joint module for weighted summation of the mask prediction loss and the image-text comparison learning loss according to preset weights to jointly optimize the parameters of the text encoder, image encoder, and cross-modal interactive encoder to obtain the pedestrian retrieval model; and an output module for extracting features from the query text and the pedestrian image database to be retrieved using the pedestrian retrieval model, calculating the similarity between the query text features and the features of each pedestrian image to be retrieved, generating a ranking result, and outputting the target pedestrian image that matches the query text.

[0010] A third aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0011] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0012] By acquiring cross-modal training data containing pedestrian images and corresponding text descriptions, semantic analysis is performed on the text descriptions to identify and label entity words and attribute words. Masking and random replacement of similar words are performed on entity words and attribute words according to set probabilities to generate masked training text. The masked training text and corresponding pedestrian images are input into the text encoder and image encoder, respectively, to obtain text feature vectors and image feature vectors. The text feature vectors and image feature vectors are input into the cross-modal interactive encoder to obtain a fused feature representation. Based on the fused feature representation, the original words at the masked positions are predicted, and the masking prediction loss is calculated. The image-text comparison learning loss is calculated based on the similarity relationship between the text feature vector and the image feature vector. The masking prediction loss and the image-text comparison learning loss are weighted and summed according to preset weights to jointly optimize the parameters of the text encoder, image encoder, and cross-modal interactive encoder, resulting in a pedestrian retrieval model. The pedestrian retrieval model is used to extract features from the query text and the pedestrian image database to be retrieved, respectively. The similarity between the query text features and the features of each pedestrian image to be retrieved is calculated, and a ranking result is generated to output the target pedestrian image matching the query text. This application can improve the robustness of visual semantic alignment in noisy scenes and significantly improve the accuracy of pedestrian retrieval. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart illustrating the text-to-image pedestrian retrieval method based on mask denoising provided in this application embodiment;

[0015] Figure 2 This is a schematic diagram of the structure of the text-to-image pedestrian retrieval device based on mask denoising provided in this application embodiment;

[0016] Figure 3 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0018] Text-to-image person retrieval (TIPR) aims to identify target pedestrians using text descriptions, finding the most relevant pedestrian images from a pedestrian image database. This task utilizes a free and flexible text query method, which is more universal than the structured image query method used in person re-identification (Re-ID) tasks, and has significant application potential in the field of public safety. Existing technologies mainly employ global and local alignment methods of visual and textual features, combined with contrastive learning and local region matching to improve recognition accuracy. However, existing methods generally assume that the image-text pairings in the training data are perfectly correct, making it difficult to handle the problem of incorrect image-text correspondences caused by pose, angle, lighting, or inaccurate descriptions in real-world scenarios. This makes the model susceptible to noise interference, reducing retrieval performance. Therefore, how to achieve robust and efficient TIPR under imperfect image-text pairings has become an important research direction in this field.

[0019] Current common solutions to the TIPR task primarily focus on designing effective alignment strategies to achieve good matching between image and text modalities. These are categorized into global matching and local matching. Global matching methods directly align global text and visual representations by designing a reasonable cross-modal matching loss function, but this method ignores fine-grained information. Local matching methods align fine-grained image and text information, enhancing cross-modal alignment. Local matching is further divided into explicit local matching and implicit local matching. The former relies on external techniques and cross-modal information, increasing the computational complexity of the inference stage; the latter does not rely on external explicit dependencies, significantly reducing additional computational overhead.

[0020] Numerous methods have been proposed to achieve noise-robust learning, categorized into sample selection methods and robust loss function methods. Sample selection methods leverage the memory effect of deep neural networks to progressively distinguish noisy data, thereby enabling the model to focus more on clean data and less on noisy data during training. Robust loss functions aim to design loss functions with noise tolerance to improve the model's robustness against noise during training.

[0021] Existing technical solutions have the following drawbacks:

[0022] 1. Existing TIPR methods generally assume that the training data is accurately matched with images and text. However, they are unable to cope with the problem of incorrect image and text matching (noise correspondence) caused by incomplete descriptions, occlusion, etc. in real-world scenarios, which leads to a significant decrease in retrieval accuracy in real complex environments.

[0023] 2. Existing noise robustness methods are mostly focused on classification tasks or single-modal domains. When directly applied to TIPR, they are difficult to effectively identify and eliminate erroneous image-text correspondences, resulting in insufficient robustness. Furthermore, they lack stable and efficient processing mechanisms in large-scale image-text matching retrieval scenarios.

[0024] In view of the problems existing in the prior art, this application proposes a text-to-image person retrieval method based on mask denoising to solve the noise correspondence problem in text-to-image person retrieval tasks. Compared with the traditional TIPR task, which is easily affected by incorrect image-text matching in real-world environments, leading to a decrease in retrieval accuracy, this application mainly solves the following problems:

[0025] 1. During the training phase, entity words and attribute words in the text description are masked and replaced with a certain probability. Combined with a cross-modal interactive encoder and mask prediction loss, the model can still learn robust and accurate visual-semantic alignment relationships even when there is text description noise and incorrect image-text pairing.

[0026] 2. During training, image-text contrast learning loss is introduced and jointly optimized with mask prediction loss. This enables the model to not only rely on global features when constructing the shared feature space, but also to make full use of cross-modal matching information of local entities and attributes, thereby enhancing the robustness and generalization ability of text-to-image pedestrian retrieval tasks.

[0027] The technical solution of this application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0028] Figure 1 This is a schematic flowchart of the text-to-image pedestrian retrieval method based on mask denoising provided in this application embodiment. Figure 1 As shown, the text-to-image pedestrian retrieval method based on mask denoising can specifically include:

[0029] S101, acquire cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words;

[0030] S102, Perform masking and random replacement of similar words on entity words and attribute words according to the set probability to generate training text with masking and replacement;

[0031] S103, input the masked training text and the corresponding pedestrian image into the text encoder and image encoder respectively to obtain the text feature vector and the image feature vector;

[0032] S104, Input the text feature vector and image feature vector into the cross-modal interactive encoder to obtain the fused feature representation;

[0033] S105, Based on the fused feature representation, predict the original words at the masked positions and calculate the mask prediction loss;

[0034] S106, Calculate the image-text comparison learning loss based on the similarity relationship between text feature vectors and image feature vectors;

[0035] S107, The mask prediction loss and the image-text comparison learning loss are weighted and summed according to the preset weights to jointly optimize the parameters of the text encoder, image encoder and cross-modal interactive encoder to obtain the pedestrian retrieval model;

[0036] S108: Use a pedestrian retrieval model to extract features from the query text and the pedestrian image database to be retrieved, calculate the similarity between the features of the query text and the features of each pedestrian image to be retrieved, and generate a ranking result to output the target pedestrian image that matches the query text.

[0037] In some embodiments, semantic analysis is performed on the text description to identify and label entity words and attribute words, including:

[0038] The text description is segmented into words to obtain a sequence of text tags arranged in word order;

[0039] Input the text tag sequence into a context encoding network based on a pre-trained language model to obtain the semantic vector of each text tag;

[0040] Each semantic vector is matched with a preset set of entity category vectors and a set of attribute category vectors for similarity to obtain the entity category confidence score and the attribute category confidence score.

[0041] When the confidence level of an entity category reaches the first threshold, the corresponding text tag is labeled as an entity word; when the confidence level of an attribute category reaches the second threshold and it is not labeled as an entity word, the corresponding text tag is labeled as an attribute word.

[0042] Output a sequence of tags containing entity word tags and attribute word tags, which can be used in subsequent masking and random replacement of similar words.

[0043] Specifically, this embodiment performs semantic analysis on text descriptions based on the publicly available CUHK-PEDES dataset to identify and label entity words and attribute words. This process can be completed by calling the pre-trained language model BERT-Base-uncased in the Python development environment. The inference stage is performed on a single NVIDIA A40 graphics card, with an average inference latency of about 8 ms per sentence.

[0044] First, the English text description to be processed is input into the WordPiece tokenizer that works with the BERT model. The tokenizer outputs a one-dimensional sequence of tokens in word order, with each token carrying the original character range offset information for subsequent tag writing. For example, the text "The person wears a red backpack and white sneakers." is segmented into [CLS], the, person, wears, a, red, backpack, and, white, sneakers, ., [SEP]. This sequence also preserves the word-sub-word relationships, such as "backpack".

[0045] The aforementioned token sequence is then fed into a BERT context encoding network with frozen parameters. After performing a twelve-layer transformation on the sequence, the network generates a 768-dimensional semantic vector for each token in the output layer. The vector is implicitly integrated into the sentence context and can represent both local word meaning and overall context.

[0046] Next, we construct a set of entity category vectors and a set of attribute category vectors. Entity categories cover 25 high-frequency wearable or accessory items such as backpacks, coats, and hats, while attribute categories cover 15 common descriptive dimensions such as color, material, and length. For each category, we select 3-5 typical terms, obtain term vectors using the same BERT model, and then take the average to obtain the category prototype vectors, denoted as entity category vector Ve_i and attribute category vector Va_j.

[0047] For any tag in the sequence, its semantic vector Vs is used to calculate the cosine similarity with all Ve_i and Va_j, respectively, to obtain the entity confidence pe = max(cos(Vs,Ve_i)) and the attribute confidence pa = max(cos(Vs,Va_j)). In this embodiment, the first threshold Te is set to 0.65 and the second threshold Ta is set to 0.55. When pe ≥ Te, the tag is labeled as an entity word; when pa ≥ Ta and the tag has not yet been labeled as an entity word, the tag is labeled as an attribute word; otherwise, no label is assigned.

[0048] For the aforementioned example text, the tag "back##pack" has a similarity of 0.82 to the category "knapsack," exceeding the Te threshold, and is therefore labeled as an entity word; "red" has a similarity of 0.78 to the color category, meeting the attribute threshold and not being labeled as an entity word, so it is labeled as an attribute word; "white" is similarly labeled as an attribute word; the remaining tags remain unlabeled. The final generated tag sequence is output in its original order for direct use in subsequent masking and random replacement of similar words. No fine-tuning of the BERT parameters was performed throughout the process to ensure tag consistency and reproducibility.

[0049] The implementation method described above can efficiently and stably generate entity word and attribute word labels without manual annotation, providing accurate semantic anchors for subsequent masking and denoising training.

[0050] In some embodiments, masking and random replacement of similar words are performed on entity words and attribute words according to a set probability to generate masked and replaced training text, including:

[0051] Based on the preset first mask probability and second mask probability, entity words and attribute words to be masked are randomly selected from the text description, respectively.

[0052] Replace the selected entity words with the first mask marker, and replace the selected attribute words with the second mask marker;

[0053] For the text tag located at the first mask tag position, randomly select a replacement word from the entity class word set and replace the corresponding first mask tag with the replacement word;

[0054] For the text tag located at the second mask mark position, randomly select a replacement word from the set of words with the same attribute, and replace the corresponding second mask mark with the replacement word;

[0055] Keeping the original text tag sequence in its original order, output the training text after masking and random replacement of similar words.

[0056] Specifically, the dual-probability masking mechanism in this embodiment includes: a first mask probability for controlling the masking frequency of entity words; and a second mask probability for controlling the masking frequency of attribute words. These two probabilities can be set independently to meet the perturbation strength requirements of different semantic granularities. During the training phase, the labeled entity words and attribute words are sampled probabilistically within each sentence to ensure that the masking position distribution in each batch is random and conforms to the global expected value over the long term.

[0057] The masking settings include: a first masking tag specifically for placeholder entity words; a second masking tag specifically for placeholder attribute words; using discriminative tags enables the subsequent interactive encoder to explicitly distinguish different categories of missing information, avoiding confusion.

[0058] The random replacement strategy for similar words in this embodiment includes: for the masked position, instead of directly restoring the original word, randomly selecting replacement words from the candidate library of the same semantic category.

[0059] The candidate library is built offline in advance, including the following: a set of entity-related words is archived by category, such as "backpack, hat, coat"; a set of attribute-related words is aggregated by attributes such as color, material, length, etc., to describe synonyms or near-synonyms; random replacement can simulate description errors and ambiguities in real-world scenarios, enabling the model to learn to eliminate inconsistent noise.

[0060] Order-preserving output means that all masking and replacement operations are performed on the original tag sequence dimension, and the word order is kept unchanged and output directly after completion, ensuring that the text encoder can use positional encoding to capture contextual structure information.

[0061] For example, in a specific example, taking the English text "A tall woman in a green coat carries a black handbag." as an example, "woman," "coat," and "handbag" have been labeled as entity words, and "tall," "green," and "black" as attribute words, as described in Example 1. With a first mask probability of 0.4 and a second mask probability of 0.3, the following steps are performed:

[0062] 1. Randomly select mask positions

[0063] Randomly select "handbag" from entity phrases to match the mask; randomly select "green" from attribute phrases to match the mask, and keep the other tags as original words.

[0064] II. Mask Placeholder

[0065] Replace "handbag" with the first mask marker and "green" with the second mask marker to form the intermediate sequence: "A tall woman in a [mask1] coat carries a black [mask1]." Note: If multiple entities are masked within the same text, the same first mask marker is used, and the same applies to attributes.

[0066] III. Random replacement of similar words

[0067] For the first mask marker at position one, randomly select a word from the entity class set "package". Assuming "backpack" is selected, the text fragment after replacement becomes "backpack".

[0068] For the second mask marker at position two, randomly select a word from the set of words with the same attribute, "color". Suppose "yellow" is selected, the text fragment after replacement becomes "yellow".

[0069] IV. Order-preserving output

[0070] The final training text after masking and random replacement is: "A tall woman in a yellow coat carries a black backpack." This text forms a new training pair with the original paired image, while the original masked words "green" and "handbag" are used as supervision signals for the masking prediction task.

[0071] Through the methods described in the above embodiments, this embodiment introduces semantic noise without disrupting the syntactic structure, providing rich, controllable input samples that closely approximate the real error distribution for the robust training of subsequent cross-modal models.

[0072] In some embodiments, text feature vectors and image feature vectors are input into a cross-modal interactive encoder to obtain a fused feature representation, including:

[0073] The text feature vector and the image feature vector are concatenated into a joint sequence in a preset order, and a corresponding modality identifier and position code are added to each vector in the joint sequence.

[0074] The joint sequence is fed into a cross-modal interactive encoder consisting of multiple layers of self-attention sub-layers and cross-attention sub-layers stacked alternately, in order to establish a bidirectional attention association between the text modality and the image modality, and output an updated joint sequence.

[0075] Extract the fusion vector located at the preset aggregation position from the updated joint sequence as the fusion feature representation.

[0076] Specifically, in this embodiment, the text feature vector and the image feature vector are first concatenated into a joint sequence in a fixed order of "text first, image last". For each vector in this sequence, the system adds two types of auxiliary information: a modality identifier to distinguish whether its source is text or an image, and an absolute positional encoding to maintain the original word order or block order relationship. The modality identifier is implemented by concatenating a learnable binary embedding to the end of the vector, while the positional encoding uses the same calculation formula as in the transformer pre-training stage to ensure that the downstream network can directly parse it.

[0077] After concatenation, the joint sequence is fed into a cross-modal interactive encoder. Structurally, this encoder consists of four stacked transformer layers. Each layer contains a text self-attention sub-layer, an image self-attention sub-layer, and a bidirectional cross-attention sub-layer. The self-attention sub-layers update contextual information within their respective modalities, while the cross-attention sub-layers use query key-value cross-interaction to ensure that text vectors focus on image semantics while simultaneously ensuring that image vectors focus on text semantics, thus establishing fine-grained alignment relationships within the same latent space. Each sub-layer is followed by a residual path and layer normalization to stabilize the gradients.

[0078] Furthermore, after the joint sequence has undergone all four layers of interaction, the CLS aggregation position reserved at the beginning of the sequence for the text side will accumulate cross-modal context. The system directly extracts this CLS vector as the fusion vector output and records it as the final fusion feature representation. This avoids the extra computation of traversing all vectors for pooling and ensures the centralized expression of fusion information.

[0079] In some examples, taking the query sentence "A man in a blue jacket holding a white umbrella." and its corresponding pedestrian image as an example, the text encoder first outputs an L=77-dimensional text feature sequence, and the image encoder outputs M=196 image patch features. The length of the two is 273 after concatenation. After calculation by the interactive encoder, the CLS position finally gathers the association information between "bluejacket", "white umbrella" and the corresponding clothing and umbrella areas in the image. When calculating cosine similarity in the subsequent retrieval stage, the CLS fusion vector can simultaneously represent the global appearance and key details, thereby improving the cross-modal matching accuracy.

[0080] In some embodiments, the original words at the masked positions are predicted based on the fused feature representation, and the mask prediction loss is calculated, including:

[0081] Extract the feature vector of the masked position in the corresponding fused feature representation;

[0082] The feature vectors are sequentially input into a prediction head containing a linear transformation layer and a normalized activation function to obtain the probability distribution for a preset vocabulary.

[0083] The prediction error of a single mask position is calculated based on the cross-entropy between the probability distribution and the original word identifier of the masked position.

[0084] The mask prediction loss for the corresponding training text is obtained by averaging the prediction errors of all mask positions in the same training text.

[0085] Specifically, the fusion feature representation preserves the cross-modal contextual information of each marker in the text sequence, where the fusion vector corresponding to the masked marker has fused image cues and textual context. During the training phase, the system performs word reconstruction prediction separately for each masked location. The specific process is as follows.

[0086] I. Extracting the masked location features

[0087] For a training text, the model generates a fusion vector of equal length for all text tags during the cross-modal interaction phase. When the tag is the first or second mask tag, its fusion vector is directly extracted from the joint sequence according to the original index, denoted as v_mask. The dimension of this vector is consistent with the hidden layer of the interactive encoder, and in this embodiment, it is set to d=1024.

[0088] II. Predicting Head Structure and Output Probability

[0089] The extracted v_mask is sequentially processed through two levels of linear mapping. The first level of mapping compresses the dimension from 1024 to 256 and then applies GELU activation to enhance non-linear expressive power. The second level of mapping directly projects onto a fixed vocabulary space of size V, which is consistent with the pre-trained vocabulary of the text encoder; in this embodiment, V is 30522. The projection result is then normalized by Softmax to form a one-dimensional probability distribution p_mask, which is used to characterize the model's confidence in each candidate word.

[0090] III. Single-location loss calculation

[0091] The system records the index id_true of the original label in the vocabulary before masking. The natural logarithm of the probability p_true corresponding to this index is taken, and this logarithm is used to form the cross-entropy error e_mask with -log(p_true). To maintain consistent loss scaling for different sentence lengths, this embodiment does not introduce additional label weighting factors.

[0092] IV. Average Text-Level Loss Within Sentences

[0093] If the same training text contains n mask positions, the system calculates e_1 to e_n separately and then takes the arithmetic mean to obtain the mask prediction loss L_mask_text=(e_1+…+e_n) / n for that text. The batch loss is then averaged over all texts to obtain the batch-level mask loss.

[0094] In a specific example, the previously masked and randomly replaced text "A tall woman in a yellowcoat carries a black backpack." corresponds to the original words "green" and "handbag" being masked. After interactive encoding, the system extracts v_green and v_handbag from these two indices. The prediction head predicts "green" for v_green with a probability of 0.72 and a cross-entropy error of 0.331; it predicts "handbag" for v_handbag with a probability of 0.64 and an error of 0.446. The average of the two results in a masked prediction loss of 0.389 for this sentence. This value, along with the loss of the remaining text in the same batch, is used for backpropagation, prompting the model to more accurately recover the masked information and enhance the robustness of visual-semantic alignment.

[0095] In another example, embodiments of this application can use the following formula to calculate the mask prediction loss, in order to assist in cross-modal feature learning and improve the robustness and generalization ability of the model, as shown in the following formula:

[0096]

[0097] Where M represents the position to be masked. , The one-hot encoding representing the true category label. This represents the predicted value.

[0098] In some embodiments, the image-text comparison learning loss is calculated based on the similarity relationship between text feature vectors and image feature vectors, including:

[0099] Extract the global vector of the text and the global vector of the image from the text feature vector and the image feature vector respectively;

[0100] Within the same training batch, similarity calculations are performed between each text global vector and all image global vectors, and between each image global vector and all text global vectors, to obtain a bidirectional similarity matrix.

[0101] Based on the text-image matching labels corresponding to the similarity matrix, a normalized contrast loss function with a temperature coefficient is used to calculate the cross-entropy loss for the text-to-image direction and the image-to-text direction respectively.

[0102] The image-to-image cross-entropy loss is averaged with the image-to-text cross-entropy loss to obtain the image-to-text contrast learning loss.

[0103] Specifically, in this embodiment, the calculation of the image-text comparison learning loss relies on a batch-level cross-modal similarity alignment mechanism. First, the text encoder retains a CLS marker at the beginning of the output sequence to aggregate global text semantics; the image encoder similarly sets a CLS marker at the beginning of the input image block sequence to represent the global appearance of the image. The model extracts the hidden vectors corresponding to the two types of CLS markers, denoted as the text global vector and the image global vector, respectively. To ensure the stability of subsequent similarity measurements, each global vector undergoes L2 normalization to normalize its magnitude.

[0104] Subsequently, within a single training batch, the system calculates the cosine similarity between the global vector of any text sample and the global vectors of all images in that batch. The resulting similarity values ​​are then filled into the rows or columns of a matrix according to the order of text index and image index. The same operation is performed on any image sample, using the image vector as the query and all text vectors as the comparison target, resulting in another set of similarity matrices. In this way, bidirectional similarity matrices in both the "text-to-image" and "image-to-text" directions can be obtained simultaneously. To avoid an overly narrow numerical distribution, both matrices are divided by a temperature coefficient before Softmax normalization. This coefficient is set to 0.07 in the early stages of training and gradually reduced according to a preset schedule to improve the resolution of hard samples.

[0105] The system determines true matching pairs based on batch labels: the element in the i-th row and i-th column is a positive sample, and the rest are negative samples. For the text-to-image direction, the model calculates the cross-entropy loss on a per-row basis using the probability distribution normalized by Softmax; for the image-to-text direction, it calculates it on a per-column basis in the same way. The average of the two loss values ​​yields the image-text contrast learning loss for that batch, and its gradient will jointly drive the text encoder and image encoder to tighten the positive alignment distance and widen the negative alignment distance within the same embedding space.

[0106] To deepen understanding, the following explanation uses three batches of image-text pairing samples. Assume text T1 describes "white shirt and blue jeans," corresponding to image I1 showing a pedestrian of the same color; text T2 describes "red dress with handbag," corresponding to image I2; and text T3 describes "black coat and hat," corresponding to image I3. The model calculates a six-dimensional, nine-component similarity matrix from the three text CLS vectors and the three image CLS vectors. After temperature scaling and Softmax, the probabilities at T1-I1, T2-I2, and T3-I3 are significantly higher than the remaining terms. The system uses diagonal elements as positive sample target probabilities and off-diagonal elements as zero labels, calculating row and column losses, and then averaging them as the final image-text comparison learning loss. This loss, during backpropagation, makes positively aligned vectors like T1 and I2 closer together, while reducing the similarity of incorrect pairings like T1-I2, thus improving the overall discriminative power of cross-modal embedding.

[0107] Through the above mechanism, this embodiment simultaneously constrains two mapping paths, text-to-image and image-to-text, at the batch granularity, enabling the model to obtain accurate alignment of cross-modal global semantics within the shared feature space, and significantly enhancing the discriminability of noisy pairings, providing higher matching accuracy and more stable robustness for subsequent retrieval stages.

[0108] In a specific example, the image-text contrastive learning loss is used to construct a shared feature space across different modalities (images and text). By maximizing the similarity of true matching pairs and minimizing the similarity of non-matching pairs, it improves cross-modal alignment and discrimination capabilities, thereby enhancing the retrieval accuracy and robustness in text-to-image pedestrian retrieval tasks. This embodiment can use the following formula to calculate the image-text contrastive learning loss:

[0109]

[0110] Among them, middle image To text Based on the similarity, image i and text j are matched. image The similarity to text k is not necessarily a match. Indicates the temperature coefficient. This represents the number of training samples. The meaning is similar; This represents the contrast loss of the image relative to the text. This represents the contrast loss from text to image. This represents the total comparative loss.

[0111] In some embodiments, the mask prediction loss and the image-text comparison learning loss are weighted and summed according to preset weights to jointly optimize the parameters of the text encoder, image encoder, and cross-modal interactive encoder, thereby obtaining a pedestrian retrieval model, including:

[0112] The mask prediction loss and the image-text comparison learning loss are multiplied by the first weight coefficient and the second weight coefficient, respectively, and the weighted results are summed to construct the total loss function;

[0113] Gradient backpropagation is performed based on the total loss function, and the resulting gradients are applied in parallel to the text encoder, image encoder, and cross-modal interactive encoder to update their respective parameters synchronously.

[0114] The total loss function construction and parameter update are repeatedly executed according to the preset training iteration strategy until the convergence condition is met, and the pedestrian retrieval model is obtained. The first weight coefficient and the second weight coefficient are dynamically adjusted according to the training progress and the preset scheduling curve.

[0115] Specifically, in this embodiment, the training system simultaneously considers mask prediction loss and image-text contrast learning loss, constructing a unified optimization objective through weighted summation. The mask prediction loss primarily drives the model to recover masked words and strengthens the alignment of local entities and attributes; the image-text contrast learning loss focuses on compressing the distance between positive samples across modalities and expanding the interval between negative samples, thereby improving global discriminative ability. To balance the importance of the two tasks at different training stages, the system assigns a first weight coefficient λ1 and a second weight coefficient λ2 to each, with their sum remaining at 1 to ensure stable gradient magnitudes.

[0116] At the start of training, λ1 is initialized to 0.7 and λ2 to 0.3. As iterations progress and the model has acquired basic alignment capabilities, further emphasis needs to be placed on cross-modal differentiation. Therefore, starting from the 15th epoch, λ2 is increased by 0.04 each epoch using a linear scheme, while λ1 is correspondingly decreased, until λ1=0.3 and λ2=0.7 by the 25th epoch. The weight scheduling curve is pre-written into the configuration file and automatically updated by the training scheduler during epoch switching, avoiding inconsistencies caused by manual intervention.

[0117] In each training step, the system selects 32 pairs of image-text samples to form a batch. First, it uses forward propagation to obtain the mask prediction loss L_mask and the image-text contrast loss L_contrast. Then, it calculates the total loss L_total = λ1·L_mask + λ2·L_contrast according to the current weight coefficients. Gradients are applied to the text encoder, image encoder, and cross-modal interactive encoder respectively through backpropagation. The three types of networks share the same learning rate plan but maintain momentum terms independently. The optimizer uses AdamW with a base learning rate of 3×10⁻⁶. -5 Linear warm-up is used for the first 60% of training steps, and cosine decay is used for the remaining steps until the convergence threshold of 1×10⁻⁶ is reached. -6 .

[0118] To ensure synchronized updates, the system uses mixed-precision training when sufficient GPU memory is available. Gradients are aggregated on the main GPU and then uniformly clipped before being written back according to the weight decay rate of each network parameter group, ensuring consistent learning speeds for the text and vision branches. The system was trained for 30 epochs on the CUHK-PEDES dataset. Convergence was considered achieved when the mAP improvement on the validation set was less than 0.1% for three consecutive epochs, at which point training was terminated early and the pedestrian retrieval model was saved.

[0119] Through the aforementioned dynamic weight adjustment and joint optimization strategies, the model fully learns mask reconstruction capabilities and stabilizes fine-grained alignment in the early stages, and gradually emphasizes global contrast discrimination in the later stages to eliminate noise interference. Ultimately, it balances local robustness and cross-modal discriminability. In tests on public datasets, the retrieval accuracy is improved by about 5 percentage points compared to the single loss baseline, and the training convergence speed is also improved by about 18%.

[0120] In some embodiments, a pedestrian retrieval model is used to extract features from the query text and the pedestrian image database to be retrieved, respectively. The similarity between the query text features and the features of each pedestrian image to be retrieved is calculated, and a ranking result is generated to output the target pedestrian image matching the query text. This includes:

[0121] Input the query text into the jointly optimized text encoder to obtain the query text feature vector;

[0122] Each pedestrian image in the pedestrian image database to be retrieved is input into the jointly optimized image encoder to obtain the corresponding set of pedestrian image feature vectors;

[0123] Calculate the similarity value between the query text feature vector and each pedestrian image feature vector in the pedestrian image feature vector set, and generate a similarity list;

[0124] Sort the pedestrian image feature vector set in descending order according to the similarity value to obtain the sorted result containing pedestrian image identifiers and similarity values;

[0125] Based on the ranking results, at least one pedestrian image with the highest similarity is determined and output as the target pedestrian image that matches the query text.

[0126] Specifically, in this embodiment, the pedestrian retrieval system pre-completes two main stages: offline feature database construction and online retrieval. In the offline stage, each pedestrian image in the database is first scaled to a uniform resolution, maintaining a long side of 256 pixels and preserving the aspect ratio. It is then divided into fixed-size, non-overlapping blocks and input into a jointly optimized image encoder. The encoder retains the CLS marker at the first block position and outputs a 1024-dimensional global vector. To eliminate amplitude differences caused by varying image brightness and contrast, the system performs L2 normalization on this vector, obtaining image features with a magnitude of 1. The normalized features are then written into the vector index along with the image primary key.

[0127] If the image library contains fewer than 50,000 images, the index uses a pure dense vector storage method and is loaded into memory in segments. If the library contains more than 50,000 images, the index uses a modular inverted folder structure combined with product quantization encoding. The specific steps are as follows: First, perform two-level K-means clustering on all features using 256 code clusters, recording the pointer from each feature to the cluster center. Then, divide the residual vector into sixteen subspaces, quantize each into an eight-bit subcode, and finally concatenate the cluster pointers and subcodes to create a compressed index, and build an inverted index based on the cluster centers. The above operations are completed on a workstation with four GPUs and 48GB of video memory. Indexing one million images takes approximately three hours and occupies 5.2GB of disk space.

[0128] During the online retrieval phase, the user inputs a natural language description as the query text. The system first calls the text encoder with the same parameters as during the training phase to perform WordPiece word segmentation on the sentence and obtain the 1024-dimensional global text vector corresponding to the CLS tag. Then, it performs L2 normalization to obtain the query vector q. If the index is a full dense storage, the system directly iterates through and calculates the cosine similarity between q and each image feature v_i, with the numerical formula being q·v_i. On a single A40 graphics card, calculating 100,000 similarities takes an average of nine milliseconds. If the index is a compressed inverted index structure, the system first calculates the Euclidean distance between q and the 256 cluster centers, selects the ten centers with the smallest distances into the candidate pool, decodes the residual subcode in the corresponding inverted index to recover the approximate v_i, and calculates the inner product approximate cosine value with q. The candidate pool size accounts for about four percent of the total image library. Therefore, even if the image library contains one million images, the actual number of similarity calculations is still controlled at around 40,000, with an average time of fifteen milliseconds, which is within an acceptable latency range.

[0129] The system sorts the obtained similarity values ​​from largest to smallest and generates a corresponding list of image primary keys. By default, the top fifty items are returned for front-end display. If the user needs more precise results, the complete floating-point features of the top fifty items can be read again and recalculated precisely with q. This "two-stage retrieval" balances system throughput and accuracy. After sorting, the system directly displays the image with the highest similarity ranking as the target pedestrian image to the user, and simultaneously lists the remaining candidates on the interface.

[0130] To illustrate the retrieval results, the query text "aman with a brown leather jacket and a red backpack" is used for demonstration. The text encoder outputs vector q, which is then compared with an index of one million images in the image database. The index returns the image with the highest initial similarity (#865432, score 0.91) and the second highest (#342117, score 0.86). The system further loads precise features onto these two images for recalculation, resulting in scores of 0.927 and 0.835 respectively. The rankings remain unchanged, and image #865432 is ultimately output to the user. Manual verification confirms that the image does indeed depict a man wearing a brown leather jacket and carrying a red backpack, achieving the expected retrieval accuracy.

[0131] This embodiment combines offline quantized indexing with online hierarchical retrieval, controlling the total time of a single retrieval within 20 milliseconds in a million-level image library, while maintaining Top1 accuracy and full-precision brute-force search to be basically consistent, effectively ensuring the response speed and accuracy of text-to-image pedestrian retrieval in large-scale real-time application scenarios.

[0132] According to the technical solutions of the above embodiments of this application, this application has at least the following advantages:

[0133] 1. By training through mask prediction at the entity and attribute levels, the model can still accurately capture visual-semantic correspondences even when there are some errors, omissions or ambiguities in the text description, effectively alleviating the training bias problem caused by incorrect image-text pairing and text noise in the real environment;

[0134] 2. It can effectively resist interference caused by incorrect image-text matching and description noise in real-world environments in text-to-image pedestrian retrieval tasks. By training based on entity and attribute-level mask prediction, it improves the model's fine-grained cross-modal alignment capability, thereby significantly improving retrieval accuracy and robustness.

[0135] 3. By predicting entities and attributes using masks and combining image-text comparison learning loss, while supervising local entities / attributes and global representations, the model is equipped with fine-grained cross-modal alignment capabilities.

[0136] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0137] Figure 2 This is a schematic diagram of the structure of the text-to-image pedestrian retrieval device based on mask denoising provided in an embodiment of this application. Figure 2 As shown, the text-to-image pedestrian retrieval device based on mask denoising includes:

[0138] The recognition module 201 is used to acquire cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words.

[0139] The masking module 202 is used to perform masking and random replacement of similar words on entity words and attribute words according to a set probability, and generate training text with masking replacement.

[0140] The encoding module 203 is used to input the masked training text and the corresponding pedestrian image into the text encoder and the image encoder respectively to obtain the text feature vector and the image feature vector.

[0141] The fusion module 204 is used to input the text feature vector and the image feature vector into the cross-modal interactive encoder to obtain the fused feature representation;

[0142] The prediction module 205 is used to predict the original words at the masked positions based on the fused feature representation and to calculate the mask prediction loss.

[0143] Calculation module 206 is used to calculate the image-text comparison learning loss based on the similarity relationship between text feature vectors and image feature vectors;

[0144] The joint module 207 is used to perform a weighted summation of the mask prediction loss and the image-text comparison learning loss according to preset weights, so as to jointly optimize the parameters of the text encoder, image encoder and cross-modal interactive encoder to obtain the pedestrian retrieval model.

[0145] The output module 208 is used to extract features from the query text and the pedestrian image database to be retrieved using a pedestrian retrieval model, calculate the similarity between the features of the query text and the features of each pedestrian image to be retrieved, and generate a ranking result to output the target pedestrian image that matches the query text.

[0146] In some embodiments, Figure 2The recognition module 201 performs word segmentation on the text description to obtain a text tag sequence arranged in word order; the text tag sequence is input into a context encoding network based on a pre-trained language model to obtain the semantic vector of each text tag; each semantic vector is matched with a preset set of entity category vectors and a set of attribute category vectors to obtain entity category confidence and attribute category confidence; when the entity category confidence reaches the first threshold, the corresponding text tag is labeled as an entity word; when the attribute category confidence reaches the second threshold and is not labeled as an entity word, the corresponding text tag is labeled as an attribute word; the output is a tag sequence containing entity word labels and attribute word labels for subsequent masking and random replacement of similar words steps.

[0147] In some embodiments, Figure 2 The masking module 202, based on preset first and second masking probabilities, randomly selects entity words and attribute words to be masked from the text description; replaces the selected entity words with the first masking marker and the selected attribute words with the second masking marker; for text markers at the first masking marker position, randomly extracts replacement words from the entity word set and replaces the corresponding first masking marker with the replacement words; for text markers at the second masking marker position, randomly extracts replacement words from the attribute word set and replaces the corresponding second masking marker with the replacement words; keeping the original text marker sequence order unchanged, the training text after masking and random replacement of word-like words is output.

[0148] In some embodiments, Figure 2 The fusion module 204 concatenates the text feature vector and the image feature vector in a preset order to form a joint sequence, and adds a corresponding modality identifier and position code to each vector in the joint sequence; the joint sequence is fed into a cross-modal interactive encoder consisting of multiple layers of self-attention sub-layers and cross-attention sub-layers stacked alternately to establish a bidirectional attention association between the text modality and the image modality, and outputs an updated joint sequence; the fusion vector located at a preset aggregation position is extracted from the updated joint sequence as the fusion feature representation.

[0149] In some embodiments, Figure 2 The prediction module 205 extracts the feature vectors of the masked positions in the corresponding fused feature representation; inputs the feature vectors sequentially into the prediction head containing a linear transformation layer and a normalized activation function to obtain the probability distribution for the preset vocabulary; calculates the prediction error of a single masked position based on the cross-entropy between the probability distribution and the original word identifier of the masked position; and averages the prediction errors of all masked positions in the same training text to obtain the mask prediction loss of the corresponding training text.

[0150] In some embodiments, Figure 2The computation module 206 extracts the text global vector and image global vector from the text feature vector and image feature vector, respectively. Within the same training batch, it performs similarity calculations between each text global vector and all image global vectors, and between each image global vector and all text global vectors, to obtain a bidirectional similarity matrix. Based on the text-image matching labels corresponding to the similarity matrix, it uses a normalized contrast loss function with a temperature coefficient to calculate the cross-entropy loss in the text-to-image direction and the image-to-text direction, respectively. It averages the cross-entropy loss in the text-to-image direction and the cross-entropy loss in the image-to-text direction to obtain the image-text contrast learning loss.

[0151] In some embodiments, Figure 2 The joint module 207 multiplies the mask prediction loss and the image-text comparison learning loss by the first weight coefficient and the second weight coefficient, respectively, and sums the weighted results to construct the total loss function. Based on the total loss function, gradient backpropagation is performed, and the obtained gradients are applied in parallel to the text encoder, image encoder and cross-modal interactive encoder to update their respective parameters synchronously. The construction of the total loss function and parameter updates are repeatedly performed according to the preset training iteration strategy until the convergence condition is met, and the pedestrian retrieval model is obtained. The first weight coefficient and the second weight coefficient are dynamically adjusted according to the training progress according to the preset scheduling curve.

[0152] In some embodiments, Figure 2 The output module 208 takes the query text as input to the jointly optimized text encoder to obtain the query text feature vector; it takes each pedestrian image in the pedestrian image library to be retrieved as input to the jointly optimized image encoder to obtain the corresponding pedestrian image feature vector set; it calculates the similarity value between the query text feature vector and each pedestrian image feature vector in the pedestrian image feature vector set to generate a similarity list; it sorts the pedestrian image feature vector set in descending order according to the size of the similarity value to obtain a sorted result containing pedestrian image identifiers and similarity values; based on the sorted result, it determines at least one pedestrian image with the highest similarity as the target pedestrian image matching the query text and outputs it.

[0153] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0154] Figure 3 This is a schematic diagram of the structure of the electronic device 3 provided in an embodiment of this application. Figure 3As shown, the electronic device 3 of this embodiment includes a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program 303, it implements the steps in the various method embodiments described above. Alternatively, when the processor 301 executes the computer program 303, it implements the functions of each module / unit in the various device embodiments described above.

[0155] For example, computer program 303 may be divided into one or more modules / units, which are stored in memory 302 and executed by processor 301 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 303 in electronic device 3.

[0156] Electronic device 3 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 3 may include, but is not limited to, processor 301 and memory 302. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0157] Processor 301 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0158] The memory 302 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 302 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 302 can include both internal and external storage units of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0159] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0160] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0161] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0162] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0163] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0165] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0166] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although the technical solutions of this application have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A text-to-image pedestrian retrieval method based on mask denoising, characterized in that, include: Obtain cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words; According to a set probability, the entity words and attribute words are respectively masked and randomly replaced with similar words to generate training text with mask replacement; The masked training text and the corresponding pedestrian image are input into the text encoder and image encoder respectively to obtain the text feature vector and the image feature vector. The text feature vector and image feature vector are input into a cross-modal interactive encoder to obtain a fused feature representation; Based on the fused feature representation, the original words at the masked positions are predicted, and the mask prediction loss is calculated. The image-text comparison learning loss is calculated based on the similarity relationship between the text feature vector and the image feature vector. The mask prediction loss and the image-text comparison learning loss are weighted and summed according to preset weights to jointly optimize the parameters of the text encoder, image encoder and cross-modal interactive encoder, so as to obtain the pedestrian retrieval model. The pedestrian retrieval model is used to extract features from the query text and the pedestrian image database to be retrieved, respectively. The similarity between the features of the query text and the features of each pedestrian image to be retrieved is calculated and a ranking result is generated to output the target pedestrian image that matches the query text. The step of performing masking and random replacement of similar words on the entity words and attribute words according to a set probability to generate masked and replaced training text includes: Based on the preset first mask probability and second mask probability, entity words and attribute words to be masked are randomly selected from the text description, respectively. Replace the selected entity words with the first mask marker, and replace the selected attribute words with the second mask marker; For the text mark located at the first mask mark position, a replacement word is randomly extracted from the entity class word set, and the corresponding first mask mark is replaced with the replacement word; For the text mark located at the second mask mark position, a replacement word is randomly selected from the set of words with the same attribute, and the corresponding second mask mark is replaced with the replacement word; Keeping the original text tag sequence in its original order, output the training text after masking and random replacement of similar words.

2. The method according to claim 1, characterized in that, The semantic analysis of the text description, identifying and labeling entity words and attribute words, includes: The text description is segmented into words to obtain a text tag sequence arranged in word order; The text tag sequence is input into a context encoding network based on a pre-trained language model to obtain the semantic vector of each text tag; Each semantic vector is matched with a preset set of entity category vectors and a set of attribute category vectors for similarity to obtain entity category confidence and attribute category confidence. When the confidence level of an entity category reaches the first threshold, the corresponding text tag is labeled as an entity word; when the confidence level of an attribute category reaches the second threshold and it is not labeled as an entity word, the corresponding text tag is labeled as an attribute word. Output a sequence of tags containing entity word tags and attribute word tags, which can be used in subsequent masking and random replacement of similar words.

3. The method according to claim 1, characterized in that, The step of inputting the text feature vector and image feature vector into a cross-modal interactive encoder to obtain a fused feature representation includes: The text feature vector and the image feature vector are concatenated into a joint sequence in a preset order, and a corresponding modality identifier and position code are added to each vector in the joint sequence. The joint sequence is fed into a cross-modal interactive encoder consisting of multiple layers of self-attention sublayers and cross-attention sublayers stacked alternately, so as to establish a bidirectional attention association between the text modality and the image modality and output an updated joint sequence. The fusion vector located at the preset aggregation position is extracted from the updated joint sequence and used as the fusion feature representation.

4. The method according to claim 1, characterized in that, The prediction of the original word at the masked position based on the fused feature representation, and the calculation of the mask prediction loss, include: Extract the feature vector corresponding to the masked position in the fused feature representation; The feature vectors are sequentially input into a prediction head containing a linear transformation layer and a normalized activation function to obtain the probability distribution for a preset vocabulary. The prediction error of a single mask position is calculated based on the cross-entropy between the probability distribution and the original word identifier of the masked position; The mask prediction loss for the corresponding training text is obtained by averaging the prediction errors of all mask positions in the same training text.

5. The method according to claim 1, characterized in that, The calculation of the image-text comparison learning loss based on the similarity relationship between the text feature vector and the image feature vector includes: Extract the global text vector and the global image vector from the text feature vector and the image feature vector, respectively; Within the same training batch, similarity calculations are performed between each text global vector and all image global vectors, and between each image global vector and all text global vectors, to obtain a bidirectional similarity matrix. Based on the text-image matching labels corresponding to the similarity matrix, a normalized contrast loss function with a temperature coefficient is used to calculate the cross-entropy loss for the text-to-image direction and the image-to-text direction respectively. The image-text contrast learning loss is obtained by averaging the cross-entropy loss in the text-to-image direction and the cross-entropy loss in the image-to-text direction.

6. The method according to claim 1, characterized in that, The step of weighting and summing the mask prediction loss and the image-text comparison learning loss according to preset weights to jointly optimize the parameters of the text encoder, image encoder, and cross-modal interactive encoder to obtain the pedestrian retrieval model includes: The mask prediction loss and the image-text comparison learning loss are multiplied by the first weight coefficient and the second weight coefficient, respectively, and the weighted results are summed to construct the total loss function; Gradient backpropagation is performed based on the total loss function, and the resulting gradients are applied in parallel to the text encoder, image encoder, and cross-modal interactive encoder to update their respective parameters synchronously. The total loss function construction and parameter update are repeatedly executed according to the preset training iteration strategy until the convergence condition is met, and the pedestrian retrieval model is obtained. The first weight coefficient and the second weight coefficient are dynamically adjusted according to the training progress and the preset scheduling curve.

7. The method according to claim 1, characterized in that, The process of extracting features from the query text and the pedestrian image database using the pedestrian retrieval model, calculating the similarity between the query text features and the features of each pedestrian image to be retrieved, and generating a ranking result to output the target pedestrian image matching the query text includes: Input the query text into the jointly optimized text encoder to obtain the query text feature vector; Each pedestrian image in the pedestrian image database to be retrieved is input into the jointly optimized image encoder to obtain the corresponding set of pedestrian image feature vectors; Calculate the similarity value between the query text feature vector and each pedestrian image feature vector in the pedestrian image feature vector set, and generate a similarity list; The pedestrian image feature vector set is sorted in descending order according to the similarity value to obtain a sorting result containing pedestrian image identifiers and similarity values; Based on the sorting results, at least one pedestrian image with the highest similarity is determined and output as the target pedestrian image that matches the query text.

8. A text-to-image pedestrian retrieval device based on mask denoising, characterized in that, include: The recognition module is used to acquire cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words; The masking module is used to perform masking and random replacement of similar words on the entity words and attribute words according to a set probability, and generate training text with masking and replacement. The encoding module is used to input the masked training text and the corresponding pedestrian image into the text encoder and image encoder respectively to obtain the text feature vector and the image feature vector. The fusion module is used to input the text feature vector and the image feature vector into the cross-modal interactive encoder to obtain the fused feature representation; The prediction module is used to predict the original words at the masked positions based on the fused feature representation and calculate the mask prediction loss. The calculation module is used to calculate the image-text comparison learning loss based on the similarity relationship between the text feature vector and the image feature vector; The joint module is used to perform a weighted summation of the mask prediction loss and the image-text comparison learning loss according to preset weights, so as to jointly optimize the parameters of the text encoder, image encoder and cross-modal interactive encoder to obtain the pedestrian retrieval model. The output module is used to extract features from the query text and the pedestrian image database to be retrieved using the pedestrian retrieval model, calculate the similarity between the features of the query text and the features of each pedestrian image to be retrieved, and generate a ranking result to output the target pedestrian image that matches the query text. The masking module is used to randomly select entity words and attribute words to be masked in the text description based on a preset first masking probability and a second masking probability, respectively; replace the selected entity words with the first masking marker, and replace the selected attribute words with the second masking marker; For a text tag at the first mask tag position, a replacement word is randomly selected from the entity class word set, and the corresponding first mask tag is replaced with the replacement word; for a text tag at the second mask tag position, a replacement word is randomly selected from the attribute class word set, and the corresponding second mask tag is replaced with the replacement word; keeping the original text tag sequence order unchanged, the training text after masking and random replacement of class words is output.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images