Text-to-image pedestrian retrieval method and device based on mask denoising and medium

Abstract

The invention provides a text-to-image pedestrian retrieval method and device based on mask denoising and a medium. The method comprises the following steps: respectively executing mask and similar word random replacement on entity words and attribute words according to a set probability, and generating a training text replaced by a mask; inputting the text feature vector and the image feature vector into a cross-modal interaction encoder to obtain fusion feature representation; predicting the original words at the masked positions based on the fused feature representation, and calculating mask prediction loss; calculating image-text contrast learning loss based on a similarity relationship between the text feature vector and the image feature vector; and extracting features of the query text and the to-be-retrieved pedestrian image library by using the pedestrian retrieval model, calculating the similarity between the features of the query text and the features of the to-be-retrieved pedestrian images, and generating a sorting result so as to output a target pedestrian image matched with the query text. According to the method, the robustness of visual semantic alignment in a noise scene can be improved, and the pedestrian retrieval accuracy is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a text-to-image person retrieval method and device based on mask denoising and a medium. BACKGROUND

[0002] Text-to-image person retrieval (TIPR) is a cross-modal retrieval technology that uses natural language description to locate target pedestrians in a large-scale pedestrian image library. Compared with person re-identification (Re-ID) that can only rely on image queries, TIPR allows users to directly input free and flexible textual descriptions, which has significant application value in public security, smart city monitoring, criminal investigation, and other scenarios.

[0003] To achieve accurate alignment of visual and language modalities, existing TIPR research mainly focuses on feature extraction and matching strategies, which can be summarized into two directions: Global matching method: similarity is calculated by unified visual global vector and text global vector, and optimization is performed with the help of contrast learning or metric learning loss. The advantage is that the network structure is simple and the inference speed is fast, and the disadvantage is that it cannot finely describe local semantics such as "clothing color" and "personal belongings".

[0004] Local matching method: further subdivide image regions and text phrases, and use explicit region detection or implicit attention mechanism to realize word-block level alignment, which can improve fine-grained retrieval performance.

[0005] The existing technology still has the following problems: 1) relying on the perfect text-image pairing assumption: most TIPR methods assume that the text-image pairing in the training set is completely correct, and do not consider the incorrect pairing caused by posture occlusion, environmental lighting, or incomplete description in actual scenarios. Once this assumption is broken, the model is easy to learn noise samples as positive examples, resulting in a significant decrease in retrieval accuracy. 2) Limited transfer effect of single-modal noise resistance mechanism: existing sample selection and robust loss function are mainly designed for image classification or single-modal tasks, and when directly applied to cross-modal TIPR, it is difficult to accurately identify and eliminate incorrect text-image correspondence, and the robustness improvement is limited. 3) Fine-grained local alignment and computational overhead are difficult to balance: explicit local matching requires calling external detectors, and implicit local matching is more sensitive to noise, both of which lack a stable solution that balances accuracy and real-time performance in large-scale retrieval scenarios. SUMMARY

[0006] In view of this, the embodiments of the present application provide a text-to-image pedestrian retrieval method, device and medium based on mask denoising to solve the problems of cross-modal alignment distortion and decreased retrieval accuracy under the interference of erroneous image-text pairing noise in the prior art.

[0007] The first aspect of the embodiment of the present application provides a text-to-image pedestrian retrieval method based on mask denoising, comprising: obtaining cross-modal training data containing pedestrian images and corresponding text descriptions, performing semantic analysis on the text descriptions, identifying and labeling entity words and attribute words; performing masking and random replacement of similar words on entity words and attribute words according to set probabilities to generate masked training text; inputting the masked training text and the corresponding pedestrian images into a text encoder and an image encoder respectively to obtain a text feature vector and an image feature vector; inputting the text feature vector and the image feature vector into a cross-modal interactive encoder to obtain a fused feature representation ; Based on the fused feature representation, the original words at the masked position are predicted and the mask prediction loss is calculated; based on the similarity relationship between the text feature vector and the image feature vector, the image-text contrast learning loss is calculated; according to the preset weights, the mask prediction loss and the image-text contrast learning loss are weightedly summed to jointly optimize the parameters of the text encoder, image encoder and cross-modal interaction encoder to obtain a pedestrian retrieval model; the pedestrian retrieval model is used to extract features from the query text and the pedestrian image library to be retrieved, the similarity between the query text features and the features of each pedestrian image to be retrieved is calculated, and a ranking result is generated to output the target pedestrian image that matches the query text.

[0008] In a second aspect, the embodiment of the present application provides a text-to-image pedestrian retrieval device based on mask denoising, comprising: an identification module configured to obtain cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words; a mask module configured to perform mask and random replacement of similar words on the entity words and attribute words according to a set probability, and generate training text after mask replacement; an encoding module configured to input the training text after mask replacement and the corresponding pedestrian images into a text encoder and an image encoder respectively, and obtain a text feature vector and an image feature vector; a fusion module configured to input the text feature vector and the image feature vector into a cross-modal interaction encoder to obtain a fusion feature representation; a prediction module configured to predict original words at masked positions based on the fusion feature representation, and calculate a mask prediction loss; a calculation module configured to calculate a text-image contrast learning loss based on a similarity relationship between the text feature vector and the image feature vector; a joint module configured to weight and sum the mask prediction loss and the text-image contrast learning loss according to a preset weight, to jointly optimize parameters of the text encoder, the image encoder and the cross-modal interaction encoder, and obtain a pedestrian retrieval model; and an output module configured to extract features of a query text and a to-be-retrieved pedestrian image library using the pedestrian retrieval model, calculate similarities between the query text feature and each to-be-retrieved pedestrian image feature, and generate a ranking result, to output a target pedestrian image matched with the query text.

[0009] In a third aspect, the embodiment of the present application provides a computer-readable storage medium storing a computer program, and the computer program is executed by a processor to implement the steps of the above method.

[0010] The above at least one technical solution adopted by the embodiment of the present application can achieve the following beneficial effects: The cross-modal training data containing the pedestrian image and the corresponding text description is acquired, semantic analysis is performed on the text description, entity words and attribute words are recognized and labeled, the entity words and the attribute words are respectively subjected to mask and random replacement of the same type of words according to a set probability, the training text subjected to the mask replacement is generated, the training text subjected to the mask replacement and the corresponding pedestrian image are respectively input into a text encoder and an image encoder, a text feature vector and an image feature vector are obtained, the text feature vector and the image feature vector are input into a cross-modal interaction encoder, a fusion feature representation is obtained, the original words at the masked positions are predicted based on the fusion feature representation, a mask prediction loss is calculated, a text-image contrast learning loss is calculated based on the similarity relationship between the text feature vector and the image feature vector, the mask prediction loss and the text-image contrast learning loss are weighted and summed according to a preset weight, the parameters of the text encoder, the image encoder and the cross-modal interaction encoder are jointly optimized, and a pedestrian retrieval model is obtained, the pedestrian retrieval model is used to extract features of a query text and a to-be-retrieved pedestrian image library, the similarity between the query text feature and each to-be-retrieved pedestrian image feature is calculated, and a ranking result is generated, so that a target pedestrian image matched with the query text is output. The application can improve the robustness of visual semantic alignment in a noise scene and significantly improve the pedestrian retrieval accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creating any creative labor.

[0012] Figure 1 is a flowchart of a text-to-image pedestrian retrieval method based on mask denoising provided by the embodiments of the present application; Figure 2 is a structural schematic diagram of a text-to-image pedestrian retrieval device based on mask denoising provided by the embodiments of the present application; Figure 3 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures, techniques, etc. are presented in order to illustrate, but not to limit, the embodiments of the present application, so that the embodiments of the present application can be thoroughly understood. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted, so as not to obscure the description of the present application with unnecessary details.

[0014] Text-to-image person retrieval (TIPR) aims to identify the target person using a text description and find the most relevant person image from a person image library. The text query used in this task is free and flexible, and compared with the structured image query of the person re-identification (Re-ID) task, it is more versatile and has great application potential in the field of public security. Existing technologies mainly use global and local alignment methods of visual and text features, combined with contrast learning and local region matching to improve the recognition accuracy. However, existing methods generally assume that the image-text pairing in the training data is completely correct, and it is difficult to deal with the problem of incorrect image-text pairing caused by posture, angle, light or inaccurate description in the actual scene, which makes the model vulnerable to noise interference and reduces the retrieval performance. Therefore, how to achieve robust and efficient TIPR in the presence of imperfect image-text pairing has become an important research direction in this field.

[0015] The current common solution to the TIPR task is mainly focused on designing an effective alignment strategy to achieve good matching between image and text modalities. It is divided into global matching and local matching. The global matching method directly aligns the global text and visual representation by designing a reasonable cross-modal matching loss function, which ignores fine-grained information. The local matching method aligns fine-grained image and text information, enhancing cross-modal alignment. Local matching is further divided into explicit local matching and implicit local matching. The former relies on external technology and cross-modal information, increasing the computational complexity of the inference stage; the latter does not rely on external display and significantly reduces the additional computational overhead.

[0016] A large number of methods have been proposed to achieve noise-robust learning, which are divided into sample selection methods and robust loss function methods. Sample selection methods use the memory effect of deep neural networks to gradually distinguish noise data, so that the model pays more attention to clean data and reduces attention to noise data during training. Robust loss functions aim to design loss functions with noise tolerance to improve the robustness of the model to noise during training.

[0017] The existing technical solutions have the following disadvantages: 1. Existing TIPR methods generally assume that the image-text pairing in the training data is accurate, and it is difficult to deal with the problem of incorrect image-text pairing (noise correspondence) caused by incomplete description, angle occlusion, etc. in the actual scene, resulting in a significant decrease in retrieval accuracy in real complex environments.

[0018] 2. Existing noise robustness methods are mostly focused on classification tasks or single-modal fields, and it is difficult to effectively identify and eliminate incorrect image-text pairings when directly applied to TIPR, which lacks robustness and lacks a stable and efficient processing mechanism in large-scale image-text matching retrieval scenarios.

[0019] In view of the problems existing in the prior art, the present application proposes a text-to-image pedestrian retrieval method based on mask denoising, which is used to solve the noise correspondence problem in the text-to-image pedestrian retrieval task. Compared with the traditional TIPR task, it is easy to be affected by incorrect image-text pairing in real environment, resulting in a decrease in retrieval accuracy. The present application mainly solves the following problems: 1. In the training phase, the entity words and attribute words in the text description are masked and replaced with a certain probability, combined with a cross-modal interaction encoder and a mask prediction loss, so that the model can still learn a robust and accurate visual-semantic alignment relationship when there is text description noise and incorrect image-text pairing.

[0020] 2. Introduce an image-text contrast learning loss in the training process, combined with the mask prediction loss for joint optimization, so that the model not only relies on global features when constructing a shared feature space, but also fully utilizes the cross-modal matching information of local entities and attributes, enhancing the robustness and generalization ability in the text-to-image pedestrian retrieval task.

[0021] The technical solutions of the present application will be described in detail below in conjunction with the drawings and specific embodiments.

[0022] Figure 1 is a flowchart of the text-to-image pedestrian retrieval method based on mask denoising provided by the embodiments of the present application. As Figure 1 shown, the text-to-image pedestrian retrieval method based on mask denoising can specifically include: S101, obtain cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words; S102, mask and randomly replace the entity words and attribute words according to a set probability, respectively, to generate training text after mask replacement; S103, input the training text after mask replacement and the corresponding pedestrian image into a text encoder and an image encoder, respectively, to obtain a text feature vector and an image feature vector; S104, input the text feature vector and the image feature vector into a cross-modal interaction encoder to obtain a fused feature representation; S105, predict the original words at the masked positions based on the fused feature representation, and calculate a mask prediction loss; S106, calculate an image-text contrast learning loss based on the similarity relationship between the text feature vector and the image feature vector; S107, the mask prediction loss and the image-text contrastive learning loss are weighted and summed according to a preset weight to jointly optimize the parameters of the text encoder, the image encoder and the cross-modal interaction encoder, and a pedestrian retrieval model is obtained; S108, the pedestrian retrieval model is used to extract features of the query text and the to-be-retrieved pedestrian image library respectively, the similarity between the query text features and each to-be-retrieved pedestrian image features is calculated, and a ranking result is generated to output a target pedestrian image matched with the query text.

[0023] In some embodiments, the text description is subjected to semantic analysis, and entity words and attribute words are identified and labeled, including: The text description is subjected to word segmentation processing to obtain a text token sequence arranged in a word order; The text token sequence is input into a context encoding network based on a pre-trained language model to obtain semantic vectors of each text token; Each semantic vector is subjected to similarity matching with a preset entity category vector set and an attribute category vector set respectively to obtain entity category confidence and attribute category confidence; When the entity category confidence reaches a first threshold, the corresponding text token is labeled as an entity word; when the attribute category confidence reaches a second threshold and has not been labeled as an entity word, the corresponding text token is labeled as an attribute word; The token sequence containing the entity word label and the attribute word label is output for subsequent mask and same-class word random replacement steps.

[0024] Specifically, based on the public CUHK-PEDES dataset, the text description is subjected to semantic analysis to identify and label entity words and attribute words. This processing flow can be completed by calling the pre-trained language model BERT-Base-uncased in the Python development environment. The inference stage is performed on a single NVIDIA A40 graphics card, and the average inference delay is about 8 ms / sentence.

[0025] First, the English text description to be processed is input into the WordPiece tokenizer matched with the BERT model. The tokenizer outputs a one-dimensional token sequence in a word order, and each token carries original character interval offset information for subsequent label writing. For example, the text "The person wears a red backpack and white sneakers." is divided into [CLS], the, person, wears, a, red, back##pack, and, white, sneakers,., [SEP]. The sequence also retains the relationship between intra-word subwords, such as "back##pack".

[0026] Subsequently, the above-mentioned marked sequence is input into the BERT context coding network of the frozen parameters. After the network performs twelve-layer transformation on the sequence, a semantic vector with a dimension of 768 is generated for each mark in the output layer. The vector has been implicitly integrated into the intra-sentence context and can represent the local word meaning and the overall context.

[0027] Next, the entity category vector set and the attribute category vector set are constructed. The entity category covers 25 high-frequency wearable or accessory items such as backpacks, coats, and hats, and the attribute category covers 15 common description dimensions such as color, material, and length. For each category, 3-5 typical word entries are selected, and the word entry vectors are averaged after being obtained by the same BERT model to obtain the category prototype vector, denoted as the entity category vector Ve_i and the attribute category vector Va_j.

[0028] For any mark in the sequence, the semantic vector Vs and all Ve_i, Va_j are calculated for the cosine similarity to obtain the entity confidence pe = max(cos(Vs, Ve_i)) and the attribute confidence pa = max(cos(Vs, Va_j)). In this embodiment, the first threshold Te is set to 0.65, and the second threshold Ta is set to 0.55. When pe ≥ Te, the mark is labeled as an entity word; when pa ≥ Ta and the mark has not been labeled as an entity word, the mark is labeled as an attribute word; and the rest is not labeled.

[0029] For the aforementioned example text, the similarity between the mark "back##pack" and the category "backpack" is 0.82, which exceeds Te, so it is labeled as an entity word; the similarity between "red" and the color category is 0.78, which satisfies the attribute threshold and has not been labeled as an entity word, so it is labeled as an attribute word; "white" is labeled as an attribute word for the same reason; and the remaining marks remain unlabeled. The final generated label sequence is output in the original order for direct calling by the subsequent mask and synonym replacement steps. Throughout the process, the BERT parameters are not fine-tuned to ensure label consistency and reproducibility.

[0030] Through the implementation method of the above-mentioned embodiment, entity words and attribute words can be efficiently and stably labeled without manual annotation, providing accurate semantic anchors for subsequent mask denoising training.

[0031] In some embodiments, the entity words and attribute words are respectively subjected to masking and synonym random replacement according to a set probability to generate a mask-replaced training text, including: Randomly selecting entity words and attribute words to be masked in the text description based on preset first and second masking probabilities; Replacing the selected entity words with a first mask mark and replacing the selected attribute words with a second mask mark; For the text mark in the first mask mark position, a replacement word is randomly extracted from the entity synonym set, and the replacement word is used to replace the corresponding first mask mark; For the text mark in the second mask mark position, a replacement word is randomly extracted from the attribute synonym set, and the replacement word is used to replace the corresponding second mask mark; The original text mark sequence order is kept unchanged, and the training text after the mask and synonym random replacement is output.

[0032] Specifically, the double-probability mask mechanism of the embodiment includes: a first mask probability for controlling the mask frequency of entity words; and a second mask probability for controlling the mask frequency of attribute words; both of which can be independently set to meet the disturbance intensity requirements of different semantic granularities. In the training stage, the annotated entity words and attribute words are respectively sampled according to the probabilities within each sentence to ensure that the mask position distribution of each batch is random and long-term subject to the global expected value.

[0033] The mask mark setting includes: the first mask mark is dedicated to occupying the entity word; and the second mask mark is dedicated to occupying the attribute word; and the use of distinguishable marks can enable the subsequent interactive encoder to explicitly distinguish different types of missing information and avoid confusion.

[0034] The synonym random replacement strategy of the embodiment includes: for the masked position, instead of directly restoring the original word, a replacement word is randomly extracted from a candidate library of the same semantic category.

[0035] The candidate library is constructed offline in advance, including the following contents: the entity synonym set is archived according to categories such as “backpack”, “hat”, and “coat”; the attribute synonym set is aggregated in the same meaning or synonymous description according to colors, materials, lengths, etc.; and the random replacement can simulate the description errors and ambiguities in real scenarios to enable the model to learn the ability to exclude inconsistent noise.

[0036] The in-order output means that all mask and replacement operations are performed on the original mark sequence dimension, and after completion, the sequence is kept unchanged for direct output, ensuring that the text encoder can capture the context structure information using position encoding.

[0037] For example, in one specific example, taking the English text “A tall woman in a green coat carries a black handbag.” as an example, “woman”, “coat”, and “handbag” have been marked as entity words, and “tall”, “green”, and “black” have been marked as attribute words by the first embodiment. The first mask probability is set to 0.4, and the second mask probability is set to 0.3. The following steps are performed: I. Randomly select mask positions In the entity phrase, "handbag" hits the mask; in the attribute phrase, "green" hits the mask, and the rest of the labels remain unchanged.

[0038] II. Mask placeholder Replace "handbag" with the first mask label and "green" with the second mask label to form the intermediate sequence: "A tall woman in a [mask1] coat carries a black [mask1]." Note: If multiple entities are masked within the same text, use the same first mask label, and the same applies to attributes.

[0039] III. Random replacement of synonyms For the first mask label in position one, randomly select a word from the entity synonym set "bag class", and assume that "backpack" is selected. The replaced text segment becomes "backpack".

[0040] For the second mask label in position two, randomly select a word from the attribute synonym set "color", and assume that "yellow" is selected. The replaced text segment becomes "yellow".

[0041] IV. Order-preserving output The final masked and randomly replaced training text is: "A tall woman in a yellow coat carries a black backpack." This text, together with the original paired image, forms a new training pair. The original words "green" and "handbag" that were masked are used as supervision signals for the mask prediction task.

[0042] Through the method of the above embodiment, the embodiment introduces semantic-level noise without destroying the syntactic structure, providing rich, controllable, and realistic error distribution input samples for subsequent robust training of cross-modal models.

[0043] In some embodiments, the text feature vector and the image feature vector are input into a cross-modal interaction encoder to obtain a fused feature representation, including: The text feature vector and the image feature vector are concatenated in a predetermined order to form a joint sequence, and a corresponding modal identifier and position encoding are added to each vector in the joint sequence; The joint sequence is input into a cross-modal interaction encoder composed of multiple layers of self-attention sublayers and cross-attention sublayers stacked alternately to establish bidirectional attention association between the text modality and the image modality, and output an updated joint sequence; The fused vector located at the preset aggregation position is extracted from the updated joint sequence as the fused feature representation.

[0044] Specifically, in the present embodiment, the text feature vectors and the image feature vectors are first concatenated into a joint sequence in a fixed order of "text first, image second". For each vector in the sequence, the system additionally attaches two types of auxiliary information: one is a modality identifier, used to distinguish whether it comes from text or image, and the other is an absolute position encoding, used to maintain the original order or block sequence relationship. The modality identifier is implemented by concatenating a learnable binary embedding at the end of the vector, and the position encoding is calculated with the same formula as in the transformer pre-training stage to ensure that the downstream network can directly parse it.

[0045] After concatenation, the joint sequence is sent to the cross-modal interaction encoder. The encoder is structurally stacked by four layers of transformers, each of which contains, in turn, a text self-attention sublayer, an image self-attention sublayer, and a bidirectional cross-attention sublayer. The self-attention sublayers update the context information within their respective modalities, and the cross-attention sublayer allows the text vectors to focus on the image semantics while the image vectors focus on the text semantics through the query-key-value cross, thereby establishing a fine-grained alignment relationship in the same hidden space. Each sublayer is connected to a residual path and layer normalization to stabilize the gradient.

[0046] Further, when the joint sequence has passed through all four layers of interaction, the model reserves a CLS aggregation position at the beginning of the sequence for the text end, which will accumulate cross-modal context. The system directly extracts this CLS vector as the fusion vector output and records it as the final fusion feature representation. In this way, not only is the additional calculation of traversing all vectors for pooling avoided, but also the centralized expression of fusion information is ensured.

[0047] In some examples, taking the query sentence "Aman in a blue jacket holds a white umbrella." and its corresponding pedestrian image as an example, the text encoder first outputs a L=77-dimensional text feature sequence, and the image encoder outputs M=196 image block features, with a length of 273 after concatenation. After interaction encoder calculation, the CLS position finally converges the association information of "blue jacket", "white umbrella" and the corresponding coat and umbrella regions in the image. When calculating the cosine similarity in the subsequent retrieval stage, the CLS fusion vector can represent both the global appearance and the key details, thereby improving the cross-modal matching accuracy.

[0048] In some embodiments, based on the fusion feature representation, the original words at the masked positions are predicted, and a mask prediction loss is calculated, including: extracting the feature vector at the masked position in the corresponding fusion feature representation; inputting the feature vector into a prediction head containing a linear transformation layer and a normalization activation function in sequence to obtain a probability distribution for a preset vocabulary; The prediction error of a single mask position is calculated according to the cross-entropy between the probability distribution and the original word identification of the masked position; The prediction errors of all mask positions in the same training text are averaged to obtain the mask prediction loss corresponding to the training text.

[0049] Specifically, the fusion feature representation retains the cross-modal context information of each token in the text sequence, where the fusion vector corresponding to the masked token has fused image clues and text context. The system performs word restoration prediction separately for each mask position during the training phase. The specific process is as follows.

[0050] I. Extracting features of masked positions For a training text, the model generates equal-length fusion vectors for all text tokens in the cross-modal interaction stage. When the token is the first or second masked token, its fusion vector is directly extracted from the joint sequence according to the original sequence index, denoted as v_mask. The vector dimension is consistent with the hidden layer of the interaction encoder, which is set to d = 1024 in this embodiment.

[0051] II. Prediction head structure and output probability The extracted v_mask is sequentially passed through two linear mappings. The first mapping compresses the dimension from 1024 to 256 and applies GELU activation to enhance non-linear representation capability; the second mapping directly projects into a fixed vocabulary space with size V, which is consistent with the vocabulary of the text encoder pre-training, and V is 30522 in this embodiment. The projection result is normalized by Softmax to form a one-dimensional probability distribution p_mask, which is used to represent the confidence of each candidate word.

[0052] III. Unit loss calculation The system records the index id_true of the original token before masking in the vocabulary. The natural logarithm of the probability p_true corresponding to this index is taken, and the cross-entropy error e_mask is formed with -log(p_true). To keep the loss scale consistent for different sentence lengths, this embodiment does not introduce an additional label weight factor.

[0053] IV. Intra-sentence averaging to form text-level loss If a training text contains n mask positions, the system calculates e_1 to e_n respectively and then takes the arithmetic mean to obtain the mask prediction loss L_mask_text corresponding to the text, which is (e_1 + … + e_n) / n. The batch loss is then averaged over all texts to obtain the batch-level mask loss.

[0054] In one specific example, the previously masked randomly replaced text "A tall woman in a yellowcoat carries a black backpack." corresponds to the original words "green" and "handbag" are masked. After the interaction encoding, the system extracts v green and v handbag at the two indices. The prediction head outputs a prediction probability of 0.72 for "green" for v green, and the cross-entropy error is 0.331; for v handbag, it outputs a prediction probability of 0.64 for "handbag", and the error is 0.446. After averaging the two, the masked prediction loss for the sentence is 0.389. This value is used for backpropagation together with the loss of the rest of the text in the same batch, prompting the model to more accurately recover the masked information and strengthen the robustness of visual semantic alignment.

[0055] In another example, the embodiments of the present application can calculate the masked prediction loss using the following formula to assist in cross-modal feature learning and improve the robustness and generalization ability of the model. The specific formula is as follows: where M represents the masked position, , represents the one-hot encoding of the true class label, represents the predicted value.

[0056] In some embodiments, the text-image contrast learning loss is calculated based on the similarity relationship between the text feature vector and the image feature vector, including: extracting a text global vector and an image global vector from the text feature vector and the image feature vector, respectively; performing similarity calculation on each text global vector and all image global vectors, and performing similarity calculation on each image global vector and all text global vectors, to obtain a bidirectional similarity matrix; based on the text-image matching labels corresponding to the similarity matrix, using a normalized contrast loss function with a temperature coefficient, calculating the cross-entropy loss in the text-to-image direction and the image-to-text direction, respectively; averaging the cross-entropy loss in the text-to-image direction and the cross-entropy loss in the image-to-text direction to obtain the text-image contrast learning loss.

[0057] Specifically, in the present embodiment, the calculation of the image-text contrastive learning loss relies on a batch-level cross-modal similarity alignment mechanism. First, the text encoder retains a CLS token at the beginning of the output sequence to aggregate the global semantics of the text; the image encoder also sets a CLS token at the beginning of the input image block sequence to represent the global appearance of the image. The model extracts the hidden vectors corresponding to the two types of CLS tokens, respectively denoted as the text global vector and the image global vector. To ensure the stability of subsequent similarity measurement, each global vector is processed by L2 normalization to normalize its length.

[0058] Subsequently, within a single training batch, the system calculates the cosine similarity between the global vector of any text sample and all image global vectors in the batch one by one, and fills the obtained similarity values into the row or column direction of the matrix according to the arrangement order of the text index and the image index. The same operation is performed for any image sample, i.e., using the image vector as the query and all text vectors as the contrast target, to obtain another group of similarity matrices. In this way, the bidirectional similarity matrices in the "text to image" and "image to text" directions can be obtained simultaneously. To avoid too narrow numerical distribution, the two groups of matrices are divided by a temperature coefficient before Softmax normalization, which is set to 0.07 in the early stage of training and slowly reduced according to a pre-set schedule in the later stage to improve the discrimination of difficult samples.

[0059] The system determines the true matching pairs according to the batch label: the element in the ith row and the ith column is a positive sample, and the rest are negative samples. For the text to image direction, the model calculates the cross-entropy loss based on the probability distribution of each row after Softmax normalization; for the image to text direction, the same calculation is performed for each column. The average of the two loss values is the image-text contrastive learning loss of the batch, and the gradient will jointly drive the text encoder and the image encoder to tighten the positive alignment distance and expand the negative alignment distance in the same embedding space.

[0060] To deepen understanding, the following explanation uses a batch of three image-text pairing examples. Suppose that text T1 describes a "white shirt and blue jeans," corresponding to image I1 showing a person of the same color; text T2 describes a "red dress with a handbag," corresponding to image I2; and text T3 describes a "black coat and hat," corresponding to image I3. The model computes a six-way, nine-component similarity matrix from the three text CLS vectors and the three image CLS vectors. After temperature scaling and softmax, the probabilities at T1-I1, T2-I2, and T3-I3 are significantly higher than the remainder. The system treats the diagonal elements as positive target probabilities and the off-diagonal elements as zero labels, calculating both a row loss and a column loss. The average of these two losses is then used as the final image-text contrastive learning loss. This loss, during backpropagation, encourages positively aligned vectors like T1 and I2 to be closer together, while simultaneously reducing the similarity of mismatched pairs like T1-I2, thereby improving the overall discriminability of cross-modal embeddings.

[0061] Through the above mechanism, this embodiment constrains both text-to-image and image-to-text mapping paths at the batch granularity, enabling the model to accurately align cross-modal global semantics in a shared feature space and significantly enhance the discrimination of noise pairings, providing higher matching accuracy and more stable robustness for the subsequent retrieval stage.

[0062] In addition, in a specific example, the image-text contrastive learning loss is used to build a shared feature space between different modalities (image and text). By maximizing the similarity of true matching pairs and minimizing the similarity of non-matching pairs, cross-modal alignment and differentiation capabilities are improved, thereby enhancing the retrieval accuracy and robustness in the text-to-image pedestrian retrieval task. In this embodiment, the image-text contrastive learning loss can be calculated using the following formula: Among them, middle image to text Similarity, image i and text j are matched, image Similarity to text k, but not necessarily a match, represents the temperature coefficient, represents the number of training samples, The meaning is similar; represents the contrast loss of image to text, represents the contrastive loss from text to image, represents the total contrast loss.

[0063] In some embodiments, the mask prediction loss and the image-text contrastive learning loss are weighted and summed according to preset weights to jointly optimize the parameters of the text encoder, the image encoder and the cross-modal interaction encoder to obtain the pedestrian retrieval model, comprising: The mask prediction loss and the image-text contrastive learning loss are multiplied by a first weight coefficient and a second weight coefficient respectively, and the weighted results are summed to construct a total loss function; Based on the total loss function, gradient backpropagation is performed, and the obtained gradient is applied to the text encoder, the image encoder and the cross-modal interaction encoder in parallel to update the parameters of each encoder synchronously; According to a preset training iteration strategy, the total loss function construction and parameter updating are repeatedly performed until the convergence condition is met, and the pedestrian retrieval model is obtained, wherein the first weight coefficient and the second weight coefficient are dynamically adjusted according to the training progress according to a preset scheduling curve.

[0064] Specifically, in the training system of the present embodiment, the mask prediction loss and the image-text contrastive learning loss are considered simultaneously, and a unified optimization target is constructed by weighted sum. The mask prediction loss mainly promotes the model to recover the masked words, and strengthens the alignment of local entities and attributes; the image-text contrastive learning loss focuses on compressing the distance between cross-modal positive samples and expanding the interval between negative samples, and improves the global discrimination ability. In order to balance the importance of the two tasks at different stages of training, the system sets a first weight coefficient λ1 and a second weight coefficient λ2 for the two tasks respectively, and the sum of the two coefficients is kept as 1 to ensure the stability of the gradient amplitude.

[0065] At the beginning of training, λ1 is initialized to 0.7 and λ2 is initialized to 0.3. As the iteration proceeds, the model has basic alignment ability and needs to be further emphasized for cross-modal distinction, so λ2 is increased by 0.04 every epoch starting from the 15th epoch, and λ1 is correspondingly adjusted downward, until the 25th epoch reaches λ1=0.3 and λ2=0.7. The weight scheduling curve is written in the configuration file in advance, and the training scheduler automatically updates it at the epoch switching time to avoid inconsistent caused by manual intervention.

[0066] In each training step, the system selects 32 pairs of image-text samples to form a batch, obtains the mask prediction loss L_mask and the image-text contrastive loss L_contrast through forward propagation, and then calculates the total loss L_total=λ1·L_mask+λ2·L_contrast according to the current weight coefficients. The gradient is applied to the text encoder, the image encoder and the cross-modal interaction encoder through backpropagation, and the three types of networks share the same learning rate plan but maintain the momentum item independently. The optimizer uses AdamW, the base learning rate is 3×10 -5 , linear warm-up is used in the first 60% of the training steps, and cosine decay is used for the remaining steps until the convergence threshold 1×10 -6 .

[0067] For synchronous update, the system uses mixed precision training when the video memory is sufficient. After the gradient is aggregated on the main GPU, gradient clipping is uniformly performed, and then it is written back according to the weight decay rate of each network parameter group, ensuring that the learning speed of the text and visual branches is consistent. On the CUHK-PEDES dataset, a total of 30 epochs are trained. When the mAP of the validation set increases by less than 0.1% for 3 consecutive epochs, it is considered to have converged, and the training is terminated in advance and the pedestrian retrieval model is saved.

[0068] Through the above weight dynamic adjustment and joint optimization strategy, the model sufficiently learns the mask restoration ability and stabilizes the fine-grained alignment in the early stage, and gradually emphasizes global contrast discrimination in the later stage to eliminate noise interference. Finally, both local robustness and cross-modal discriminability are taken into account. In the test of the public dataset, the retrieval accuracy is improved by about 5 percentage points compared with the single loss baseline, and the training convergence speed is also improved by about 18%.

[0069] In some embodiments, a pedestrian retrieval model is used to extract features from the query text and the image library of the pedestrian to be retrieved, calculate the similarity between the query text features and the features of each pedestrian image in the image library, and generate a ranking result to output the target pedestrian image matched with the query text, including: inputting the query text into the text encoder optimized by joint optimization to obtain a query text feature vector; inputting each pedestrian image in the image library of the pedestrian to be retrieved into the image encoder optimized by joint optimization to obtain a corresponding set of pedestrian image feature vectors; calculating the similarity value between the query text feature vector and each pedestrian image feature vector in the set of pedestrian image feature vectors to generate a similarity list; descendingly sorting the set of pedestrian image feature vectors according to the size of the similarity value to obtain a ranking result containing the pedestrian image identifier and the similarity value; determining at least one pedestrian image with the highest similarity based on the ranking result as the target pedestrian image matched with the query text and outputting it.

[0070] Specifically, in this embodiment, the pedestrian retrieval system completes two stages of offline feature library construction and online retrieval in advance. In the offline stage, each pedestrian image in the image library is first scaled to a uniform resolution, keeping the long side at 256 pixels and not changing the aspect ratio, and then divided into fixed-size non-overlapping blocks and input into the image encoder optimized by joint optimization. The encoder retains the CLS token at the first block position and outputs a one-thousand-twenty-four-dimensional global vector. In order to eliminate the amplitude difference caused by different image brightness and contrast, the system performs L2 normalization on the vector to obtain an image feature with a module length of 1. The normalized feature is written into the vector index together with the image primary key.

[0071] If the size of the gallery is less than 50,000, the index is stored in pure dense vector storage and loaded into memory in segments; if the size is greater than 50,000, the index uses a block-based inverted file folder combined with product quantization coding. The specific steps are as follows: first, use 256 code clusters to perform two-level K-means clustering on the full feature, and record the pointer of each feature to the cluster center; then divide the residual vector into 16 subspaces, and quantize each into an 8-bit subcode; finally, store the cluster pointer and subcode string as a compressed index, and establish an inverted table according to the cluster center. The above operation is completed on a four-GPU workstation with 48GB of video memory. Indexing one million images takes about three hours and occupies 5.2GB of disk space.

[0072] In the online retrieval stage, the user inputs a natural language description as the query text. The system first calls the text encoder consistent with the training stage parameters to obtain the 1024-dimensional global vector of the sentence after WordPiece segmentation, and then performs L2 normalization to obtain the query vector q. If the index is full-dense storage, the system directly calculates the cosine similarity between q and each image feature v_i, the numerical formula is q·v_i, and the average time for calculating 100,000 similarities on a single A40 graphics card is 9ms. If the index is a compressed inverted structure, the system first calculates the Euclidean distance between q and the 256 cluster centers, selects the top ten centers with the smallest distance into the candidate pool, then decodes the residual subcode in the corresponding inverted table to restore the approximate v_i, and calculates the inner product with q to obtain the approximate cosine value. The size of the candidate pool accounts for about 4% of the total gallery, so even if the gallery contains one million images, the actual similarity calculation amount is still controlled at about 40,000, with an average time of 15ms, which is within the acceptable delay range.

[0073] The system sorts the obtained similarity values in descending order and generates a list of image primary keys corresponding to them. The top 50 items are returned by default for front-end display, and if the user needs more accurate results, the complete floating-point features of the top 50 items can be read again and recalculated with q. This "two-stage retrieval" balances system throughput and accuracy. After sorting, the system directly displays the image ranked first in similarity as the target pedestrian image to the user, and simultaneously lists the remaining candidates in the interface.

[0074] To illustrate the retrieval effect, the query text "amanwithabrownleatherjacketandredbackpack" is demonstrated. The text encoder outputs the vector q, which is compared with the index of one million images in the gallery, and the index returns the highest similarity image number #865432 with a score of 0.91 and the second highest number #342117 with a score of 0.86. The system further loads the accurate feature recalculation for these two images, and the recalculation scores are 0.927 and 0.835 respectively, and the ranking does not change. The final image #865432 is output to the user. Artificial checking shows that the image is indeed a man wearing a brown leather jacket and a red backpack, and the retrieval accuracy reaches the expected value.

[0075] The embodiment combines offline quantization index and online hierarchical retrieval, controls the total time of single retrieval in a million-level gallery within twenty milliseconds, maintains the Top1 accuracy and full-precision brute-force search basically consistent, and effectively guarantees the response speed and accuracy of text-to-image pedestrian retrieval in large-scale real-time application scenarios.

[0076] According to the technical solutions of the above embodiments of the present application, the present application has at least the following advantages: 1. Through entity and attribute level mask prediction training, the model can still accurately capture the visual-semantic correspondence when there are partial errors, omissions or ambiguous expressions in the text description, effectively alleviating the training deviation problem caused by incorrect image-text pairing and text noise in the real environment; 2. Can effectively resist the interference caused by incorrect image-text pairing and description noise in the real environment in the text-to-image pedestrian retrieval task, improve the fine-grained cross-modal alignment ability of the model through entity and attribute level mask prediction training, and realize significant improvement of retrieval accuracy and robustness; 3. Through entity and attribute mask prediction, combined with image-text contrast learning loss, the model has fine-grained cross-modal alignment ability while being supervised by local entity / attribute and global representation.

[0077] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0078] Figure 2 is a structural schematic diagram of the text-to-image pedestrian retrieval device based on mask denoising provided by the embodiment of the present application. As shown in Figure 2 The text-to-image pedestrian retrieval device based on mask denoising includes: The recognition module 201 is configured to obtain cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and label entity words and attribute words. The mask module 202 is configured to perform mask and synonym random replacement on the entity words and the attribute words respectively according to a set probability, to generate masked and replaced training texts; The encoding module 203 is configured to input the masked and replaced training texts and corresponding pedestrian images into a text encoder and an image encoder respectively, to obtain text feature vectors and image feature vectors; The fusion module 204 is configured to input the text feature vectors and the image feature vectors into a cross-modal interaction encoder, to obtain a fusion feature representation; The prediction module 205 is configured to predict original words at masked positions based on the fusion feature representation, to calculate a mask prediction loss; The calculation module 206 is configured to calculate a text-image contrast learning loss based on a similarity relationship between the text feature vectors and the image feature vectors; The joint module 207 is configured to perform weighted summation on the mask prediction loss and the text-image contrast learning loss according to a preset weight, to jointly optimize parameters of the text encoder, the image encoder and the cross-modal interaction encoder, to obtain a pedestrian retrieval model; The output module 208 is configured to extract features from a query text and a to-be-retrieved pedestrian image library by using the pedestrian retrieval model, to calculate similarities between a query text feature and each to-be-retrieved pedestrian image feature and generate a ranking result, to output a target pedestrian image matched with the query text.

[0079] In some embodiments, Figure 2 The recognition module 201 performs word segmentation processing on the text description, to obtain a text token sequence arranged in a word order; inputs the text token sequence into a context encoding network based on a pre-trained language model, to obtain semantic vectors of the text tokens; respectively performs similarity matching on each semantic vector and a preset entity category vector set and an attribute category vector set, to obtain entity category confidence and attribute category confidence; when the entity category confidence reaches a first threshold, marks the corresponding text token as an entity word; when the attribute category confidence reaches a second threshold and is not marked as an entity word, marks the corresponding text token as an attribute word; and outputs a token sequence containing entity word labels and attribute word labels, for subsequent mask and synonym random replacement steps.

[0080] In some embodiments, Figure 2The mask module 202 randomly selects the entity word and the attribute word to be masked in the text description based on the preset first mask probability and the second mask probability, respectively; replaces the selected entity word with a first mask mark, and replaces the selected attribute word with a second mask mark; randomly extracts a replacement word from the entity synonym set for the text mark at the first mask mark position, and replaces the corresponding first mask mark with the replacement word; randomly extracts a replacement word from the attribute synonym set for the text mark at the second mask mark position, and replaces the corresponding second mask mark with the replacement word; the original text mark sequence order is kept unchanged, and the training text after the mask and synonym random replacement is output.

[0081] In some embodiments, Figure 2 The fusion module 204 concatenates the text feature vector and the image feature vector into a joint sequence in a preset order, and adds a corresponding modal identifier and position encoding to each vector in the joint sequence; inputs the joint sequence into a cross-modal interaction encoder formed by alternately stacking multiple self-attention sublayers and cross-attention sublayers, to establish bidirectional attention association between the text modal and the image modal, and outputs an updated joint sequence; extracts a fusion vector located at a preset aggregation position from the updated joint sequence as a fusion feature representation.

[0082] In some embodiments, Figure 2 The prediction module 205 extracts a feature vector of a masked position in the corresponding fusion feature representation; inputs the feature vector into a prediction head containing a linear transformation layer and a normalization activation function in sequence, to obtain a probability distribution for a preset word table; calculates a prediction error of a single mask position according to the cross entropy between the probability distribution and the original word identifier of the masked position; averages the prediction errors of all mask positions in the same training text to obtain a mask prediction loss of the corresponding training text.

[0083] In some embodiments, Figure 2 The calculation module 206 extracts a text global vector and an image global vector from the text feature vector and the image feature vector, respectively; performs similarity calculation on each text global vector and all image global vectors within the same training batch, and performs similarity calculation on each image global vector and all text global vectors, to obtain a bidirectional similarity matrix; based on the text-image matching label corresponding to the similarity matrix, uses a normalized contrast loss function with a temperature coefficient to calculate the cross entropy loss in the text-to-image direction and the image-to-text direction, respectively; averages the cross entropy loss in the text-to-image direction and the cross entropy loss in the image-to-text direction to obtain a text-image contrast learning loss.

[0084] In some embodiments, Figure 2The joint module 207 multiplies the mask prediction loss and the image-text contrastive learning loss by a first weight coefficient and a second weight coefficient respectively, and sums the weighted results to construct a total loss function; gradient back propagation is performed based on the total loss function, and the obtained gradient is applied to the text encoder, the image encoder and the cross-modal interaction encoder in parallel to update the parameters of each other synchronously; the total loss function construction and parameter updating are repeatedly performed according to a preset training iteration strategy until the convergence condition is met, and a pedestrian retrieval model is obtained, wherein the first weight coefficient and the second weight coefficient are dynamically adjusted according to the training progress according to a preset scheduling curve.

[0085] In some embodiments, Figure 2 The output module 208 inputs the query text into the text encoder optimized jointly to obtain a query text feature vector; inputs each pedestrian image in the pedestrian image library to be retrieved into the image encoder optimized jointly to obtain a corresponding set of pedestrian image feature vectors; calculates a similarity value between the query text feature vector and each pedestrian image feature vector in the set of pedestrian image feature vectors to generate a similarity list; sorts the set of pedestrian image feature vectors in descending order according to the size of the similarity value to obtain a sorting result containing pedestrian image identifiers and similarity values; and determines at least one pedestrian image with the highest similarity as a target pedestrian image matching the query text based on the sorting result.

[0086] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0087] Figure 3 is a structural schematic diagram of an electronic device 3 provided by an embodiment of the present application. As shown in Figure 3 The electronic device 3 of this embodiment includes a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. The processor 301 implements the steps in each of the above method embodiments when executing the computer program 303. Alternatively, the processor 301 implements the functions of each module / unit in the above device embodiments when executing the computer program 303.

[0088] By way of example, the computer program 303 can be divided into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 303 in the electronic device 3.

[0089] The electronic device 3 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The electronic device 3 can include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art can understand that Figure 3 The electronic device 3 is only an example and does not constitute a limitation on the electronic device 3, and can include more or fewer components than shown, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, and the like.

[0090] The processor 301 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0091] The memory 302 can be an internal storage unit of the electronic device 3, for example, a hard disk or a memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the memory 302 can include both the internal storage unit and the external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or software functional unit. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0093] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0094] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0095] In the embodiments provided in the present application, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0096] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0097] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0098] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program may include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium.

[0099] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the technical solutions of the present application are described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A text-to-image pedestrian retrieval method based on mask denoising, characterized in that: include: Obtain cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and annotate entity words and attribute words; Masking and randomly replacing similar words are performed on the entity words and attribute words according to set probabilities to generate masked and replaced training text; Input the masked training text and the corresponding pedestrian image into the text encoder and image encoder respectively to obtain the text feature vector and image feature vector; Inputting the text feature vector and the image feature vector into a cross-modal interactive encoder to obtain a fused feature representation; Predicting the original word at the masked position based on the fused feature representation and calculating the mask prediction loss; Calculating the image-text contrast learning loss based on the similarity relationship between the text feature vector and the image feature vector; Performing a weighted summation of the mask prediction loss and the image-text contrast learning loss according to preset weights to jointly optimize the parameters of the text encoder, image encoder, and cross-modal interaction encoder to obtain a pedestrian retrieval model; The pedestrian retrieval model is used to extract features from the query text and the pedestrian image library to be retrieved, calculate the similarity between the query text features and the features of each pedestrian image to be retrieved, and generate a ranking result to output the target pedestrian image that matches the query text.

2. The method according to claim 1, characterized in that The semantic analysis of the text description to identify and label entity words and attribute words includes: Perform word segmentation on the text description to obtain a text tag sequence arranged in word order; Inputting the text tag sequence into a context encoding network based on a pre-trained language model to obtain a semantic vector for each text tag; Performing similarity matching on each of the semantic vectors with a preset entity category vector set and attribute category vector set to obtain entity category confidence and attribute category confidence; When the entity category confidence reaches the first threshold, the corresponding text tag is marked as an entity word; when the attribute category confidence reaches the second threshold and is not marked as an entity word, the corresponding text tag is marked as an attribute word; The output contains a tag sequence of entity word labels and attribute word labels for subsequent masking and random replacement of similar words.

3. The method according to claim 1, characterized in that The step of performing masking and random replacement of similar words on the entity words and attribute words according to set probabilities to generate masked and replaced training text includes: Based on the preset first mask probability and second mask probability, randomly selecting entity words and attribute words to be masked in the text description respectively; The selected entity words are replaced with the first mask token, and the selected attribute words are replaced with the second mask token; For the text mark at the position of the first mask mark, randomly extract a replacement word from the entity similar word set, and replace the corresponding first mask mark with the replacement word; For the text mark at the position of the second mask mark, randomly extract a replacement word from the attribute similar word set, and replace the corresponding second mask mark with the replacement word; Keep the order of the original text token sequence unchanged and output the training text with masks and random replacement of similar words.

4. The method according to claim 1, wherein The step of inputting the text feature vector and the image feature vector into a cross-modal interactive encoder to obtain a fused feature representation includes: splicing the text feature vector and the image feature vector into a joint sequence in a preset order, and adding a corresponding modality identifier and position code to each vector in the joint sequence; The joint sequence is fed into a cross-modal interaction encoder consisting of multiple layers of self-attention sublayers and cross-attention sublayers stacked alternately to establish a bidirectional attention relationship between the text modality and the image modality, and output an updated joint sequence; A fusion vector located at a preset aggregation position is extracted from the updated joint sequence as the fusion feature representation.

5. The method according to claim 1, wherein The predicting the original word at the masked position based on the fused feature representation and calculating the mask prediction loss includes: Extracting a feature vector corresponding to a masked position in the fused feature representation; Inputting the feature vectors into a prediction head including a linear transformation layer and a normalized activation function in sequence to obtain a probability distribution for a preset vocabulary; Calculating the prediction error for a single masked position based on the cross entropy between the probability distribution and the original word identity of the masked position; The prediction errors of all mask positions in the same training text are averaged to obtain the mask prediction loss of the corresponding training text.

6. The method according to claim 1, characterized in that The calculating of the image-text contrast learning loss based on the similarity relationship between the text feature vector and the image feature vector includes: Extracting a text global vector and an image global vector from the text feature vector and the image feature vector respectively; In the same training batch, similarity calculation is performed on each text global vector and all image global vectors, and similarity calculation is performed on each image global vector and all text global vectors to obtain a bidirectional similarity matrix; Based on the text-image matching labels corresponding to the similarity matrix, a normalized contrast loss function with a temperature coefficient is used to calculate the cross entropy loss for the text-to-image direction and the image-to-text direction respectively; The cross entropy loss in the text-to-image direction and the cross entropy loss in the image-to-text direction are averaged to obtain the image-text contrastive learning loss.

7. The method according to claim 1, characterized in that The mask prediction loss and the image-text comparison learning loss are weightedly summed according to preset weights to jointly optimize the parameters of the text encoder, image encoder, and cross-modal interaction encoder to obtain a pedestrian retrieval model, including: Multiplying the mask prediction loss and the image-text contrast learning loss by a first weight coefficient and a second weight coefficient respectively, and summing the weighted results to construct a total loss function; Performing gradient backpropagation based on the total loss function, and applying the obtained gradients to the text encoder, image encoder, and cross-modal interaction encoder in parallel to synchronously update their respective parameters; The total loss function construction and parameter update are repeatedly executed according to a preset training iteration strategy until the convergence condition is met, thereby obtaining the pedestrian retrieval model, wherein the first weight coefficient and the second weight coefficient are dynamically adjusted according to a preset scheduling curve based on the training progress.

8. The method according to claim 1, characterized in that The method of using the pedestrian retrieval model to extract features from the query text and the pedestrian image library to be retrieved, calculating the similarity between the query text features and the features of each pedestrian image to be retrieved, and generating a ranking result to output a target pedestrian image matching the query text includes: Input the query text into the jointly optimized text encoder to obtain the query text feature vector; Input each pedestrian image in the pedestrian image library to be retrieved into the jointly optimized image encoder to obtain a corresponding pedestrian image feature vector set; Calculating a similarity value between the query text feature vector and each pedestrian image feature vector in the pedestrian image feature vector set to generate a similarity list; Sorting the pedestrian image feature vector set in descending order according to the similarity values ​​to obtain a sorting result including pedestrian image identifiers and similarity values; At least one pedestrian image with the highest similarity is determined based on the ranking results and output as a target pedestrian image matching the query text.

9. A text-to-image pedestrian retrieval device based on mask denoising, characterized in that: include: A recognition module is used to obtain cross-modal training data containing pedestrian images and corresponding text descriptions, perform semantic analysis on the text descriptions, and identify and annotate entity words and attribute words; A masking module is used to perform masking and random replacement of similar words on the entity words and attribute words according to set probabilities to generate masked and replaced training text; An encoding module is used to input the masked training text and the corresponding pedestrian image into a text encoder and an image encoder respectively to obtain a text feature vector and an image feature vector; a fusion module, configured to input the text feature vector and the image feature vector into a cross-modal interactive encoder to obtain a fused feature representation; A prediction module, configured to predict the original words at the masked positions based on the fused feature representation and calculate the mask prediction loss; A calculation module, configured to calculate the image-text contrast learning loss based on the similarity relationship between the text feature vector and the image feature vector; A joint module, configured to perform a weighted summation of the mask prediction loss and the image-text contrast learning loss according to preset weights, so as to jointly optimize the parameters of the text encoder, the image encoder, and the cross-modal interaction encoder to obtain a pedestrian retrieval model; The output module is used to use the pedestrian retrieval model to extract features from the query text and the pedestrian image library to be retrieved, calculate the similarity between the query text features and the features of each pedestrian image to be retrieved, and generate a ranking result to output the target pedestrian image that matches the query text.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images