Soft label based noise robust text-to-image pedestrian retrieval method and device

Abstract

The application provides a soft label-based noise-robust text-to-image pedestrian retrieval method and device. The method comprises the following steps: calculating the cosine similarity based on the global features of the image and the global features of the text, and normalizing the cosine similarity to generate a soft label representing the confidence of the image-text pairing; assigning a sample weight to each training sample according to the soft label, and combining a dynamic weight factor that increases with the training process to obtain a joint weight for the current iteration; constructing a cross-modal contrast learning loss and a similarity distribution matching loss using the joint weight, and weighting and summing the cross-modal contrast learning loss and the similarity distribution matching loss to obtain a total loss function; updating the parameters of the image encoder and the text encoder using the total loss function until the training converges to obtain a cross-modal alignment model. The application can realize robust cross-modal alignment and improve the pedestrian retrieval accuracy in a noisy environment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a noise-robust text-to-image person retrieval method and device based on soft labels. BACKGROUND

[0002] Text-to-image person retrieval (TIPR) is a cross-modal retrieval technology that uses natural language description as a query condition to locate target pedestrians from a large-scale pedestrian image library. Compared with person re-identification (Re-ID) which relies on structured image queries, TIPR has the advantages of flexible query method and low professional requirement for users, and can provide fast and accurate target search capability in public security, smart city and other occasions, so it has attracted widespread attention from academia and industry.

[0003] In order to realize effective alignment between image and text modalities, existing TIPR research mainly develops along two paths of global matching and local matching. The global matching method directly aligns the embedding vectors of the entire image and the complete text sentence by constructing a cross-modal loss function, which is simple in algorithm but ignores fine-grained differences. The local matching method further corresponds the pedestrian image region and the text segment one by one, thereby enhancing the recognition ability of details such as clothing color and carried objects; among them, explicit local matching relies on external detection or word segmentation module to complete region alignment, which has large computational complexity in the reasoning stage, while implicit local matching implicitly learns the region correspondence within the network through attention mechanism, which has higher reasoning efficiency. In order to improve the stability of the model in the noisy environment, researchers introduce a noise-robust learning framework: one method uses the memory effect of deep network "learning easy first and difficult later" for sample selection, gradually filtering out noisy data; another method designs a robust loss function with higher noise tolerance, directly suppressing the gradient interference of incorrect labels at the objective function level.

[0004] Existing TIPR methods generally assume that the text and image in the training data are one-to-one and accurately labeled, which is difficult to deal with incorrect image-text pairs caused by posture changes, shooting angles, lighting conditions or incomplete descriptions in actual scenarios, thereby significantly reducing the retrieval accuracy in complex environments. At the same time, most existing noise-robust researches are focused on single-modal classification, lacking special design for cross-modal many-to-many pairing features, which makes it difficult to accurately identify and remove incorrect corresponding relationships when directly transplanted to the TIPR task, and also lacks an efficient implementation mechanism in large-scale retrieval scenarios, resulting in that the model robustness and reasoning efficiency are difficult to meet the practical needs. SUMMARY

[0005] Therefore, the embodiment of the present application provides a soft label based noise robust text-to-image pedestrian retrieval method and device to solve the problems of incorrect text-image pairings, noise interference and insufficient retrieval robustness in the prior art.

[0006] In a first aspect, the embodiment of the present application provides a soft label based noise robust text-to-image pedestrian retrieval method, which comprises: inputting pedestrian images and corresponding text descriptions in a training set into an image encoder and a text encoder respectively to obtain image global features and text global features; calculating cosine similarity based on the image global features and the text global features, and normalizing the cosine similarity to generate soft labels representing text-image pairing confidence; assigning sample weights to each training sample according to the soft labels, and combining a dynamic weight factor that increases with the training process to obtain joint weights for the current iteration; constructing cross-modal contrast learning loss and similarity distribution matching loss using the joint weights, and weighting and summing the cross-modal contrast learning loss and the similarity distribution matching loss to obtain a total loss function; updating parameters of the image encoder and the text encoder using the total loss function until the training converges to obtain a cross-modal alignment model; in the retrieval stage, inputting a query text into the text encoder to obtain query features, inputting a pedestrian image to be retrieved into the image encoder to obtain pedestrian features, sorting according to the cosine similarity between the query features and the pedestrian features, and outputting a target pedestrian image that best matches the query text.

[0007] In a second aspect, the embodiment of the present application provides a soft label based noise robust text-to-image pedestrian retrieval device, which comprises: an input module configured to input pedestrian images and corresponding text descriptions in a training set into an image encoder and a text encoder respectively to obtain image global features and text global features; a normalization module configured to calculate cosine similarity based on the image global features and the text global features, and normalize the cosine similarity to generate soft labels representing text-image pairing confidence; a generation module configured to assign sample weights to each training sample according to the soft labels, and combine a dynamic weight factor that increases with the training process to obtain joint weights for the current iteration; a weighting module configured to construct cross-modal contrast learning loss and similarity distribution matching loss using the joint weights, and weight and sum the cross-modal contrast learning loss and the similarity distribution matching loss to obtain a total loss function; an update module configured to update parameters of the image encoder and the text encoder using the total loss function until the training converges to obtain a cross-modal alignment model; and an output module configured to, in the retrieval stage, input a query text into the text encoder to obtain query features, input a pedestrian image to be retrieved into the image encoder to obtain pedestrian features, sort according to the cosine similarity between the query features and the pedestrian features, and output a target pedestrian image that best matches the query text.

[0008] In a third aspect, the embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the above method when executing the computer program.

[0009] In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the above method when executed by a processor.

[0010] The above at least one technical scheme adopted by the embodiment of the present application can achieve the following beneficial effects:

[0011] By inputting the pedestrian image and the corresponding text description in the training set into the image encoder and the text encoder respectively, the image global feature and the text global feature are obtained; the cosine similarity is calculated based on the image global feature and the text global feature, and the cosine similarity is normalized to generate a soft label representing the confidence of the image-text pairing; the soft label is used to assign a sample weight to each training sample, and a dynamic weight factor that increases with the training process is combined to obtain a joint weight for the current iteration; the joint weight is used to construct a cross-modal contrast learning loss and a similarity distribution matching loss, and the cross-modal contrast learning loss and the similarity distribution matching loss are weighted and summed to obtain a total loss function; the parameters of the image encoder and the text encoder are updated using the total loss function until the training converges, and a cross-modal alignment model is obtained; in the retrieval stage, the query text is input into the text encoder to obtain the query feature, the pedestrian image to be retrieved is input into the image encoder to obtain the pedestrian feature, and the cosine similarity between the query feature and the pedestrian feature is used for sorting to output the target pedestrian image that best matches the query text. The present application can realize robust cross-modal alignment and improve the pedestrian retrieval accuracy in a noisy environment. BRIEF DESCRIPTION OF DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0013] Figure 1 is a flowchart of the soft label-based noise-robust text-to-image pedestrian retrieval method provided by the embodiment of the present application;

[0014] Figure 2 is a structural diagram of the soft label-based noise-robust text-to-image pedestrian retrieval device provided by the embodiment of the present application;

[0015] Figure 3FIG. 1 is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0016] In the following description, for the purpose of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

[0017] In the prior art, text-to-image pedestrian retrieval mainly relies on global or local alignment strategies of visual and text features, and constructs a common embedding space through contrast learning or region matching to improve image-text matching accuracy. In order to reduce the influence of noise, some studies attempt to use sample selection or robust loss function, but these methods are mostly derived from single-modal classification field and do not fully consider the characteristics of cross-modal many-to-many pairing, so the efficiency and stability are limited in large-scale retrieval tasks.

[0018] In actual application scenarios, there are often differences in posture, angle and illumination in pedestrian images, and the text description may be brief or inaccurate, resulting in incorrect pairing of training data. The existing TIPR model usually defaults to correct pairing and lacks a targeted noise suppression mechanism, so the retrieval accuracy is significantly reduced in complex environments, making it difficult to meet the high reliability requirements of public safety and the like.

[0019] To solve the above problems, the present application proposes a noise-robust text-to-image pedestrian retrieval method based on soft labels: first, a soft label is generated using the cosine similarity of image global features and text global features, and a continuous confidence is introduced to characterize the pairing reliability; then, combined with the soft label and a dynamic weight factor that increases with the training process, a joint weight is constructed and embedded into the cross-modal contrast learning loss and similarity distribution matching loss to form a whole differentiable optimization target; the image encoder and the text encoder are jointly updated through progressive weight regulation to obtain a robust cross-modal alignment model; in the retrieval stage, approximate nearest neighbor indexing is used to accelerate text query and pedestrian image matching.

[0020] The present application scheme can continuously suppress noise gradient interference, maintain stable training and improve cross-modal alignment accuracy in the presence of a large number of incorrect image-text pairings and label noise, thereby significantly improving the pedestrian retrieval accuracy and system generalization ability in a noisy environment.

[0021] The content of the technical scheme of the present application will be described in detail below in combination with the drawings and specific embodiments.

[0022] Figure 1is a flowchart of a soft label based noise robust text-to-image pedestrian retrieval method provided by the embodiment of the present application. As shown in Figure 1 The soft label based noise robust text-to-image pedestrian retrieval method can specifically include the following steps.

[0023] S101, inputting the pedestrian images and corresponding text descriptions in the training set into the image encoder and the text encoder respectively to obtain image global features and text global features;

[0024] S102, calculating the cosine similarity based on the image global features and the text global features, and normalizing the cosine similarity to generate soft labels representing the confidence of the image-text pair;

[0025] S103, assigning a sample weight to each training sample according to the soft label, and combining a dynamic weight factor that increases with the training process to obtain a joint weight for the current iteration;

[0026] S104, constructing a cross-modal contrast learning loss and a similarity distribution matching loss using the joint weight respectively, and weighting and summing the cross-modal contrast learning loss and the similarity distribution matching loss to obtain a total loss function;

[0027] S105, updating the parameters of the image encoder and the text encoder using the total loss function until the training converges to obtain a cross-modal alignment model;

[0028] S106, in the retrieval stage, inputting the query text into the text encoder to obtain query features, inputting the pedestrian image to be retrieved into the image encoder to obtain pedestrian features, and sorting according to the cosine similarity between the query features and the pedestrian features to output the target pedestrian image that best matches the query text.

[0029] In some embodiments, the pedestrian images and corresponding text descriptions in the training set are input into the image encoder and the text encoder respectively to obtain image global features and text global features, including:

[0030] Each pedestrian image is size standardized and divided into a plurality of non-overlapping image blocks according to a preset size to obtain an image block sequence, and a classification mark is added at the beginning of the image block sequence;

[0031] The image block sequence containing the classification mark is input into the multi-layer visual transformation module of the image encoder to output an image feature sequence, wherein the feature vector corresponding to the classification mark is taken as the image global feature;

[0032] Each text description is segmented to obtain a text mark sequence, and a start mark is added at the beginning of the text mark sequence and an end mark is added at the end of the text mark sequence;

[0033] The text mark sequence containing the start mark and the end mark is input into a sequence transformation module of the text encoder, and a text feature sequence is output, wherein a feature vector corresponding to the end mark is taken as a global feature of the text.

[0034] Specifically, in some examples, first, pedestrian images and corresponding text descriptions in a public dataset CUHK-PEDES are selected as training samples. To ensure the consistency of feature extraction, each pedestrian image is scaled to 224x224 pixels by bilinear interpolation, and is divided into M=196 non-overlapping image blocks of 16x16 pixels in row priority order. Then, a classification mark is inserted at the beginning of the image block sequence, which is used to carry global visual information in the subsequent network.

[0035] In order to map the discrete pixel blocks to a unified embedding space, the mean pixel vector of each image block is calculated, and a fixed-dimensional block embedding vector is generated through a linear transformation. At the same time, a learnable position embedding is added to the block sequence and the classification mark to explicitly indicate the spatial order of each element in the sequence. The processed sequence is input into a multi-layer visual transformation module composed of twelve stacked self-attention sublayers, each of which contains a multi-head self-attention unit, a feedforward network, a residual connection, and a layer normalization structure. In the sequence vector output by the module, the vector corresponding to the classification mark at the beginning is explicitly supervised as the global feature of the image during the training phase, which is used for subsequent cross-modal alignment.

[0036] On the text side, first, each natural language description is segmented, and a dictionary-based segmentation algorithm is used to keep the original word when encountering proper nouns or numerical combinations. If the length of the obtained text mark sequence is less than L_max, placeholders are filled at the end to keep the sequence equal in length. Then, a start mark is inserted at the beginning of the text mark sequence, and an end mark is inserted at the end, both of which are used to provide sentence-level semantic boundary information. All text marks are mapped to fixed-dimensional word embedding vectors through a lookup table, and a corresponding learnable position embedding is added, and then input into a twelve-layer stacked sequence transformation module. The module structure is consistent with the visual transformation module, except that the attention mechanism dimension and the word embedding dimension are shared, and it also contains multi-head self-attention, feedforward network, residual connection, and layer normalization. In the sequence vector output by the module, the vector corresponding to the end mark is extracted as the global feature of the text.

[0037] During the training process, to avoid gradient vanishing or explosion, the embedding dimension on both the visual and text sides is uniformly set to D=512; the multi-head self-attention unit uses H=8 attention heads; the hidden layer dimension of the feedforward network is 4D, and the element-wise linear rectified activation function is applied to each layer. To improve the convergence speed and prevent overfitting, a random dropout operation is added after all transformation layers, with a dropout rate of 0.1.

[0038] The feature extraction process of the present application is described below in conjunction with a specific example, which can include the following:

[0039] The training set of 34,054 images and corresponding 68,108 text descriptions are imported, and CLIP is used for feature extraction and subsequent training. The images are divided into non-overlapping image blocks, and image representations are obtained through an image encoder, where is the global representation of the image, is the representation of the i-th image block. The text is segmented into L text tokens, and feature representations are obtained through a text encoder, respectively. Among them is the global representation, is the feature representation corresponding to the i-th token.

[0040] The above embodiment realizes the unified sequence structure of "global feature-local feature" coexistence in image and text modalities by inserting classification markers in front of the visual sequence and boundary markers at both ends of the text sequence; the multi-layer self-attention network can automatically model long-range dependencies within the sequence, thereby providing accurate and reliable image global features and text global features for subsequent soft label generation and cross-modal alignment.

[0041] In some embodiments, the cosine similarity is calculated based on the image global feature and the text global feature, and the cosine similarity is normalized to generate a soft label representing the confidence of image-text pairing, including:

[0042] The image global feature and the text global feature are respectively normalized by two norms to obtain a first normalized vector and a second normalized vector;

[0043] The cosine similarity is calculated based on the first normalized vector and the second normalized vector;

[0044] The cosine similarity is input into a sigmoid mapping function and the output range is limited to obtain a soft label representing the confidence of image-text pairing.

[0045] Specifically, the soft label refers to a continuous confidence with a value range of 0-1, which is used to represent the reliability of image-text pairing; compared with the hard label which can only represent matching or not matching, the soft label can provide more fine-grained supervision signals for the model. The temperature coefficient is a learnable scaling factor that controls the smoothness of the confidence distribution by adjusting the slope of the cosine similarity in the sigmoid mapping function. The sigmoid mapping function is a class of monotonically increasing functions, which is used in this scheme to map the cosine similarity with a value between -1 and 1 to the 0-1 interval, and the common form is σ=1 / (1+e^{-α·x}).

[0046] For example, in some examples, the soft label can be generated in the following way:

[0047] Calculate the global feature of the image Cosine similarity with the global feature of the text , and use the Sigmoid function to normalize the cosine similarity to the soft label in the interval [0, 1], and get the final soft label for supervised training .

[0048]

[0049] where, is the sigmoid function, followed by the formula expansion, is the final true label, is the cosine similarity of the image-text pair.

[0050] The implementation method of the embodiment will be explained and described in combination with specific examples, which can include the following contents:

[0051] First, the global feature vector v_img of the collected image and the global feature vector v_txt of the text are both 512 dimensions. First, calculate the two norms of v_img and v_txt respectively, and if the two norms are less than 1×10^{-12} then set them to 1×10^{-12} to avoid division by zero. Then divide the elements of the respective two norms to get the first normalized vector u_img and the second normalized vector u_txt. The purpose of this step is to unify the dimensions of the two modal features, so that the subsequent similarity measurement is not affected by the absolute value size.

[0052] Next, multiply the corresponding dimension elements of u_img and u_txt and sum them up to get the cosine similarity sim. Since both vectors are on the unit hypersphere, the theoretical value range of sim is -1 to 1. In the extreme noise scenario, to prevent gradient explosion, this embodiment clips sim to limit it to -0.99 to 0.99.

[0053] Then, in order to convert sim to continuous confidence, a temperature coefficient α is introduced. α is set to 5 at the beginning of training and decays exponentially to 1 with the number of training rounds. Multiply sim by α and send it to the S-shaped mapping function σ=1 / (1+e^{-α·sim}) to get the soft label s_label. To eliminate numerical overflow or underflow, s_label is limited to the interval [0.01, 0.99]. The closer the value is to 1, the higher the matching degree of the text and the image, and the closer the value is to 0, the lower the matching reliability.

[0054] In actual training, a batch contains N image-text samples. For each pair of samples in the batch, the above normalization, similarity calculation and mapping process are repeated to form a soft label sequence S batch with length N. The current value of the temperature coefficient a is broadcast synchronously to all samples in the batch to ensure that the gradient can be correctly returned to a during backpropagation, achieving adaptive curvature adjustment.

[0055] Finally, the generated S batch is combined with the static sample weight and input into the weight mapping function to provide continuous confidence information for subsequent cross-modal contrastive learning loss and similarity distribution matching loss. Under the regulation of the dynamic weight factor, the model pays more attention to high-confidence samples in the early stage, and gradually absorbs low-confidence samples as the discrimination ability improves, thereby balancing noise suppression and effective learning.

[0056] Through the above implementation, the model can autonomously identify the reliability of the pairing in the training data with erroneous image-text pairings, high-confidence samples receive greater gradient update weights, and the negative impact of low-confidence samples is significantly weakened. Ultimately, more robust cross-modal alignment is achieved in the presence of noise, and the accuracy of pedestrian retrieval is improved.

[0057] In some embodiments, a sample weight is assigned to each training sample according to the soft label, and a dynamic weight factor that increases with training progress is combined to obtain a joint weight for the current iteration, including:

[0058] The soft label is input into a pre-set weight mapping function to calculate the static sample weight according to the value range of the soft label;

[0059] The incremental function is called according to the current training round to generate the dynamic weight factor, which monotonically increases with the training round and remains constant after reaching the upper limit;

[0060] The static sample weight and the dynamic weight factor are combined by weighting to obtain the joint weight for the current iteration.

[0061] Specifically, in the cross-modal robust training process, the soft label reflects the confidence of image-text pairing, and the static sample weight is used to measure the prior importance of a single sample in the current batch; the dynamic weight factor is a global adjustment coefficient that increases with the training round, used to control the balance between noise suppression and information utilization. By multiplying the static sample weight and the dynamic weight factor, an adaptive joint weight that changes with the training process can be obtained to guide the gradient update direction.

[0062] The batch soft label S_batch is input into the weight mapping function f1(·). To maintain the monotonicity of the mapping and highlight high-confidence samples, the embodiment selects an exponential amplification form f1(σ)=σ^γ, where σ is a single soft label and γ is a fixed amplification coefficient. The selection of γ needs to consider two aspects: if γ is too small, it cannot effectively distinguish sample weights, and if γ is too large, it is easy to cause gradient sparsity. After several experiments, when γ is 2, a good balance between noise suppression and sample utilization can be achieved. Taking samples with σ of 0.9 as an example, the static weight after mapping is 0.81; the weight of samples with σ of 0.3 after mapping is only 0.09, and the priority of high-confidence is significantly improved.

[0063] The dynamic weight factor is generated by an increasing function f2(t), where t represents the round to which the training has progressed. The increasing function adopts a piecewise linear form: it increases by a step η round by round within the first K rounds, and then remains unchanged at the maximum value β. If K is set to 5, η is set to 0.1, and β is set to 1.0, then f2(1)=0.1 at the first round, f2(5)=0.5 at the fifth round, and f2 is 1.0 at the sixth round and thereafter. This design makes the model more sensitive to noise in the early stage, with high-confidence samples dominating; as the rounds increase, low-confidence samples are gradually included in learning, improving feature coverage.

[0064] For the ith sample within the batch, its joint weight w_i is obtained by multiplying the static weight s_i and the current dynamic factor d_t, i.e., w_i=s_i·d_t. If the soft label σ_i of a certain sample is 0.7 in the third round of training, the corresponding static weight s_i≈0.49, and the current dynamic factor d3=0.3, then w_i≈0.147; for the same sample in the seventh round, d7 has reached 1.0, and the joint weight has increased to 0.49. This approach ensures that the weight grows smoothly with the rounds, while always maintaining the relative confidence differences between different samples.

[0065] In an example with a batch size of 4, the soft labels are 0.95, 0.60, 0.40, and 0.15, respectively. When γ is 2, the static weights are 0.90, 0.36, 0.16, and 0.02. If it is currently the fourth round of training, the dynamic factor d4=0.4, and the joint weights are 0.36, 0.14, 0.06, and 0.008, respectively. When updating the gradient, high-confidence samples obtain the maximum weight, and low-confidence samples are effectively suppressed; by the tenth round, the dynamic factor has reached 1.0, and the joint weight has returned to the static weight itself, achieving full exploitation of information for all samples.

[0066] In the early stages of training, a gradual learning strategy is used to prevent the model from converging to a wrong solution due to noisy labels. Dynamic sample weight adjustment is used in combination with soft label confidence to ensure that high-confidence samples obtain greater weight, and low-confidence samples are suppressed in the early stages, improving the proportion of effective samples and gradually stabilizing training.

[0067]

[0068] wherein, is the weight of the current training round, is the current training round, is the total training round, is the hyperparameter.

[0069]

[0070] wherein, is the final label weight of the sample, is the weight of the current training round, is the sample label.

[0071] By fusing the static sample weight with the dynamic weight factor round by round, the embodiment significantly reduces the interference of noise samples on the gradient in the early training, prevents the model from producing false convergence, gradually increases the weight of low confidence samples in the middle and late stages, enhances feature coverage, and finally realizes robust and efficient cross-modal alignment, improving the overall accuracy and robustness of text-to-image pedestrian retrieval in a noisy environment.

[0072] In some embodiments, a joint weight is used to construct a cross-modal contrast learning loss and a similarity distribution matching loss, respectively, and the cross-modal contrast learning loss and the similarity distribution matching loss are weighted and summed to obtain a total loss function, including:

[0073] The image global features and the text global features in the current training batch are combined two by two to calculate the cosine similarity and obtain a similarity matrix combined with a temperature coefficient;

[0074] According to the similarity matrix, matched pairs and non-matched pairs are distinguished, and the joint weight is multiplied to the corresponding similarity score to construct a cross-modal contrast learning loss;

[0075] The image modality similarity distribution and the text modality similarity distribution are extracted from the similarity matrix, respectively, and the difference between the image modality similarity distribution and the text modality similarity distribution is calculated using a distribution difference measurement function and a joint weight to obtain a similarity distribution matching loss;

[0076] According to a preset loss coefficient, the cross-modal contrast learning loss and the similarity distribution matching loss are weighted and summed to obtain a total loss function.

[0077] Specifically, the present embodiment revolves around the core concepts of "cross-modal contrastive learning loss", "similarity distribution matching loss", "temperature coefficient", and "distribution difference measure function". The cross-modal contrastive learning loss is a target function that measures the difference between matching pairs and non-matching pairs in the same batch. By maximizing the similarity of matching pairs and minimizing the similarity of non-matching pairs, the image modality and the text modality maintain semantic consistency in the common embedding space. The similarity distribution matching loss focuses on the consistency of the overall similarity distribution of the two modalities, avoiding optimization of only local samples. The temperature coefficient is used to scale the cosine similarity and control the degree of softening the gradient. The distribution difference measure function is a class of differentiable functions that measure the difference between two probability distributions, and common forms include log-likelihood difference or cross-entropy difference.

[0078] Let the current batch size be 4, corresponding to the image global feature set V_img = {v1, v2, v3, v4} and the text global feature set V_txt = {t1, t2, t3, t4}. First, normalize all features to two norms, then calculate the cosine similarity between v_i and t_j according to the row priority rule, and combine them into a 4x4 similarity matrix S. To amplify the similarity difference between same-class samples and different-class samples, multiply each element in S by the temperature coefficient α. Set α to 5 at the beginning of training and decay it to 1 round by round to ensure stable gradients and provide sufficient optimization pressure to difficult samples in the later stage.

[0079] The main diagonal elements of the similarity matrix S are considered as the similarity of matching pairs, and the non-diagonal elements are considered as the similarity of non-matching pairs. Apply Softmax to the i-th row S_i to get the probability distribution of image retrieval when the text is the query, and apply Softmax to the j-th column S^j to get the probability distribution of text retrieval when the image is the query. Define that when querying, the probability of matching pairs should be the maximum and the probability of non-matching pairs should be the minimum. Use the joint weight w to bind the row loss and column loss of each image-text sample: row loss L_row_i = -w_i·log(p_ii), column loss L_col_j = -w_j·log(p_jj). Sum the 4 row losses and 4 column losses and divide by 2N to get the cross-modal contrastive learning loss L_contrast, which makes high-weight samples contribute more to the loss and low-weight samples have a relatively suppressed impact on the gradient.

[0080] The image modal similarity distribution P_img is extracted from S by averaging the softmax results for each row; the text modal similarity distribution P_txt is extracted by averaging the softmax results for each column. The difference between the two modal distributions is calculated using the distribution difference metric function D(P_img||P_txt)=∑_k P_img(k)·log(P_img(k) / P_txt(k)). To highlight the contribution of high-confidence samples, the mean of the intra-batch joint weights w is multiplied by the distribution difference to obtain the similarity distribution matching loss L_align=w·D(P_img||P_txt). This design ensures the consistency of the overall geometric structure while also considering sample confidence.

[0081] Assuming the loss harmonic coefficient λ is 0.5, the total loss function L_total is obtained by weighted summation according to L_total = λ·L_contrast + (1-λ)·L_align. In this case, the cross-modal contrastive learning loss dominates sample-level pairing optimization, while the similarity distribution matching loss corrects the overall modal structure. These two factors complement each other and jointly guide the model convergence.

[0082] In an example batch, assume the joint weights of the four samples are 0.90, 0.50, 0.20, and 0.05, respectively. With an initial temperature coefficient α of 5, we calculate L_contrast ≈ 0.18, the difference between the distributions of P_img and P_txt is D ≈ 0.04, and the average weight w ≈ 0.41, resulting in L_align ≈ 0.0164. Taking λ = 0.5, we get L_total ≈ 0.0982. As training progresses, when the temperature coefficient decreases to 1 and the joint weights tend to be averaged, the values ​​of L_contrast and L_align gradually converge, allowing the model to balance pairing accuracy and distribution consistency.

[0083] In some examples, image-text contrastive learning loss is used to construct a shared feature space across different modalities (images and text). By maximizing the similarity of true matching pairs and minimizing the similarity of non-matching pairs, it enhances cross-modal alignment and discrimination capabilities, thereby improving retrieval accuracy and robustness in text-to-image person retrieval tasks.

[0084]

[0085]

[0086]

[0087] exist middle image To text The similarity score indicates that image i and text j are a match. image similarity to text k, but not necessarily matching, represents the corresponding soft label (i.e., in the last formula , represents the temperature coefficient, represents the number of training samples, has a similar meaning; represents the image-to-text contrastive loss, represents the text-to-image contrastive loss, represents the total contrastive loss.

[0088] Weighted similarity distribution matching loss. According to the cosine similarity distribution of the image-text pair embedding features divergence is introduced to associate the representations in different modalities.

[0089]

[0090]

[0091]

[0092] where, is the image to text similarity, represents the corresponding soft label, is a very small number to prevent division by zero, represents the number of training samples, represents the KL divergence, followed by the formula expansion, represents the image-to-text loss, represents the text-to-image loss, is the final loss.

[0093] Finally, according to the above loss function, the overall optimization is carried out, and the total loss is as follows:

[0094]

[0095] where, is a hyperparameter, is the weighted image-text contrastive learning loss above, is the weighted similarity distribution matching loss above.

[0096] By introducing the joint weight to participate in the construction of the two types of losses, the embodiment realizes the reinforcement learning of high-confidence samples and the global correction of the overall distribution consistency at the gradient level, significantly reduces the interference of noise samples on the paired alignment, and ensures the consistency of the structure of the two modal embedding spaces, thereby improving the accuracy and robustness of the text-to-image pedestrian retrieval task in a complex noise environment.

[0097] In some embodiments, the parameters of the image encoder and the text encoder are updated using the total loss function until training convergence, and a cross-modal alignment model is obtained, comprising:

[0098] The gradient of the image encoder parameters and the gradient of the text encoder parameters are calculated using the total loss function respectively;

[0099] According to the preset gradient descent optimization algorithm and the learning rate scheduling strategy, the image encoder parameters are iteratively updated using the gradient of the image encoder parameters, and the text encoder parameters are iteratively updated using the gradient of the text encoder parameters;

[0100] At the end of each training round, the total loss function and the preset evaluation index are calculated based on the validation dataset to determine whether the convergence condition is met;

[0101] When the convergence condition is met, the iteration is stopped, and the cross-modal alignment model containing the updated image encoder parameters and text encoder parameters is output.

[0102] Specifically, for the key link of "using the total loss function to drive parameter update until training convergence", the core concepts and overall process are first summarized, then the gradient calculation method, optimization algorithm and learning rate scheduling strategy, convergence criterion and early stopping mechanism are elaborated, finally an example training process is given combined with specific numerical values, and the technical effects are summarized in the last paragraph. Here, "total loss function" refers to the differentiable target obtained by summing the cross-modal contrastive learning loss and the similarity distribution matching loss with the weight coefficients; "gradient descent optimization algorithm" refers to the method of searching for the optimal solution in the parameter space in the opposite direction of the gradient of the total loss function; "learning rate scheduling strategy" is used to dynamically adjust the learning rate to balance the convergence speed and the final accuracy; "convergence condition" is used to determine whether the model has reached the optimal or approximately optimal state.

[0103] During training, the total loss function L_total is calculated forwardly for the same batch of data, and then the gradients of the image encoder parameters θ_v and the text encoder parameters θ_t with respect to L_total are obtained based on the automatic differentiation mechanism and To prevent gradient explosion, the two sets of gradients are clipped to a threshold of 1.0 in the L2 norm; then the AdamW optimization algorithm is used to perform parameter updates on θ v and θ t respectively, and the update formula contains momentum estimation, second moment estimation and weight decay, and the weight decay coefficient is set to 0.01 to reduce the risk of overfitting.

[0104] The initial learning rate lr init is set to 1x10^-4. The first 3 rounds use a linear warm-up strategy: the first round lr is lr init x 0.3, the second round lr is lr init x 0.6, and the third round lr is lr init. After the warm-up, enter the cosine annealing phase, and decrease to the minimum learning rate lr min = 1x10^-6 according to the cosine function within the remaining training rounds. This strategy provides a larger step size in the early stage of training to accelerate convergence, and gradually reduces the step size to refine the search in the later stage.

[0105] The upper limit of the total training rounds is set to 40. After each round of training, L total val and retrieval accuracy indicator R@1 are calculated on the independent validation set immediately. If the R@1 improvement amplitude of the last 5 rounds is less than 0.1% and L total val does not decrease significantly, it is considered that the model has converged, triggering early stopping; or when L total val is less than 0.02, i.e. the absolute loss is small enough, it is also considered to meet the convergence condition. If neither of the two conditions is met, the next round of training will continue until the upper limit of 40 rounds is reached.

[0106] By synchronously optimizing image and text encoders under the unified learning rate scheduling and early stopping strategy, the embodiment can stably converge within a limited number of rounds, avoiding training collapse caused by gradient explosion or sudden learning rate drop; at the same time, the convergence state is dynamically monitored based on the validation set to ensure that the model stops updating near the optimal point, significantly improving the cross-modal embedding alignment accuracy and the pedestrian retrieval accuracy in noisy scenarios, and reducing the risk of overfitting, meeting the dual requirements of robustness and efficiency for actual deployment.

[0107] In some embodiments, the query text is input into the text encoder to obtain query features, the pedestrian image to be retrieved is input into the image encoder to obtain pedestrian features, and the target pedestrian image that best matches the query text is output based on the cosine similarity between the query features and the pedestrian features, including:

[0108] Before the retrieval phase starts, the image encoder is used to process the set of pedestrian images to be retrieved offline, extract corresponding pedestrian features, and establish an approximate nearest neighbor index structure based on the pedestrian features;

[0109] Receiving a query text, inputting the query text into the text encoder to obtain query features;

[0110] Retrieving the query features in the approximate nearest neighbor index structure to obtain a set of candidate pedestrian features;

[0111] Calculate the cosine similarity between the query feature and each pedestrian feature in the candidate pedestrian feature set, and sort them from high to low according to the cosine similarity;

[0112] According to the preset output quantity, select the pedestrian image with the highest similarity from the sorting result as the target pedestrian image most matched with the query text.

[0113] Specifically, the approximate nearest neighbor index is a retrieval structure that uses spatial partitioning or graph structure to approximate the full vector pairwise comparison, which can reduce the computational complexity at the expense of very small accuracy; the candidate set reordering refers to performing more accurate similarity calculation on a number of high similarity vectors returned by the index to improve the hit accuracy; the recall number K represents the number of candidate images returned by the index stage, and the display number N represents the number of target images finally presented to the user.

[0114] For the set of pedestrian images to be retrieved, the trained image encoder is used to infer the 512-dimensional pedestrian feature vector in turn. To ensure that the feature dimensions of different images are consistent, first calculate the two-norm of each pedestrian feature vector, and then normalize each element with its own norm to make the vector length 1. The normalized features are stored in the feature library in sequential numbering and correspond one-to-one with the original image file path.

[0115] Further, when the size of the feature library reaches the order of 1 million, in order to balance the retrieval speed and memory occupation, the embodiment uses a hierarchical small-world graph structure to establish an approximate nearest neighbor index. The specific steps include:

[0116] Each pedestrian feature vector is inserted step by step using a heuristic greedy algorithm, and edges pointing to several far neighbors are established in the high-level graph, and edges pointing to several near neighbors are established in the bottom-level graph;

[0117] The maximum number of edges in the high-level graph is set to 16, and the maximum number of edges in the bottom-level graph is set to 32 to ensure graph sparsity;

[0118] After the construction is completed, the index file is persistently saved for subsequent fast loading. This index supports approximate K-nearest neighbor retrieval with cosine similarity as the distance measure.

[0119] Further, when the retrieval request arrives, first receive the query text and send it into the text encoder to generate a 512-dimensional query feature vector, and also perform a two-norm normalization operation to obtain a query feature with a length of 1. Normalization can make the vector dot product directly equivalent to the cosine similarity, which is convenient for unified retrieval measurement.

[0120] In some examples, the query feature is retrieved in the loaded approximate nearest neighbor index, the recall number K is set to 200, a multi-layer search is performed from top to bottom in the index graph, and 200 pedestrian features with the highest cosine similarity to the query feature and corresponding pedestrian image identifiers are returned. In order to improve the ranking accuracy, the CPU vector instruction is used to recalculate the exact cosine similarity between the 200 pedestrian features and the query feature, and the result is written into the candidate list.

[0121] The candidate list is sorted according to the cosine similarity from high to low, and the top N pedestrian image paths are taken as the final retrieval result, where N can be dynamically configured by the front-end application according to the display requirements, and the commonly used value is 10. If the similarity is tied, the stable output is maintained according to the data import order. The sorting result is returned to the upper business system through the HTTP interface, and the business system displays the image thumbnail according to the visualization template.

[0122] By offline batch extraction and normalization of pedestrian features, the embodiment moves the high computational load forward, significantly reducing the online load; the approximate nearest neighbor index is constructed by using the hierarchical small world graph, which can return a high recall rate candidate set in the order of 100 milliseconds in a million image library; through accurate reordering of the candidate set, the speed and accuracy of the retrieval result are considered; finally, the system realizes sub-second response time and high accuracy target pedestrian positioning in a real complex scene, meeting the real-time retrieval needs of public safety.

[0123] The following is an apparatus embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.

[0124] Figure 2 is a structural schematic diagram of a noise-robust text-to-image pedestrian retrieval device based on a soft label provided by an embodiment of the present application. As shown in Figure 2 , the noise-robust text-to-image pedestrian retrieval device based on a soft label includes:

[0125] The input module 201 is configured to input pedestrian images and corresponding text descriptions in a training set into an image encoder and a text encoder respectively to obtain image global features and text global features.

[0126] The normalization module 202 is configured to calculate a cosine similarity based on the image global features and the text global features, and normalize the cosine similarity to generate a soft label representing a confidence of image-text pairing.

[0127] The generation module 203 is configured to assign a sample weight to each training sample according to the soft label, and combine a dynamic weight factor that increases with the training process to obtain a joint weight for the current iteration.

[0128] The weighting module 204 is configured to construct a cross-modal contrastive learning loss and a similarity distribution matching loss by using a joint weight respectively, and to obtain a total loss function by weighting and summing the cross-modal contrastive learning loss and the similarity distribution matching loss.

[0129] The updating module 205 is configured to update parameters of the image encoder and the text encoder by using the total loss function until training converges, so as to obtain a cross-modal alignment model.

[0130] The output module 206 is configured to, in a retrieval stage, input a query text into the text encoder to obtain query features, input a to-be-retrieved pedestrian image into the image encoder to obtain pedestrian features, sort according to cosine similarity between the query features and the pedestrian features, and output a target pedestrian image that is most matched with the query text.

[0131] In some embodiments, Figure 2 The input module 201 of the cross-modal alignment model is configured to perform size standardization on each pedestrian image, divide the pedestrian image into a plurality of non-overlapping image blocks according to a preset size, obtain an image block sequence, and add a classification mark at the front of the image block sequence; input the image block sequence containing the classification mark into a multi-layer visual transformation module of the image encoder, and output an image feature sequence, wherein a feature vector corresponding to the classification mark is taken as an image global feature; perform word segmentation on each text description to obtain a text mark sequence, add a start mark at the beginning of the text mark sequence, and add an end mark at the end of the text mark sequence; input the text mark sequence containing the start mark and the end mark into a sequence transformation module of the text encoder, and output a text feature sequence, wherein a feature vector corresponding to the end mark is taken as a text global feature.

[0132] In some embodiments, Figure 2 The normalization module 202 of the cross-modal alignment model is configured to perform two-norm normalization on the image global feature and the text global feature respectively, to obtain a first normalized vector and a second normalized vector; calculate a cosine similarity based on the first normalized vector and the second normalized vector; input the cosine similarity into a S-shaped mapping function and limit the output range, to obtain a soft label representing a confidence of image-text pairing.

[0133] In some embodiments, Figure 2 The generation module 203 of the cross-modal alignment model is configured to input the soft label into a preset weight mapping function, to calculate a static sample weight according to a value range of the soft label; call an incremental function to generate a dynamic weight factor according to a current training round, the dynamic weight factor monotonically increases with the training round and remains constant after reaching an upper limit value; and weight and combine the static sample weight and the dynamic weight factor, to obtain a joint weight for a current iteration.

[0134] In some embodiments, Figure 2The weighting module 204 combines the image global features and the text global features in the current training batch two by two, calculates the cosine similarity, and obtains a similarity matrix by combining a temperature coefficient; the similarity matrix is used to distinguish matched pairs and non-matched pairs, and the joint weight is multiplied by the corresponding similarity score to construct a cross-modal contrast learning loss; the image modality similarity distribution and the text modality similarity distribution are extracted from the similarity matrix, respectively, and the difference between the image modality similarity distribution and the text modality similarity distribution is calculated by using a distribution difference measurement function and a joint weight to obtain a similarity distribution matching loss; the cross-modal contrast learning loss and the similarity distribution matching loss are weighted and summed according to a preset loss coefficient to obtain a total loss function.

[0135] In some embodiments, Figure 2 The updating module 205 calculates the gradient of the image encoder parameters and the gradient of the text encoder parameters by using the total loss function, respectively; the image encoder parameters are iteratively updated by using the gradient of the image encoder parameters according to a preset gradient descent optimization algorithm and a learning rate scheduling strategy, and the text encoder parameters are iteratively updated by using the gradient of the text encoder parameters; at the end of each training round, the total loss function and a preset evaluation index are calculated based on a validation dataset to determine whether a convergence condition is met; the iteration is stopped when the convergence condition is met, and a cross-modal alignment model containing the updated image encoder parameters and the text encoder parameters is output.

[0136] In some embodiments, Figure 2 The output module 206 performs offline processing on a set of to-be-retrieved pedestrian images by using the image encoder before the retrieval stage starts, extracts corresponding pedestrian features, and establishes an approximate nearest neighbor index structure based on the pedestrian features; receives a query text, inputs the query text into the text encoder to obtain a query feature; retrieves the query feature in the approximate nearest neighbor index structure to obtain a set of candidate pedestrian features; calculates the cosine similarity between the query feature and each pedestrian feature in the set of candidate pedestrian features, and sorts the pedestrian features from high to low according to the cosine similarity; and selects the pedestrian image with the highest similarity from the sorted results as the target pedestrian image that best matches the query text according to a preset output quantity.

[0137] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0138] Figure 3 is a structural schematic diagram of an electronic device 3 provided by the embodiments of the present application. As shown in Figure 3As shown, the electronic device 3 of this embodiment includes a processor 301, a memory 302, and a computer program 303 stored in the memory 302 and executable on the processor 301. The processor 301 implements the steps in each of the above method embodiments when executing the computer program 303. Alternatively, the processor 301 implements the functions of each module / unit in each of the above apparatus embodiments when executing the computer program 303.

[0139] By way of example, the computer program 303 can be segmented into one or more modules / units, which are stored in the memory 302 and executed by the processor 301 to complete the present application. The one or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program 303 in the electronic device 3.

[0140] The electronic device 3 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The electronic device 3 can include but is not limited to the processor 301 and the memory 302. Those skilled in the art can understand that the electronic device 3 can include more or fewer components, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, and the like. Figure 3 The electronic device 3 is merely an example and does not constitute a limitation on the electronic device 3, and can include more or fewer components than shown, or combine certain components, or different components, for example, the electronic device can also include an input / output device, a network access device, a bus, and the like.

[0141] The processor 301 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0142] The memory 302 can be an internal storage unit of the electronic device 3, for example, a hard disk or a memory of the electronic device 3. The memory 302 can also be an external storage device of the electronic device 3, for example, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 3. Further, the memory 302 can include both the internal storage unit and the external storage device of the electronic device 3. The memory 302 is used to store computer programs and other programs and data required by the electronic device. The memory 302 can also be used to temporarily store data that has been output or will be output.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0144] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in detail in a certain embodiment can refer to the relevant description of other embodiments.

[0145] Those of ordinary skill in the art can appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0146] In the embodiments of the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other manners. For example, the described apparatus / computer device embodiments are merely schematic. For example, the division of the modules or units can be different, and each can include multiple sub-modules or units. Some or all of the modules or units can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0147] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0148] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0149] If the integrated module / unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the flow of the above-mentioned embodiment methods can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program can include computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0150] The above examples are only used to illustrate the technical solutions of the present application, but not limit the same; although the technical solutions of the present application are described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can still be modified, or some technical features thereof can be replaced by equivalent ones; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A soft label based noise robust text to image pedestrian retrieval method, characterized in that, The method comprises the following steps: inputting pedestrian images and corresponding text descriptions in a training set into an image encoder and a text encoder respectively to obtain image global features and text global features; calculating a cosine similarity based on the image global features and the text global features, normalizing the cosine similarity, and generating a soft label representing the confidence of image-text pairing; assigning a sample weight to each training sample according to the soft label, and combining a dynamic weight factor that increases with the training process to obtain a joint weight for the current iteration; using the joint weight to construct a cross-modal contrast learning loss and a similarity distribution matching loss respectively, and weighting and summing the cross-modal contrast learning loss and the similarity distribution matching loss to obtain a total loss function; updating the parameters of the image encoder and the text encoder using the total loss function until the training converges, and obtaining a cross-modal alignment model; in the retrieval stage, inputting a query text into the text encoder to obtain a query feature, inputting a pedestrian image to be retrieved into the image encoder to obtain a pedestrian feature, sorting according to the cosine similarity between the query feature and the pedestrian feature, and outputting a target pedestrian image that best matches the query text; wherein the step of using the joint weight to construct a cross-modal contrast learning loss and a similarity distribution matching loss respectively, and weighting and summing the cross-modal contrast learning loss and the similarity distribution matching loss to obtain a total loss function comprises: combining the image global features and the text global features in the current training batch two by two to calculate a cosine similarity and obtain a similarity matrix combined with a temperature coefficient; distinguishing matched pairs and unmatched pairs according to the similarity matrix, and multiplying the joint weight by the corresponding similarity scores to construct a cross-modal contrast learning loss; extracting an image modality similarity distribution and a text modality similarity distribution from the similarity matrix, calculating the difference between the image modality similarity distribution and the text modality similarity distribution using a distribution difference measure function and the joint weight, and obtaining a similarity distribution matching loss; weighting and summing the cross-modal contrast learning loss and the similarity distribution matching loss according to a preset loss coefficient to obtain a total loss function.

2. The method of claim 1, wherein, The step of inputting pedestrian images and corresponding text descriptions in a training set into an image encoder and a text encoder respectively to obtain image global features and text global features comprises: standardizing the size of each pedestrian image, dividing it into a plurality of non-overlapping image blocks according to a preset size to obtain an image block sequence, and adding a classification mark at the beginning of the image block sequence; inputting the image block sequence containing the classification mark into a multi-layer visual transformation module of the image encoder to output an image feature sequence, wherein the feature vector corresponding to the classification mark is the image global feature; segmenting each text description to obtain a text token sequence, adding a start token at the beginning of the text token sequence, and adding an end token at the end of the text token sequence; inputting the text token sequence containing the start token and the end token into a sequence transformation module of the text encoder to output a text feature sequence, wherein the feature vector corresponding to the end token is the text global feature.

3. The method of claim 1, wherein, The cosine similarity is calculated based on the image global feature and the text global feature, and the cosine similarity is normalized to generate a soft label representing the confidence of the image-text pair, including: The image global feature and the text global feature are respectively normalized by two norms to obtain a first normalized vector and a second normalized vector; The cosine similarity is calculated based on the first normalized vector and the second normalized vector; The cosine similarity is input into an S-shaped mapping function and the output range is limited to obtain a soft label representing the confidence of the image-text pair.

4. The method of claim 1, wherein, The soft label is input into a preset weight mapping function to calculate a static sample weight according to the value range of the soft label; An incremental function is called to generate a dynamic weight factor according to the current training round, and the dynamic weight factor monotonically increases with the training round and remains constant after reaching an upper limit value; The static sample weight and the dynamic weight factor are combined by weighting to obtain a joint weight for the current iteration. The total loss function is used to update the parameters of the image encoder and the text encoder until the training converges to obtain a cross-modal alignment model, including:

5. The method of claim 1, wherein, The total loss function is used to calculate the gradient of the image encoder parameters and the gradient of the text encoder parameters respectively; According to a preset gradient descent optimization algorithm and a learning rate scheduling strategy, the image encoder parameters are iteratively updated using the gradient of the image encoder parameters, and the text encoder parameters are iteratively updated using the gradient of the text encoder parameters; At the end of each training round, the total loss function and a preset evaluation indicator are calculated based on a validation dataset to determine whether the convergence condition is met; When the convergence condition is met, the iteration is stopped, and a cross-modal alignment model containing updated image encoder parameters and text encoder parameters is output. The query text is input into the text encoder to obtain query features, and the pedestrian image to be searched is input into the image encoder to obtain pedestrian features, and the cosine similarity between the query features and the pedestrian features is used for sorting to output the target pedestrian image most matched with the query text, including:

6. The method of claim 1, wherein, Before the retrieval stage starts, the image encoder is used to perform offline processing on the set of pedestrian images to be searched, and corresponding pedestrian features are extracted and an approximate nearest neighbor index structure is established based on the pedestrian features; The query text is received, and the query text is input into the text encoder to obtain the query features; The query features are used to search in the approximate nearest neighbor index structure to obtain a set of candidate pedestrian features; The cosine similarity between the query features and each pedestrian feature in the set of candidate pedestrian features is calculated, and the cosine similarity is sorted from high to low; According to a preset output quantity, the pedestrian image with the highest similarity is selected from the sorted results as the target pedestrian image most matched with the query text. including:

7. A soft label based noise robust text to image pedestrian retrieval apparatus, comprising: ​ The input module is configured to input pedestrian images and corresponding text descriptions in a training set into an image encoder and a text encoder respectively to obtain image global features and text global features. The normalization module is configured to calculate a cosine similarity based on the image global features and the text global features, normalize the cosine similarity, and generate a soft label representing a confidence of image-text pairing. The generation module is configured to assign a sample weight to each training sample according to the soft label and combine a dynamic weight factor that increases with a training process to obtain a joint weight for a current iteration. The weighting module is configured to construct a cross-modal contrastive learning loss and a similarity distribution matching loss using the joint weight respectively, and sum the cross-modal contrastive learning loss and the similarity distribution matching loss to obtain a total loss function. The update module is configured to update parameters of the image encoder and the text encoder using the total loss function until training converges to obtain a cross-modal alignment model. The output module is configured to input a query text into the text encoder to obtain a query feature, input a pedestrian image to be retrieved into the image encoder to obtain a pedestrian feature, sort according to a cosine similarity between the query feature and the pedestrian feature, and output a target pedestrian image that best matches the query text. The weighting module is configured to combine image global features and text global features in a current training batch two by two to calculate a cosine similarity and obtain a similarity matrix by combining a temperature coefficient; distinguish matched pairs and non-matched pairs according to the similarity matrix, and multiply the joint weight to the corresponding similarity score to construct a cross-modal contrastive learning loss. The similarity matrix is extracted to obtain an image modality similarity distribution and a text modality similarity distribution, a distribution difference measurement function and the joint weight are used to calculate a difference between the image modality similarity distribution and the text modality similarity distribution to obtain a similarity distribution matching loss; and a total loss function is obtained by weighting and summing the cross-modal contrastive learning loss and the similarity distribution matching loss according to a preset loss coefficient.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the steps of the method of any one of claims 1-6.

Images