A method for constructing a text-based image pedestrian retrieval model and an image pedestrian retrieval method

Abstract

The application belongs to the technical field of image-text cross-modal retrieval, and particularly relates to a construction method of an image pedestrian retrieval model based on text and an image pedestrian retrieval method. The model comprises two key components: an accurate image feature enhancement module which improves the accuracy of local image features by extracting foreground information and removing background noise; and a text interference suppression module which effectively reduces the influence of irrelevant or misleading text interference by identifying and masking inconsistent words with a high probability, thereby achieving the effect of text noise reduction; in addition, a unified visual-text network is adopted, which helps to obtain a joint embedding space in which visual and text features can be harmoniously aligned. The method has the advantages of strong generalization ability and good retrieval performance, can retrieve images of specific individuals according to text descriptions, and solves the challenge in cross-modal alignment through the accurate feature enhancement module and the text interference suppression module, thereby improving the retrieval performance and noise resistance.

Description

Technical Field

[0001] This invention belongs to the technical field of image-text cross-modal retrieval, and more specifically, relates to a method for constructing a text-based image pedestrian retrieval model and an image pedestrian retrieval method. Background Technology

[0002] Text-based person retrieval is a subtask of image-text retrieval and person re-identification, aiming to retrieve images of specific individuals from a large database based on textual descriptions. Its importance is self-evident when only textual information is available, making it indispensable for applications such as video surveillance and security assistance. However, the inherent differences between visual and textual modalities present significant challenges to this task.

[0003] Current methods typically employ global and local feature alignment to bridge the gap between modalities. Global-level methods aim to learn global features shared by both modalities in the embedding space, but they may overlook associations related to fine-grained features. Local-level methods focus on extracting local features specific to a particular modality and achieving accurate alignment. However, these methods face significant challenges, primarily related to noise in the text and limitations of small-scale training data. Noise in text and images can lead to a mismatch between text queries and individual visual representations. Furthermore, small-scale training data hinders the model's ability to learn robust and accurate features.

[0004] Previous global-level methods often neglected the refinement of extracted image features. This oversight can be problematic because images typically contain regions unrelated to the paired text, which can introduce noise and interfere with the alignment of image and text features. While some local-level methods have made progress in feature refinement, they generally divide text and visual features into equal-sized blocks and calculate similarity between blocks sequentially. However, this block-to-block comparison lacks rigorous semantic correspondence or precise word-to-region alignment, potentially leading to unintentional misalignments. Furthermore, in addition to noise in the image, previous research overlooked a crucial fact: some words in the text description may not precisely match the paired image. Mismatches between local text descriptions and their corresponding image regions can cause discrepancies. These discrepancies can be amplified when using local feature alignment techniques. Tan et al. proposed a noise-aware masking method to mask mismatched words in the text, but their primary focus was on text generated by a large language model, and their method ignored the distinction between individual lemmas and the entire text, leading to some discrepancies and increasing the difficulty of achieving accurate alignment. Summary of the Invention

[0005] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method for constructing a text-based image pedestrian retrieval model and an image pedestrian retrieval method. The purpose is to retrieve images of specific individuals based on text descriptions, and to solve the challenges of cross-modal alignment through a precise feature enhancement module and a text interference reduction module, thereby improving retrieval performance and noise resistance.

[0006] To achieve the above objectives, according to one aspect of the present invention, a method for constructing a text-based image pedestrian retrieval model is provided, comprising:

[0007] An image pedestrian retrieval network is constructed, comprising an image feature enhancement module, a text denoising module, and a pre-trained bimodal feature extraction backbone network. The bimodal feature extraction backbone network is used to encode and obtain global and local image feature vectors of the input image. The image feature enhancement module processes the input image to obtain binarized block image vectors, performs a vector dot product between the block image vectors and the local image feature vectors to obtain a local image feature vector with background information removed, and averages and maps all features in this vector to obtain the enhanced pedestrian features of the input image.

[0008] The text denoising module is used to obtain local text features of the input text corresponding to the input image based on the dual-modal feature extraction backbone network, determine the degree to which the similarity between each local text feature and each feature in the local image feature vector deviates from the average value of all local text features and the similarity between each feature in the local image feature vector, and mask the local text corresponding to the local text features whose deviation quantization value is greater than a threshold in the input text to finally obtain a new input text;

[0009] An image-text pair training sample set is constructed. Based on the global image features, pedestrian enhancement features, and global text features encoded by the bimodal feature extraction backbone network based on the new input text for each sample, the parameters of the bimodal feature extraction backbone network and the operation parameters of the mapping are iteratively adjusted through image-text feature similarity loss to realize the construction of the image pedestrian retrieval model.

[0010] Furthermore, the image feature enhancement module is implemented as follows when processing the obtained binarized block image vector: human body analysis processing is performed on the pedestrian in the input image to obtain a pedestrian contour mask; the mask is binarized and divided into blocks to obtain a binarized block image vector.

[0011] Furthermore, the text denoising module determines the quantization value of the deviation degree as follows:

[0012] The similarity between each local text feature and each feature in the local image feature vector after removing background information is calculated to obtain the first similarity vector corresponding to the local text feature. All local text features are averaged, and the average local text feature is then compared with each feature in the local image feature vector after removing background information to obtain the second similarity vector corresponding to the average local text feature. The difference between each first similarity vector and the second similarity vector is calculated, and the resulting vector is the quantified value of the deviation degree corresponding to the local text feature.

[0013] Furthermore, the text denoising module implements local text masking based on the quantization value of the deviation degree as follows:

[0014] The average of all elements in the difference vector is added to the preset balance parameter value to obtain the probability of the local text error corresponding to the first similarity vector. The local text corresponding to the probability that meets the threshold is masked in the input text to obtain a new input text.

[0015] Furthermore, the image-text feature similarity loss includes ID loss and SDM loss.

[0016] Furthermore, the method also includes: in each iteration, storing all global image features, pedestrian enhancement features, and global text features encoded by the bimodal feature extraction backbone network based on the new input text corresponding to the current batch into the image feature memory and the text feature memory, and combining the momentum update model based on exponential moving average to stabilize the feature extraction process of the bimodal feature extraction backbone network.

[0017] Image-text feature similarity loss also includes MITC loss, whose loss function is expressed as:

[0018]

[0019] In the formula, The MITC loss represents the conversion from global image features to global text features encoded based on the new input text. This represents the MITC loss, which is the loss from global text features encoded based on new input text to global image features of the image. The MITC loss represents the conversion of pedestrian enhancement features in an image to global text features encoded from the new input text. The MITC loss is based on the global text features encoded from the new input text to the pedestrian enhancement features of the image.

[0020] in, In the formula, τ is a temperature hyperparameter. BN represents the current batch size during training, and N represents the total number of image-text pairs. This represents the true matching label of the j-th image-text pair in the i-th batch. This indicates that the current image-text pair is a match from the same pedestrian identity. This indicates that the current image-text pair does not match; For the global image features of all images in the i-th batch, Let $T$ be the global text feature corresponding to the $j$-th image-text pair in the text memory $T$, encoded based on the new input text. The global text feature corresponding to the nth image-text pair in the text memory is obtained by encoding the new input text. express and The similarity between them; express and The similarity between them;

[0021] exist formula lieutenant general Replace with corresponding image features, and Replace it with the corresponding image features, that is The expression; in in the formula This represents the pedestrian enhancement features of all images in the i-th batch, i.e. The expression; in formula lieutenant general Replace with corresponding image features. and Replace with the corresponding image features, that is The expression.

[0022] Furthermore, the pre-trained bimodal feature extraction backbone network is composed of the image encoder and text encoder from the CLIP large model.

[0023] According to another aspect of the present invention, a text-based image pedestrian retrieval method is provided, comprising:

[0024] Obtain a set of pedestrian images and target text for the image to be searched, and input them into the bimodal feature extraction backbone network in the image pedestrian retrieval model, respectively, to determine the pedestrian images corresponding to the target text from the image set, thereby realizing text-based image pedestrian retrieval;

[0025] The image pedestrian retrieval model is constructed using the method described above.

[0026] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0027] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0028] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages:

[0029] 1. This invention considers that cross-modal alignment is mainly affected by image background interference and mismatched words in the text, leading to inaccuracies in fine-grained image features and misalignment of text and image features. To address this challenging task, a novel framework for accurate image feature enhancement and text interference suppression is proposed. This framework comprises two key components: an accurate image feature enhancement module, which improves the accuracy of local image features by extracting foreground information and removing background noise; and a text interference suppression module (i.e., a text denoising module), which effectively mitigates the influence of irrelevant or misleading text interference by identifying and masking inconsistent words with high probability, achieving text denoising. Experimental results show that this method has advantages such as strong generalization ability and excellent retrieval performance. Therefore, this invention can retrieve images of specific individuals based on text descriptions and, through the accurate feature enhancement module and the text interference reduction module, solves the challenges of cross-modal alignment, improving retrieval performance and noise resistance.

[0030] 2. This invention further proposes a dual-modal feature extraction backbone network that uses the image and text encoders from the pre-trained CLIP large model as the basic visual and text feature extractors in the entire image person retrieval model. This method effectively transfers the knowledge from CLIP to text-based person retrieval, extracting more accurate image and text features. Attached Figure Description

[0031] Figure 1 This is a schematic diagram illustrating a method for constructing a text-based image pedestrian retrieval model according to an embodiment of the present invention;

[0032] Figure 2 This is a schematic diagram illustrating the problems existing in text-based image pedestrian retrieval provided by the embodiments of the present invention;

[0033] Figure 3 This is a schematic diagram illustrating another method for constructing a text-based image pedestrian retrieval model provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0035] Example 1

[0036] A method for constructing a text-based image pedestrian retrieval model, such as... Figure 1 As shown, it includes:

[0037] An image pedestrian retrieval network is constructed, which includes an image feature enhancement module, a text denoising module, and a pre-trained bimodal feature extraction backbone network;

[0038] A dual-modal feature extraction backbone network is used to encode and obtain global image features and local image feature vectors of the input image. The image feature enhancement module is used to process the input image to obtain binarized block image vectors. The block image vectors are multiplied by the local image feature vectors to obtain local image feature vectors (also known as main body embedding features) after removing background information. All features in this vector are averaged and mapped to obtain the pedestrian enhancement features of the input image. The text denoising module is used to obtain local text features of the input text corresponding to the input image based on the dual-modal feature extraction backbone network. It judges the degree to which the similarity between each local text feature and each feature in the local image feature vector deviates from the average similarity between all local text features and each feature in the local image feature vector. The local text corresponding to the local text features with a deviation value greater than a threshold is masked in the input text to finally obtain the new input text.

[0039] Construct an image-text pair training sample set;

[0040] Based on the global image features, pedestrian enhancement features, and global text features encoded by the bimodal feature extraction backbone network based on the new input text for each sample, the parameters of the bimodal feature extraction backbone network and the operation parameters of the mapping are iteratively adjusted through image-text feature similarity loss to realize the construction of an image pedestrian retrieval model.

[0041] The image pedestrian retrieval model constructed by the method in this embodiment includes a bimodal feature extraction backbone network (containing an image encoder and a text encoder) and two implicit alignment strategies (i.e., a precise image feature enhancement module and a text interference suppression module). The bimodal feature extraction backbone network is a unified visual-text network, which helps to obtain a joint embedding space in which visual and text features can be harmoniously aligned. The fine-grained image feature enhancement module aims to improve the accuracy of local image features by extracting foreground information and removing background information. By meticulously integrating global features from the image encoder, it further obtains enhanced features from the image, which help to establish a closer association with the text description. The text denoising module aims to mitigate the influence of irrelevant or misleading text information by identifying and masking inconsistent words with high probability based on local image feature vectors.

[0042] like Figure 2 As shown, text-to-pedestrian retrieval methods often contain regions in the image that are unrelated to the paired text. These regions may introduce noise and interfere with the alignment of image and text features. Figure 2 The left-hand image (a) vividly illustrates this problem, showing that the area marked in red fails to align with the text description, potentially obscuring the feature alignment process. The fine-grained image feature enhancement module utilizes a self-correcting human body parsing algorithm to selectively extract foreground information and filter out background noise, such as... Figure 2 As shown in the right-hand figure (a) of the diagram, this reduces semantic errors and improves the accuracy of feature localization. Furthermore, besides noise in the image, some words in the text description may not precisely match the paired image. For example, in... Figure 2 In (b), there is a discrepancy between the description of the words highlighted in red in the text on the right and the actual clothing features worn by the pedestrian shown in the image on the left.

[0043] Text-based image retrieval of people aims to retrieve images of specific individuals based on text descriptions. However, this task faces the challenge of cross-modal alignment. This embodiment considers that this mainly stems from background interference and the influence of mismatched words in the text, leading to inaccuracies in fine-grained image features and misalignment between text and image features. To address this challenging task, a novel framework for accurate image feature enhancement and text interference suppression is proposed. This framework comprises two key components: an accurate image feature enhancement module, which improves the accuracy of local image features by extracting foreground information and removing background noise; and a text interference suppression module (i.e., a text denoising module), which effectively mitigates the influence of irrelevant or misleading text interference by identifying and masking inconsistent words with high probability. This enhances the retrieval model's ability to focus on more accurate text features during the learning process, achieving the effect of text denoising. Experimental results show that this method has advantages such as strong generalization ability and excellent retrieval performance.

[0044] Overall, this embodiment provides a framework for fine-grained feature refinement and text noise suppression, thereby improving the retrieval performance of text-based person retrieval models.

[0045] As a preferred approach, the bimodal feature extraction backbone network employs the image encoder and text encoder from the pre-trained CLIP large model, serving as the foundational visual and text feature extractors in the entire image-based person retrieval model. This method effectively transfers CLIP's knowledge to text-based person retrieval, extracting more accurate image and text features.

[0046] Specifically, for image feature extraction. Given an image I∈R W×H×C First, the image is divided into a series of non-overlapping image patches, totaling M patches. Then, each image patch is mapped to a one-dimensional token, and a learnable [CLS] token is added to the beginning of the token sequence. Finally, these linear tokens are fed into an image encoder to extract a set of image features, denoted as F. v ={f cls v f1 v ,…,f M v Here, f cls v F represents global-level image features. vl ={f1 v ,…,f M v} represents local-level image features.

[0047] For text feature extraction. Given a text T o The CLIP text encoder is used to extract text tokens. The text tokens are then tokenized using the Lowercase Byte Pair Encoding (BPE) method, with [SOS] (start of sequence) and [EOS] (end of sequence) markers added to the beginning and end of the text token sequence, respectively. This tokenized sequence is then processed by a text transformer block to generate a set of text features, denoted as F. t ={f sos t f1 t ,…,f eos t}, where f eos t F represents global-level text features. tl ={f1 t ,…,f N t} represents local-level text features.

[0048] As a preferred implementation, the image feature enhancement module processes and obtains the binarized block image vector by performing human body analysis on the pedestrian in the input image to obtain a pedestrian contour mask; and performing binarization and block division on the mask to obtain a binarized block image vector.

[0049] like Figure 3 As shown, in order to obtain more accurate image features and filter out background noise, human body analysis technology is used to generate a mask I for the main body parts of each image. m The main body parts include the head, limbs, and torso. For a single image, its local image features F are obtained. vl and image mask of the main body parts I m .

[0050] The main body image mask is binarized, setting the pixel values ​​of the body region to 255 and the pixel values ​​of the background region to 0. Then, a summation operation is performed on all three color channels, setting any non-zero values ​​to 1, thus obtaining the intermediate feature representation I. v This process effectively captures human features while discarding irrelevant background information. Subsequently, we use the nearest neighbor interpolation algorithm to perform I... v Make size adjustments, then with F vl Perform element-wise multiplication to obtain the main body embedding vector F′. vl After averaging, F′ vl The input is fed into the projection layer to generate the main body features fmain of the image. This series of operations can robustly extract the main human body features, facilitating the alignment of image and text features.

[0051] The fine-grained image feature enhancement module utilizes a self-correcting human body parsing algorithm to selectively extract foreground information and filter out background noise, thereby reducing semantic errors and improving the accuracy of feature localization. By meticulously integrating global features from the image encoder, enhanced features are further extracted from the image, which help establish a closer association with the text description. The fine-grained image feature enhancement module skillfully analyzes key components in the image, thus refining the feature extraction stage and improving retrieval accuracy.

[0052] As a preferred implementation, the text denoising module determines the quantization value of the deviation as follows:

[0053] The similarity between each local text feature and each feature in the local image feature vector after removing background information is calculated to obtain the first similarity vector corresponding to the local text feature. All local text features are averaged, and the average local text feature is then compared with each feature in the local image feature vector after removing background information to obtain the second similarity vector corresponding to the average local text feature. The difference between each first similarity vector and the second similarity vector is calculated, and the resulting vector is the quantified value of the deviation degree corresponding to the local text feature.

[0054] As a further preferred implementation, the text denoising module performs local text masking based on the quantization value of the deviation degree in the following manner:

[0055] The average of all elements in the difference vector is added to the preset balance parameter value to obtain the probability of the local text error corresponding to the first similarity vector. The local text corresponding to the probability that meets the threshold is masked in the input text to obtain a new input text.

[0056] This embodiment introduces a balancing parameter value to ensure that the average value is within a set range, forming an operational standard that facilitates comparison with a set threshold.

[0057] Specifically, such as Figure 3 As shown, during training, one input image corresponds to one input text for training. For an image-text pair, the text encoder obtains local-level text features F. tl The difference between the average of all image words corresponding to each text word (local text) and the average of all image words corresponding to all words is calculated to determine whether a text word is noise. The local image feature vector F′ is also obtained from the fine-grained image feature enhancement module. vl Calculate the average text lexical features and F′. vl The similarity between the features in the data can be represented as: in This represents the average degree of consistency between each text term and the image feature.

[0058] Then, for F tl Each text lexical feature calculate With F′ vl Similarity S between i : in Let S represent the cosine similarity between the i-th text word embedding and the j-th image word embedding. Next, calculate S. i and Deviation between: Then calculate S′i The mean of the noise level P is defined. i for: Where p is a balancing parameter. The optimized P i Representing text terms The possibility of being masked. Use the text noise suppression module to mask the initial text T. o Converted into new input text, called refined text T r In this process, words with low noise or relevance are selectively masked. This masked text T r The text is then fed into the text encoder to obtain more accurate global text features, which are then used in model training. The text noise suppression module improves the robustness and effectiveness of the model by focusing on effective text terms and mitigating the influence of irrelevant terms.

[0059] As a preferred implementation, image-text feature similarity loss includes ID loss and SDM loss.

[0060] ID loss L id The definition is as follows:

[0061]

[0062] Where N represents the number of images in each batch, y i Representing global features f i The corresponding pedestrian ID, where C represents the number of pedestrian categories included. All of these represent parameters used in training.

[0063] The SDM loss is described below:

[0064] Given a minibatch of data containing N image-text pairs, define the global feature representation of each image as follows: Construct a set of image-text representation pairs Where y i，j It is a true matching label. If y i，j =1, then it means They are matches from the same identity; if y i，j =0, which means this pair does not match. Let Let u represent the dot product (cosine similarity) between u and v after L2 normalization. The probability of a match can then be easily calculated using the following softmax function:

[0065]

[0066] In the formula, τ is a temperature hyperparameter used to control the peak value of the probability distribution. The matching probability p i，j This can be considered as being in small batches. and The cosine similarity score between them accounts for and The proportion of total cosine similarity scores. Then, the SDM loss from images to text in the mini-batch is calculated as follows:

[0067]

[0068] In the formula, ∈ is a small value to avoid the problem of the denominator being 0. This represents the actual matching probability.

[0069] Similarly, the SDM loss L is derived from global text features encoded based on new input text to global image features of the image. t2i It can be achieved by swapping f v and f t The SDM loss is formulated in equations (1) and (2) to convert pedestrian enhancement features of the image into global text features encoded from the new input text. Pedestrian enhancement features in the image are derived from global text features encoded from the new input text. Similarly.

[0070] The total SDM loss is calculated as follows:

[0071] As a preferred implementation, the method further includes: in each iteration, storing all global image features, pedestrian enhancement features, and global text features encoded by the bimodal feature extraction backbone network based on the new input text corresponding to the current batch into the image feature memory and the text feature memory, and combining the momentum update model based on exponential moving average to stabilize the feature extraction process of the bimodal feature extraction backbone network.

[0072] Image-text feature similarity loss also includes MITC loss, whose loss function is expressed as:

[0073]

[0074] In the formula, The MITC loss represents the conversion from global image features to global text features encoded based on the new input text. This represents the MITC loss, which is the loss from global text features encoded based on new input text to global image features of the image. The MITC loss represents the conversion of pedestrian enhancement features in an image to global text features encoded from the new input text. The MITC loss is based on the global text features encoded from the new input text to the pedestrian enhancement features of the image.

[0075] in, In the formula, τ is a temperature hyperparameter. B N represents the current batch size during training, and N represents the total number of image-text pairs. This represents the true matching label of the j-th image-text pair in the i-th batch. This indicates that the current image-text pair is a match from the same pedestrian identity. This indicates that the current image-text pair does not match; For the global image features of all images in the i-th batch, Let $T$ be the global text feature corresponding to the $j$-th image-text pair in the text memory $T$, encoded based on the new input text. The global text feature corresponding to the nth image-text pair in the text memory is obtained by encoding the new input text. express and The similarity between them; express and The similarity between them;

[0076] exist formula lieutenant general Replace with corresponding image features, and Replace with the corresponding image features, that is The expression; in in the formula This represents the pedestrian enhancement features of all images in the i-th batch, i.e. The expression; in formula lieutenant general Replace with corresponding image features. and Replace with the corresponding image features, that is The expression.

[0077] Considering that the limitations of mini-batch size may not be able to reflect full detail, image memory and text memory are proposed. By storing all image and text features in the current dataset, these memories can provide a more detailed and comprehensive understanding of the main image details, thereby enhancing the model's ability to learn and utilize a large amount of relational information.

[0078] Specifically, using currently volatile models to build the memory bank can introduce instability. To address this, a momentum update framework is introduced. Specifically, an exponential moving average (EMA) is employed to stabilize and optimize the main part of the image and text encoders, ensuring consistent and reliable memory updates throughout training. EMA can be represented as:

[0079]

[0080] Only online models are available. and Updates are performed via backpropagation, where m∈[0,1] are momentum coefficients. The momentum-updated encoder generates features stored in memory, resulting in a more stable representation.

[0081] Subsequently, a comprehensive metric for bimodal similarity was introduced:

[0082]

[0083] Where B is the current batch size. It is a pedestrian enhancement feature in the image. This is the j-th feature in the text memory. Correspondingly, the text-to-image loss can also be obtained. Similarly, a momentum update model is applied to both the global image features and the text features. Therefore, the contrastive loss can be expressed as:

[0084]

[0085] `main2t` represents the loss from image pedestrian enhancement features to text features, `t2main` represents the loss from text features to image pedestrian enhancement features, `i2t` represents the loss from global image features to text features, and `t2i` represents the loss from text features to global image features. The '2' stands for 'to' in English, and both are pronounced the same.

[0086] Because the number of samples in a batch is limited, the relationship between positive and negative samples becomes simple. Therefore, there is an urgent need to expand the sampling range to find more challenging negative samples. This preferred implementation constructs an image memory and a text memory to expand the range of each batch. This expansion allows training to use more diverse and challenging positive and negative samples, enhancing the robustness of the model representation.

[0087] To address the limitations of mini-batch size, an image and text memory is proposed to store all image and text features from the current dataset. This facilitates a more detailed understanding of key body image details and enhances the model's learning performance. Specifically, to mitigate the impact of the unstable current model, a momentum update framework using exponential moving average (EMA) is introduced to stabilize and optimize the image and text encoders, ensuring consistent and reliable memory updates during training.

[0088] Example 2

[0089] A text-based image pedestrian retrieval method includes:

[0090] Obtain a set of pedestrian images and target text for the image to be searched, and input them into the bimodal feature extraction backbone network in the image pedestrian retrieval model, respectively, to determine the pedestrian images corresponding to the target text from the image set, thereby realizing text-based image pedestrian retrieval;

[0091] The image pedestrian retrieval model is constructed using the method described above.

[0092] To demonstrate that the text-person retrieval method combining feature enhancement and text denoising has advantages in both performance and adaptability, this invention verifies and analyzes the following experiments:

[0093] A series of experiments were conducted on three text-to-pedestrian retrieval datasets (CUHK-PEDES, ICFG-PEDES, and RSTPReid). Mean Average Precision (MAP) and the probability (rank-n) of the top-ranked (highest confidence) images in the search results having correct results were used as accuracy metrics for pedestrian re-identification. For image processing, the CLIP-VIT-B / 16 model was used; for text input, the CLIP text encoder was used. During training, various image enhancement techniques, including random cropping, flipping, and erasing, were applied to enhance data diversity. Similarly, text enhancement strategies involved random deletion, masking, and word replacement to enhance text input. Images were resized to a resolution of 128*384 pixels, and the length of the text input was limited to 77 characters. The Adam optimizer was selected with an initial learning rate of 1*10^-5, which decayed to 10 at epochs 20 and 40, for a total of 60 epochs of training. Furthermore, the temperature parameter τ was set to 0.02.

[0094] This invention was compared with other methods in the field of text-person retrieval, and achieved state-of-the-art performance with significant improvements on three datasets. The experimental results are shown in the table below:

[0095] Table 1 Performance Comparison

[0096]

[0097] As can be seen from the two tables above, under the same experimental conditions, our method demonstrates superior performance compared to current advanced algorithms.

[0098] The relevant technical solutions are the same as in Embodiment 1, and will not be repeated here.

[0099] Example 3

[0100] This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0101] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0102] The relevant technical solutions are the same as above, and will not be repeated here.

[0103] Example 4

[0104] This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0105] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0106] The relevant technical solutions are the same as above, and will not be repeated here.

[0107] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for constructing a text-based image pedestrian retrieval model, characterized in that, include: An image pedestrian retrieval network is constructed, comprising an image feature enhancement module, a text denoising module, and a pre-trained bimodal feature extraction backbone network. The bimodal feature extraction backbone network encodes global image features and local image feature vectors of the input image. The image feature enhancement module processes the input image to obtain binarized block image vectors, performs a vector dot product between the block image vectors and the local image feature vectors to obtain a local image feature vector with background information removed, averages all features in this vector, and maps them to obtain the enhanced pedestrian features of the input image. The text denoising module obtains local text features of the input text corresponding to the input image based on the bimodal feature extraction backbone network, determines the degree to which the similarity between each local text feature and each feature in the local image feature vector deviates from the average similarity between all local text features and each feature in the local image feature vector, and masks the local text corresponding to local text features with deviation values ​​greater than a threshold in the input text to obtain new input text. An image-text pair training sample set is constructed. Based on the global image features, pedestrian enhancement features, and global text features encoded by the bimodal feature extraction backbone network based on the new input text for each sample, the parameters of the bimodal feature extraction backbone network and the operation parameters of the mapping are iteratively adjusted through image-text feature similarity loss to realize the construction of the image pedestrian retrieval model.

2. The construction method as described in claim 1, characterized in that, The image feature enhancement module is implemented as follows when processing the obtained binary block image vector: human body analysis is performed on the pedestrian in the input image to obtain a pedestrian contour mask; the mask is divided into blocks and binarized to obtain a binary block image vector.

3. The construction method as described in claim 1, characterized in that, The text noise reduction module determines the quantization value of the deviation as follows: The similarity between each local text feature and each feature in the local image feature vector after removing background information is calculated to obtain the first similarity vector corresponding to the local text feature. All local text features are averaged, and the average local text feature is then compared with each feature in the local image feature vector after removing background information to obtain the second similarity vector corresponding to the average local text feature. The difference between each first similarity vector and the second similarity vector is calculated, and the resulting vector is the quantified value of the deviation degree corresponding to the local text feature.

4. The construction method as described in claim 3, characterized in that, The text denoising module implements local text masking based on the quantization value of the deviation degree as follows: The average of all elements in the difference vector is added to a preset balance parameter value to obtain the probability of a local text error corresponding to the first similarity vector. The local text corresponding to the probability that meets the threshold is masked in the input text to obtain a new input text.

5. The construction method as described in claim 1, characterized in that, The image-text feature similarity loss includes ID loss and SDM loss.

6. The construction method as described in claim 5, characterized in that, The method also includes: in each iteration, storing all global image features, pedestrian enhancement features, and global text features encoded by the bimodal feature extraction backbone network based on the new input text into the image feature memory and the text feature memory, and combining the momentum update model based on exponential moving average to stabilize the feature extraction process of the bimodal feature extraction backbone network. Image-text feature similarity loss also includes MITC loss, whose loss function is expressed as: In the formula, The MITC loss represents the conversion from global image features to global text features encoded based on the new input text. This represents the MITC loss, which is the loss from global text features encoded based on new input text to global image features of the image. The MITC loss represents the conversion of pedestrian enhancement features in an image to global text features encoded from the new input text. The MITC loss is based on the global text features encoded from the new input text to the pedestrian enhancement features of the image. in, In the formula, For a temperature hyperparameter, B N represents the current batch size during training, and N represents the total number of image-text pairs. Indicates the first i The first batch j The true matching label of an image-text pair, if =1 indicates that the current image-text pair is a match from the same pedestrian identity. =0 indicates that the current image-text pair does not match; For the first i Global image features of all images in the batch. For the first in the text memory T j The global text features corresponding to each image-text pair, encoded based on the new input text. For the first in the text memory n The global text features corresponding to each image-text pair, encoded based on the new input text; express and The similarity between them; express and The similarity between them; exist formula lieutenant general and Change position and By changing the positions, it becomes The expression; in in the formula Indicates the first i The pedestrian enhancement features of all images in the batch are... The expression; in formula lieutenant general and Change position and By changing the positions, it becomes The expression.

7. The construction method as described in claim 1, characterized in that, The pre-trained bimodal feature extraction backbone network is composed of the image encoder and text encoder from the CLIP large model.

8. A text-based image pedestrian retrieval method, characterized in that, include: Obtain a set of pedestrian images and target text for the image to be searched, and input them into the bimodal feature extraction backbone network in the image pedestrian retrieval model, respectively, to determine the pedestrian images corresponding to the target text from the image set, thereby realizing text-based image pedestrian retrieval; The image pedestrian retrieval model is constructed using the construction method described in any one of claims 1 to 7.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method steps as described in any one of claims 1 to 7 or the method steps as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program is run by a processor, it controls the device on which the storage medium is located to perform the method steps as described in any one of claims 1 to 7 or the method steps as described in claim 8.

Images