A pedestrian search method and system based on text image matching

Abstract

The application provides a pedestrian search method and system based on text image matching, and relates to the technical field of image processing, which comprises the following steps: performing image segmentation on a pedestrian image sample to obtain a corresponding local attribute image; performing feature extraction on the local attribute image by using an MLLM model, and generating annotation text related to the local attribute image based on the extracted attribute features to obtain an initial image text pair; determining the normalized cosine similarity between the pedestrian image sample and the generated annotation text, and the noise level of a preset mark word in the annotation text by using an image encoder and a text encoder, screening the initial image text pair by using the cosine similarity and the noise level to obtain an intermediate image text pair; training a preset matching model by using the intermediate image text pair and a similarity distribution matching loss to obtain a target matching model; and matching the target annotation text corresponding to a to-be-processed pedestrian image by using the target matching model to complete the pedestrian search.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a pedestrian search method and system based on text image matching. Background Technology

[0002] Text-based person search (TBPS) remains a challenging task due to the modal heterogeneity between vision and language. Early global feature alignment methods mapped the entire image and text features to a shared global feature space by designing cross-modal matching loss functions, but this approach ignored key local information, making the model insensitive to fine-grained differences. Subsequent local feature alignment methods improved retrieval performance by constructing relationships between body parts and local text entities, but this method requires segmentation of the original image and text, which not only introduces noise but also increases computational costs. Later, implicit local feature alignment methods achieved more efficient local alignment by masking parts of the text words and using the unmasked words and local images to predict the masked content.

[0003] However, current methods of searching for people still have some problems. On the one hand, text-based datasets for searching people are still scarce, and collecting new real-world datasets faces issues of pedestrian privacy and high annotation costs. On the other hand, the text descriptions generated by multimodal large language models (MLLM) are often inevitably mixed with noise, leading to biases in the final pedestrian search results. Summary of the Invention

[0004] In view of this, the present invention proposes a pedestrian search method and system based on text-image matching.

[0005] The technical solution of the present invention is implemented as follows: The first aspect of the present invention provides a pedestrian search method based on text-image matching, comprising: Image segmentation is performed on pedestrian image samples to obtain corresponding local attribute images; The MLLM model is used to extract features from the local attribute image, and annotation text related to the local attribute image is generated based on the extracted attribute features to obtain an initial image-text pair; The image encoder and text encoder determine the cosine similarity between the pedestrian image sample and the generated annotation text after normalization, as well as the noise level of the preset marker words in the annotation text. The cosine similarity and the noise level are used to filter the initial image-text pairs to obtain intermediate image-text pairs. The intermediate image-text pair is input into a preset matching model, and the preset matching model is trained by combining similarity distribution matching loss to obtain a target matching model; the target matching model is used to match the corresponding target annotation text for the pedestrian image to be processed, thus completing the pedestrian search.

[0006] Based on the above technical solutions, preferably, the step of using an MLLM model to extract features from the local attribute image and generating annotation text related to the local attribute image based on the extracted attribute features to obtain an initial image-text pair includes: The MLLM model is guided to extract features from the local attribute image using a dynamic template library with multiple attribute slots, resulting in multiple attribute features with different focuses.

[0007] Based on the above technical solutions, preferably, the step of using a dynamic template library with multiple attribute slots to guide the MLLM model to extract features from the local attribute image, obtaining multiple attribute features with different emphases, including: Select an attribute template from a predefined template library that matches the local attribute image; The MLLM model is used to extract features from the local attribute image, and the obtained attribute features are filled into the attribute slots reserved in the attribute template to generate annotation text related to the local attribute image.

[0008] Based on the above technical solutions, preferably, the step of filtering the initial image-text pairs using the cosine similarity and the noise level to obtain intermediate image-text pairs includes: The local attribute images in the initial image-text pair are concatenated with the [CLS] tag and then input into the image encoder to generate the global image features of the pedestrian image sample; The annotation text in the initial image text pair is marked using [SOS] and [EOS] tags to obtain marked text. The marked text is then processed by an adaptive noise-aware masking algorithm and input into a text encoder to generate the text features of the marked text. Obtain the cosine similarity between the global image features and the text features, and determine the initial image-text pairs whose cosine similarity is greater than a first threshold as intermediate image-text pairs.

[0009] Based on the above technical solutions, preferably, the step of processing the marked text using an adaptive noise-aware masking algorithm and then inputting it into a text encoder to generate the text features of the marked text further includes: Obtain the global similarity between the local attribute image and the labeled text in any initial image-text pair, and the local similarity between each labeled text and the adjacent local attribute image; The noise level of each labeled text is determined based on the global similarity and the local similarity. After the labeled text is filtered by the adaptive noise-aware masking algorithm, it is input into the text encoder to generate the text features of the labeled text.

[0010] Based on the above technical solutions, preferably, the step of determining the noise level of each labeled text based on the global similarity and the local similarity, and then filtering the labeled text using an adaptive noise-aware masking algorithm, includes: In the process of filtering the labeled text using the adaptive noise-aware masking algorithm, the masking threshold decreases monotonically with the number of training rounds.

[0011] Based on the above technical solutions, preferably, the step of inputting the intermediate image-text pair into a preset matching model and training the preset matching model using a similarity distribution matching loss to obtain a target matching model includes: The intermediate image-text pairs are input into a preset matching model for forward propagation to obtain the similarity distribution matching loss for each intermediate image-text pair. The preset matching model is backpropagated and its parameters are updated based on the similarity distribution matching loss until the similarity distribution matching loss is less than the second threshold, thus obtaining the target matching model.

[0012] More preferably, a second aspect of the present invention provides a pedestrian search system based on text-image matching, comprising: an image segmentation module, a text generation module, a text filtering module, and a text matching module; wherein, The image segmentation module is configured to perform image segmentation on pedestrian image samples to obtain corresponding local attribute images; The text generation module is configured to use an MLLM model to extract features from the local attribute image and generate annotation text related to the local attribute image based on the extracted attribute features, thereby obtaining an initial image-text pair. The text filtering module is configured to determine the cosine similarity between the pedestrian image sample and the generated annotation text after normalization, as well as the noise level of the preset marker words in the annotation text, through an image encoder and a text encoder, and to filter the initial image-text pair using the cosine similarity and the noise level to obtain an intermediate image-text pair. The text matching module is configured to input the intermediate image text pair into a preset matching model, train the preset matching model by combining similarity distribution matching loss to obtain a target matching model, and use the target matching model to match the corresponding target annotation text for the pedestrian image to be processed, thereby completing the pedestrian search.

[0013] More preferably, a third aspect of the present invention provides an electronic device, including a processor and a memory; the memory has a computer program stored thereon, wherein the computer program, when executed by the processor, implements the pedestrian search method based on text image matching described in the first aspect.

[0014] More preferably, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the pedestrian search method based on text image matching described in the first aspect.

[0015] The pedestrian search method and system based on text-image matching of the present invention have the following advantages over the prior art: 1. By segmenting pedestrian image samples to obtain local attribute images, and using an MLLM model to generate annotation text to form initial image-text pairs, this approach avoids the need for large-scale collection of real-world text-image datasets. This avoids pedestrian privacy issues and high annotation costs associated with collecting new datasets, effectively alleviating the scarcity of text-based person search datasets. By calculating the normalized cosine similarity between pedestrian image samples and generated annotation text, as well as the noise level of pre-defined markers in the annotation text, and using these two metrics to filter the initial image-text pairs, this approach effectively removes image-text pairs with low similarity and high noise levels. This reduces the interference of noisy data on model training, improves data quality, ensures the training effect of the model, and enhances the accuracy and reliability of pedestrian search results.

[0016] 2. By utilizing a dynamic template library to set multiple attribute slots, the MLLM model is guided to extract features from local attribute images from different perspectives, resulting in multiple attribute features with different focuses. The attribute slots clearly define the types of features to be extracted, enabling the model to more accurately locate key information in the image. Filling the extracted attribute features into the reserved attribute slots in the attribute template to generate annotation text can cover multiple aspects of the image information, making the annotation text richer and more detailed, improving its accuracy, and facilitating flexible adjustment and expansion according to different application scenarios and task requirements.

[0017] 3. Obtain the cosine similarity between global image features and text features, and identify initial image-text pairs with a cosine similarity greater than a first threshold as intermediate image-text pairs. This quickly eliminates initial image-text pairs with mismatched image and text features, ensuring that the images and text in intermediate image-text pairs have a certain degree of correlation. Based on this, global and local similarity are used to determine the noise level of each labeled text. An adaptive noise-aware masking algorithm is then used to filter the labeled text, more comprehensively evaluating the matching degree between the labeled text and the image. This accurately identifies and filters noisy labeled text, further improving the quality of intermediate image-text pairs and providing higher-quality data for subsequent model training. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a pedestrian search method based on text-image matching provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of multi-scale feature fusion provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the adaptive noise perception masking process provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a pedestrian search system based on text image matching provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] In some embodiments, such as Figure 1 As shown, Figure 1 This is a flowchart illustrating a pedestrian search method based on text-image matching provided in an embodiment of the present invention; the pedestrian search method based on text-image matching provided by the present invention includes: S110, perform image segmentation on the pedestrian image sample to obtain the corresponding local attribute image.

[0022] In this embodiment, the LUPerson dataset can be used as the image source to obtain a large number of pedestrian image samples captured in different environments. The pedestrian image samples obtained from the LUPerson dataset are segmented to construct a multi-granular visual input. Local attribute images may include images of the pedestrian's shirt, pants, shoes, and hair, etc.

[0023] The splitting expression can be represented as: ; In the formula, It is the dot product of the binary mask corresponding to each attribute and the original image. Its principle can be expressed as: ; This represents the new image obtained after the masking operation. This represents the original image. Represents a binary mask. and These represent the height and width of the image, respectively, which are the number of pixels in the vertical and horizontal directions.

[0024] S120: Use the MLLM model to extract features from the local attribute image, and generate annotation text related to the local attribute image based on the extracted attribute features to obtain the initial image-text pair.

[0025] MLLM models, trained jointly with visual and textual sources, can understand the semantic information hidden in local attribute images, such as clothing textures, accessory details, and pose features, and automatically associate them with the text attribute space. For example, for the striped pattern features of pedestrian clothing, MLLM models can simultaneously extract visual texture patterns and semantic labels.

[0026] In some embodiments, an MLLM model is used to extract features from the local attribute image, and annotation text related to the local attribute image is generated based on the extracted attribute features to obtain an initial image-text pair, including: By using a dynamic template library with multiple attribute slots, the MLLM model is guided to extract features from local attribute images, resulting in multiple attribute features with different focuses.

[0027] Attribute slots clearly define the types of features to be extracted, enabling MLLM models to more accurately locate key information in images. For example, when processing images containing local attributes of faces, attribute slots specifically designed for facial expressions, facial features, etc., can be set up to allow the model to focus on extracting these important features, thereby improving the accuracy and efficiency of feature extraction.

[0028] In some embodiments, a dynamic template library with multiple attribute slots is used to guide the MLLM model to extract features from local attribute images, resulting in multiple attribute features with different emphases, including: Select an attribute template from the predefined template library that matches the local attribute image; The MLLM model is used to extract features from the local attribute image, and the obtained attribute features are filled into the attribute slots reserved in the attribute template to generate annotation text related to the local attribute image.

[0029] In this embodiment, multiple extracted attribute features are filled into the attribute slots reserved in the attribute template to generate annotation text. This can cover multiple aspects of image information, making the annotation text richer and more detailed. Taking a pedestrian image as an example, the generated annotation text can simultaneously include the pedestrian's appearance features, such as height and body shape, clothing details, such as clothing color and style, and items carried, providing users with a more comprehensive image description. The predefined attribute template provides a fixed framework and standard for annotation text generation, ensuring that the extracted attribute features can be accurately filled into the corresponding positions, avoiding semantic errors or information confusion caused by arbitrary feature combinations. At the same time, the one-to-one correspondence between attribute slots and attribute features also helps reduce ambiguity in the text generation process and improve the accuracy of the annotation text.

[0030] In one example, see Figure 2 , Figure 2This is a flowchart illustrating the annotation text generation stage provided in this embodiment of the invention. The original pedestrian image is segmented using a segmentation model to obtain local attribute images such as hair, shirt, pants, and shoes. Combined with the prompt "Generate a comprehensive description of the person's appearance by analyzing both the full-body image and available clear segments," a comprehensive description of the person's appearance is generated by analyzing the full-body image and available clear segments. The MLLM model generates the annotation text "A woman is walking away. She dons a vibrant red long coat, black pants, and black sneakers with white details. A dark shoulder bag hangs from her shoulder," indicating that a woman is walking away. She is wearing a vibrant red long coat, black pants, and black sneakers with white details, and a dark shoulder bag is slung over her shoulder.

[0031] S130: The image encoder and text encoder determine the cosine similarity between the pedestrian image sample and the generated annotation text after normalization, as well as the noise level of the preset marker words in the annotation text. The cosine similarity and noise level are used to filter the initial image-text pairs to obtain intermediate image-text pairs.

[0032] Considering that the text descriptions generated by the MLLM model are prone to produce illusory descriptions that do not match the image content, introducing significant noise, we first roughly calculate the similarity between the generated annotation text and the pedestrian image samples before training. We then filter out image-text pairs with low similarity by setting a threshold, and then calculate the words with high noise at the word level. These noisy words are then identified and masked.

[0033] In some embodiments, initial image-text pairs are filtered using cosine similarity and noise level to obtain intermediate image-text pairs, including: The local attribute images in the initial image-text pairs are concatenated with the [CLS] tags and then input into the image encoder to generate global image features for pedestrian image samples; The annotation text in the initial image text pair is marked using [SOS] and [EOS] tags to obtain marked text. The marked text is then processed by an adaptive noise-aware masking algorithm and input into a text encoder to generate the text features of the marked text. Obtain the cosine similarity between global image features and text features, and determine the initial image-text pairs with a cosine similarity greater than a first threshold as intermediate image-text pairs.

[0034] In this embodiment, pedestrian image samples are divided into M non-overlapping blocks. These image tags are concatenated with [CLS] tags and fed into the image encoder. The [CLS] tags from the last layer of the image encoder are embedded as global image features. For annotation text, each annotation text T is labeled as a sequence of N tags. N varies depending on the length of each sentence. The tag sequence is enclosed in [SOS] and [EOS] to indicate the start and end of the sequence. Simultaneously, the noise level of each annotation text T is examined. This noise level is calculated and stored in previous training rounds and used to perform Adaptive Noise-Aware Masking (ANAM) on T to obtain... After that, T and It is independently input into the text encoder. In the final text encoder layer, it utilizes... global features To calculate the loss, the normalized cosine similarity between the original image and the generated text description is calculated using the image encoder and text encoder.

[0035] ; in, and This refers to image text encoders and text encoders.

[0036] Calculate the dynamic threshold based on the preset retention ratio: ; It is the average masking rate. This indicates operations on hundreds place fractions.

[0037] The dataset that passes the filter is constructed by retaining the samples with the highest similarity in the top 90%.

[0038] ; In the subsequent training phase, the filtered dataset is used for training.

[0039] In some embodiments, the marked text is processed by an adaptive noise-aware masking algorithm and then input into a text encoder to generate text features of the marked text, further comprising: Obtain the global similarity between the local attribute images and the labeled text in any initial image-text pair, as well as the local similarity between each labeled text and its adjacent local attribute images; The noise level of each labeled text is determined based on global and local similarity. After the labeled text is filtered by an adaptive noise-aware masking algorithm, it is input into the text encoder to generate the text features of the labeled text.

[0040] In this embodiment, the noise level of each word is calculated at the word level, and words with high noise levels are masked out before the next training round. The noise level of the annotated text T is estimated using the labeled embeddings of the image encoder and text encoder, which are represented as follows: (M image patch labels) and (N text tags).

[0041] The global label similarity between individual text images is calculated as follows: ; in, It is a global similarity matrix, where each element... The cosine similarity between the i-th text tag embedding and the j-th image tag embedding is represented. Indicates the dimension of the embedding vector.

[0042] If a text tag doesn't match an image, the similarity score between its embedding and the embeddings of all image tags will remain low. Furthermore, since the image encoder's patch segmentation might happen to divide certain key attributes of a pedestrian into two distinct patches, a single maximum similarity calculation might underestimate the correlation between the text tag embedding and the ground truth region. In addition to calculating global similarity... It also introduces local context similarity:

[0043] ; Let m represent the local similarity between each text tag and its neighboring image patches (k*k windows), where m is the image tag index. Therefore, the noise level of the i-th text tag in the annotated text T can be estimated as follows:

[0044] ; in, It is a balance coefficient, and the variance term captures local semantic conflicts.

[0045] Through the Calculate the noise level for each row to obtain a vector. This characterizes the noise level of all text tags. Furthermore, ANAM applies a masking operation to all text tags in the annotated text T with different probabilities, which can be determined based on... The noise level value recorded in the data is used to determine this.

[0046] In some embodiments, the noise level of each labeled text is determined based on global similarity and local similarity, and the labeled text is filtered using an adaptive noise-aware masking algorithm, including: In the process of filtering labeled text using the adaptive noise-aware masking algorithm, the masking threshold decreases monotonically with the number of training rounds.

[0047] During the initial training phase, the model parameters are not fully updated. The values ​​of elements in the middle can be very high, leading to over-masking of important labels and hindering learning. To address this, a dynamic masking threshold is set that varies with the number of training epochs:

[0048] ; in, It is a protection factor that decays with the number of training rounds. The initial protection factor is set to 0.3, where t is the current training epoch and T is the total number of training epochs. Therefore, the final masking probability for each text tag is:

[0049] ; in, It is the average masking rate, so in the early stages of training A higher masking probability and a lower probability ensure that the model can update its parameters more effectively as training progresses. Approaching 0 increases the masking probability, ensuring that noisy words can be better masked.

[0050] In an alternative embodiment, please refer to Figure 3 , Figure 3This is a schematic diagram of the adaptive noise-aware masking process provided in an embodiment of the present invention. The original annotation text is filtered by an MLLM model to obtain "A woman is walking away. She dons avibrant red long coat, black pants, and black sneakers with white details. Dark shoulder bag hangs...", which means "A woman is walking away. She is wearing a bright red long coat, black pants, and black sneakers with white details. A dark shoulder bag hangs...". The adaptive noise-aware masking algorithm is then used to filter the labeled text to generate "A woman is walking away. She dons a [mask] longcoat, black pants, and black sneakers with [mask] details. A dark shoulder bag hangs...", where the red and white colors are masked.

[0051] S140, input the intermediate image-text pair into the preset matching model, and train the preset matching model by combining the similarity distribution matching loss to obtain the target matching model; use the target matching model to match the corresponding target annotation text for the pedestrian image to be processed to complete the pedestrian search.

[0052] In this embodiment, the similarity distribution matching SDM loss is used to optimize the matching model, given N matching image-text pairs. Establish pairing relationships between each image and text. ,in and Let represent negative sample pairs and positive sample pairs, respectively. Calculate the true matching distribution for the i-th image. The j-th element This represents the pairing of the i-th image with the j-th text. Finally, the probability distribution predicted by the model is aligned with the true distribution as follows:

[0053] ; in, It is a very small number, to avoid numerical problems.

[0054] Probability distribution predicted by the model as follows: ; In the formula, Represents the cosine similarity between u and v. This is the temperature coefficient.

[0055] Through the above exchange formula and The location can be used to obtain the text-to-image loss. The final complete SDM loss is: .

[0056] In some embodiments, the intermediate image-text pair is input into a preset matching model, and the preset matching model is trained by combining a similarity distribution matching loss to obtain a target matching model, including: The intermediate image-text pairs are input into the preset matching model for forward propagation, and the similarity distribution matching loss corresponding to each intermediate image-text pair is obtained. The preset matching model is backpropagated and its parameters are updated based on the similarity distribution matching loss until the similarity distribution matching loss is less than the second threshold, thus obtaining the target matching model.

[0057] In some embodiments, please refer to Figure 4 , Figure 4 This is a schematic diagram of the structure of a pedestrian search system based on text-image matching provided in an embodiment of the present invention. The present invention provides a pedestrian search system 400 based on text-image matching, comprising: an image segmentation module 410, a text generation module 420, a text filtering module 430, and a text matching module 440; wherein,

[0058] The image segmentation module 410 is configured to perform image segmentation on pedestrian image samples to obtain corresponding local attribute images; The text generation module 420 is configured to use the MLLM model to extract features from the local attribute image and generate annotation text related to the local attribute image based on the extracted attribute features, thus obtaining an initial image-text pair. The text filtering module 430 is configured to determine the cosine similarity between the pedestrian image sample and the generated annotation text after normalization, as well as the noise level of the preset marker words in the annotation text, through the image encoder and the text encoder, and to filter the initial image-text pairs using the cosine similarity and noise level to obtain intermediate image-text pairs. The text matching module 440 is configured to input intermediate image text pairs into a preset matching model, train the preset matching model by combining similarity distribution matching loss, and obtain a target matching model; and use the target matching model to match the corresponding target annotation text for the pedestrian image to be processed, thereby completing the pedestrian search.

[0059] In some embodiments, the text generation module 420 is specifically configured as follows: By using a dynamic template library with multiple attribute slots, the MLLM model is guided to extract features from local attribute images, resulting in multiple attribute features with different focuses.

[0060] In some embodiments, the text generation module block 420 is specifically configured as follows: Select an attribute template from the predefined template library that matches the local attribute image; The MLLM model is used to extract features from the local attribute image, and the obtained attribute features are filled into the attribute slots reserved in the attribute template to generate annotation text related to the local attribute image.

[0061] In some embodiments, the text filtering module block 430 is specifically configured as follows: The local attribute images in the initial image-text pairs are concatenated with the [CLS] tags and then input into the image encoder to generate global image features for pedestrian image samples; The annotation text in the initial image text pair is marked using [SOS] and [EOS] tags to obtain marked text. The marked text is then processed by an adaptive noise-aware masking algorithm and input into a text encoder to generate the text features of the marked text. Obtain the cosine similarity between global image features and text features, and determine the initial image-text pairs with a cosine similarity greater than a first threshold as intermediate image-text pairs.

[0062] In some embodiments, the text filtering module 430 is specifically configured as follows: Obtain the global similarity between the local attribute images and the labeled text in any initial image-text pair, as well as the local similarity between each labeled text and its adjacent local attribute images; The noise level of each labeled text is determined based on global and local similarity. After the labeled text is filtered by an adaptive noise-aware masking algorithm, it is input into the text encoder to generate the text features of the labeled text.

[0063] In some embodiments, the text filtering module 430 is specifically configured as follows: In the process of filtering labeled text using the adaptive noise-aware masking algorithm, the masking threshold decreases monotonically with the number of training rounds.

[0064] In some embodiments, the text matching module 440 is specifically configured as follows: The intermediate image-text pairs are input into the preset matching model for forward propagation, and the similarity distribution matching loss corresponding to each intermediate image-text pair is obtained. The preset matching model is backpropagated and its parameters are updated based on the similarity distribution matching loss until the similarity distribution matching loss is less than the second threshold, thus obtaining the target matching model.

[0065] It should be noted that the pedestrian search system based on text image matching provided in this application embodiment and the pedestrian search method based on text image matching provided in this application embodiment are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned pedestrian search method based on text image matching, and the repeated parts will not be described again.

[0066] In some embodiments, please refer to Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 500 provided in this application includes a processor 510 and a memory 520; the memory 520 stores a computer program, wherein the computer program, when executed by the processor, implements the aforementioned pedestrian search method based on text image matching.

[0067] Specifically, processor 510 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. Processor 510 may also include onboard memory for caching purposes. Processor 510 may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0068] Memory 520 may be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory 520 may include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory 520 include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and may also be random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0069] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned text-image matching-based pedestrian search method. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0070] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0071] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments, but should be defined not only by the appended claims, but also by their equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the protection scope of this invention.

Claims

1. A pedestrian search method based on text-image matching, characterized in that, include: Image segmentation is performed on pedestrian image samples to obtain corresponding local attribute images; The MLLM model is used to extract features from the local attribute image, and annotation text related to the local attribute image is generated based on the extracted attribute features to obtain an initial image-text pair; The image encoder and text encoder determine the cosine similarity between the pedestrian image sample and the generated annotation text after normalization, as well as the noise level of the preset marker words in the annotation text. The cosine similarity and the noise level are used to filter the initial image-text pairs to obtain intermediate image-text pairs. The intermediate image-text pair is input into a preset matching model, and the preset matching model is trained by combining similarity distribution matching loss to obtain a target matching model; the target matching model is used to match the corresponding target annotation text for the pedestrian image to be processed, thus completing the pedestrian search.

2. The pedestrian search method based on text-image matching as described in claim 1, characterized in that, The step involves using an MLLM model to extract features from the local attribute image and generating annotation text related to the extracted attribute features to obtain initial image-text pairs, including: The MLLM model is guided to extract features from the local attribute image using a dynamic template library with multiple attribute slots, resulting in multiple attribute features with different focuses.

3. The pedestrian search method based on text-image matching as described in claim 2, characterized in that, The method utilizes a dynamic template library with multiple attribute slots to guide the MLLM model in extracting features from the local attribute image, resulting in multiple attribute features with different emphases, including: Select an attribute template from a predefined template library that matches the local attribute image; The MLLM model is used to extract features from the local attribute image, and the obtained attribute features are filled into the attribute slots reserved in the attribute template to generate annotation text related to the local attribute image.

4. The pedestrian search method based on text-image matching as described in claim 1, characterized in that, The step of filtering the initial image-text pairs using the cosine similarity and the noise level to obtain intermediate image-text pairs includes: The local attribute images in the initial image-text pair are concatenated with the [CLS] tag and then input into the image encoder to generate the global image features of the pedestrian image sample; The annotation text in the initial image text pair is marked using [SOS] and [EOS] tags to obtain marked text. The marked text is then processed by an adaptive noise-aware masking algorithm and input into a text encoder to generate the text features of the marked text. Obtain the cosine similarity between the global image features and the text features, and determine the initial image-text pairs whose cosine similarity is greater than a first threshold as intermediate image-text pairs.

5. The pedestrian search method based on text-image matching as described in claim 4, characterized in that, The step of processing the marked text using an adaptive noise-aware masking algorithm and then inputting it into a text encoder to generate text features of the marked text further includes: Obtain the global similarity between the local attribute image and the labeled text in any initial image-text pair, and the local similarity between each labeled text and the adjacent local attribute image; The noise level of each labeled text is determined based on the global similarity and the local similarity. After the labeled text is filtered by the adaptive noise-aware masking algorithm, it is input into the text encoder to generate the text features of the labeled text.

6. The pedestrian search method based on text-image matching as described in claim 5, characterized in that, The process of determining the noise level of each labeled text based on the global similarity and the local similarity, and then filtering the labeled text using an adaptive noise-aware masking algorithm, includes: In the process of filtering the labeled text using the adaptive noise-aware masking algorithm, the masking threshold decreases monotonically with the number of training rounds.

7. The pedestrian search method based on text-image matching as described in claim 1, characterized in that, The step of inputting the intermediate image-text pair into a preset matching model and training the preset matching model using a similarity distribution matching loss to obtain a target matching model includes: The intermediate image-text pairs are input into a preset matching model for forward propagation to obtain the similarity distribution matching loss for each intermediate image-text pair. The preset matching model is backpropagated and its parameters are updated based on the similarity distribution matching loss until the similarity distribution matching loss is less than the second threshold, thus obtaining the target matching model.

8. A pedestrian search system based on text-image matching, characterized in that, include: The system includes an image segmentation module, a text generation module, a text filtering module, and a text matching module; among them, The image segmentation module is configured to perform image segmentation on pedestrian image samples to obtain corresponding local attribute images; The text generation module is configured to use an MLLM model to extract features from the local attribute image and generate annotation text related to the local attribute image based on the extracted attribute features, thereby obtaining an initial image-text pair. The text filtering module is configured to determine the cosine similarity between the pedestrian image sample and the generated annotation text after normalization, as well as the noise level of the preset marker words in the annotation text, through an image encoder and a text encoder, and to filter the initial image-text pair using the cosine similarity and the noise level to obtain an intermediate image-text pair. The text matching module is configured to input the intermediate image text pair into a preset matching model, train the preset matching model by combining similarity distribution matching loss to obtain a target matching model, and use the target matching model to match the corresponding target annotation text for the pedestrian image to be processed, thereby completing the pedestrian search.

9. An electronic device comprising a processor and a memory; said memory storing a computer program, wherein, When the computer program is executed by the processor, it implements the pedestrian search method based on text image matching as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, characterized in that, It stores a computer program, wherein the computer program, when executed by a processor, implements the pedestrian search method based on text image matching as described in any one of claims 1 to 7.

Images