A part-of-speech guided hierarchical semantic alignment cross-modal pedestrian re-identification method and system

Abstract

The application provides a hierarchical semantic alignment cross-modal pedestrian re-identification method and system based on part-of-speech guidance. The method comprises: constructing a pedestrian re-identification model, including a modal-specific feature embedding module for extracting feature representations of input pedestrian images and text, and generating an image affinity matrix and a text affinity matrix; a global cross-modal alignment module for synchronously performing cross-modal complementary feature learning and global semantic similarity alignment; a local salient region alignment module for filtering and retaining the most salient tokens using an attention-driven token selection mechanism; a part-of-speech-aware implicit fine-grained alignment module for masking and then reconstructing the text tokens using part-of-speech-related language prior knowledge; setting a loss function of the pedestrian re-identification model, and training the pedestrian re-identification model by minimizing the loss function; inputting a search text or a pedestrian image into the trained pedestrian re-identification model, and outputting a matched pedestrian image.

Description

Technical Field

[0001] This invention relates to the field of pedestrian re-identification technology, and in particular to a hierarchical semantic alignment cross-modal pedestrian re-identification method and system based on part-of-speech guidance. Background Technology

[0002] Text-to-Image Person Re-identification (TIReID) is a crucial cross-modal retrieval task that aims to identify target pedestrian images from a large-scale image database based on given text descriptions. This technology holds significant application potential in public safety and smart city development, enabling solutions such as locating missing persons based on detailed descriptions of clothing, appearance, and other characteristics provided by companions. As a subfield that integrates the challenges of text-image retrieval and person re-identification, TIReID requires models not only to bridge significant differences between modalities but also to extract highly discriminative identity feature representations.

[0003] In recent years, the field of text-image person re-identification has seen rapid development. Early related technologies mainly relied on global features output by visual and text encoders to achieve cross-modal alignment. These global alignment methods excel at capturing overall semantic information, ensuring global consistency between text descriptions and image content. Furthermore, global features are naturally robust to local noise and irrelevant details, providing a reliable initial alignment foundation. Subsequent related technologies focus on establishing precise, fine-grained alignments between image regions and text words and phrases through auxiliary models or implicit attention mechanisms. This paradigm solves the semantic ambiguity problem by directly associating linguistic elements with their corresponding visual parts, which is crucial for distinguishing pedestrians who have similar overall appearances but differ in key local features.

[0004] Despite significant advancements in feature extraction techniques, a clear performance bottleneck remains due to substantial cross-modal differences and subtle intra- and extramodal variations across different identities. This invention argues that the root of this challenge lies not only in the quality of the extracted features themselves but also in the lack of a collaborative learning objective, which fails to explicitly guide features to achieve cross-modal alignment across multiple semantic levels. Summary of the Invention

[0005] To address the problem that existing text-image person re-identification methods cannot achieve cross-modal alignment at multiple semantic levels by explicitly guiding features, resulting in insufficient accuracy of the identification results, this invention provides a hierarchical semantic alignment cross-modal person re-identification method and system based on part-of-speech guidance.

[0006] In a first aspect, embodiments of the present invention provide a hierarchical semantic alignment cross-modal person re-identification method based on part-of-speech guidance, comprising:

[0007] Obtain a set of pedestrian images and a set of text describing each pedestrian image;

[0008] A pedestrian re-identification model is constructed, including a modality-specific feature embedding module, a global cross-modal alignment module, a local salient region alignment module, and a part-of-speech-aware implicit fine-grained alignment module;

[0009] The modality-specific feature embedding module is used to extract feature representations of the input pedestrian image and text, and generate image affinity matrix and text affinity matrix; the global cross-modal alignment module is used to simultaneously perform cross-modal complementary feature learning and global semantic similarity alignment based on image feature representation and text feature representation; the local salient region alignment module is used to filter and retain the most salient tokens based on the image affinity matrix and text affinity matrix using an attention-driven token selection mechanism; the part-of-speech-aware implicit fine-grained alignment module is used to mask text tokens using part-of-speech-related language prior knowledge, and then guide the model to reconstruct text tokens through self-supervised learning;

[0010] Set a loss function for the pedestrian re-identification model, use the pedestrian image set and the text set as training datasets, and train the pedestrian re-identification model by minimizing the loss function;

[0011] Input the retrieved text or pedestrian image into the trained pedestrian re-identification model, and output the matched pedestrian image.

[0012] Furthermore, the modality-specific feature embedding module uses the CLIP model to extract feature representations of pedestrian images and text.

[0013] Furthermore, the cross-modal complementary feature learning includes:

[0014] The global features in the feature representations of pedestrian images and text are fused to obtain the fused global features;

[0015] The fused global features are input into the classifier to obtain the predicted identity, and the complementary identity loss between the predicted identity and the real identity is calculated.

[0016] Furthermore, the global semantic similarity alignment includes:

[0017] Calculate the cosine similarity score between global features in the feature representations of pedestrian images and text, and convert the cosine similarity score into a probability distribution using the softmax function, denoted as the prediction distribution; calculate the KL divergence between the prediction distribution and the true label distribution and use it as the global similarity distribution matching loss.

[0018] Furthermore, the filtering process of the local salient region alignment module includes:

[0019] Based on the attention scores between the global image tokens and all local image tokens given by the image affinity matrix, the set of local image tokens with the highest attention scores is retained and denoted as the visual token set; based on the attention scores between the global text tokens and all local text tokens given by the text affinity matrix, the set of local text tokens with the highest attention scores is retained and denoted as the text token set.

[0020] Visual token sets and text token sets are projected into a common feature space through a learnable linear layer, and then max pooling is used to aggregate the transformed features to obtain local image salient embedding features and local text salient embedding features.

[0021] Calculate the cosine similarity score between local image salient embedding features and local text salient embedding features. Convert the cosine similarity score into a probability distribution using the softmax function, denoted as the similarity distribution. Calculate the KL divergence between the similarity distribution and the true label distribution and use it as the local salient alignment loss.

[0022] Furthermore, the masking process of the part-of-speech awareness implicit fine-grained alignment module includes: firstly, verifying whether each token in the input text sequence belongs to the valid vocabulary; then, tagging and classifying the decoded tokens according to their parts of speech; next, based on the part-of-speech classification results, employing a differentiated masking strategy for tokens of different parts of speech, including: assigning higher masking probabilities to noun, adjective, and verb tokens, and assigning lower masking probabilities to adverb, article, and preposition tokens; finally, for the selected masked tokens, replacing 80% with [MASK] tokens, replacing 10% with random tokens, and leaving the remaining 10% unchanged, thereby obtaining the masked text;

[0023] The reconstruction process of the part-of-speech awareness implicit fine-grained alignment module includes: denoting the masked text as... Extract text Feature representation; in text The feature representations of the pedestrian images are used as query vectors, and the feature representations of the pedestrian images are used as key and value vectors. The query, key, and value vectors are jointly input into a multimodal interactive encoder based on cross-attention; the output features of the multimodal interactive encoder are then analyzed. The predicted text is obtained through a multilayer perceptron classifier; the loss between the predicted text and the original text is calculated; wherein, the processing procedure of the multimodal interactive encoder is represented as follows:

[0024]

[0025]

[0026] in, The output feature of the multimodal interactive encoder is S, where S represents text. Length; Representation layer normalization, This represents a multi-head cross-attention mechanism, where d represents the feature dimension.

[0027]

[0028] in, This represents the loss between the predicted text and the original text. Represents the set of masked token indices. Indicates the size of the vocabulary. This represents the predicted probability of the j-th word in the vocabulary at the i-th mask position. This represents the actual label distribution on the vocabulary.

[0029] Furthermore, the loss function of the pedestrian re-identification model is:

[0030]

[0031] in, This represents the loss corresponding to global semantic similarity alignment in the global cross-modal alignment module. The loss corresponds to the alignment module for locally significant regions. The loss is used for learning cross-modal complementary features in the global cross-modal alignment module. This represents the loss corresponding to the part-of-speech awareness implicit fine-grained alignment module. and As weight;

[0032] Correspondingly, it also includes: training the pedestrian re-identification model by combining a dynamic loss scheduling strategy; the dynamic loss scheduling strategy refers to dynamically adjusting the weights according to the following formula. and :

[0033] |

[0034] |

[0035] in, Indicates the total number of training rounds. and Indicates the initial value. and They represent and The rate of change.

[0036] Secondly, the present invention provides a hierarchical semantic alignment cross-modal person re-identification system based on part-of-speech guidance, comprising:

[0037] The dataset acquisition module is used to acquire a set of pedestrian images and a set of text describing each pedestrian image;

[0038] The model building module is used to construct a pedestrian re-identification model, including a modality-specific feature embedding module, a global cross-modal alignment module, a local salient region alignment module, and a part-of-speech-aware implicit fine-grained alignment module. Specifically, the modality-specific feature embedding module extracts feature representations from the input pedestrian image and text, and generates image affinity matrices and text affinity matrices. The global cross-modal alignment module simultaneously performs cross-modal complementary feature learning and global semantic similarity alignment based on image and text feature representations. The local salient region alignment module uses an attention-driven token selection mechanism based on the image and text affinity matrices to filter and retain the most salient tokens. The part-of-speech-aware implicit fine-grained alignment module masks text tokens using part-of-speech-related linguistic prior knowledge, and then guides the model to reconstruct the text tokens through self-supervised learning.

[0039] The model training module is used to set the loss function of the pedestrian re-identification model, use the pedestrian image set and the text set as training datasets, and train the pedestrian re-identification model by minimizing the loss function;

[0040] The retrieval module is used to input retrieval text or pedestrian images into the trained pedestrian re-identification model and output the matched pedestrian images.

[0041] Thirdly, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in the first aspect.

[0042] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in the first aspect.

[0043] The beneficial effects of this invention are as follows:

[0044] This invention proposes a language-prior-guided hierarchical semantic alignment framework for high-performance text-image person re-identification. This framework can simultaneously achieve cross-modal alignment of global features, local salient regions, and part-of-speech-aware local regions. To further optimize model performance, this invention also proposes a dynamic loss scheduling mechanism.

[0045] For global cross-modal alignment, traditional methods typically project two modalities into a unified embedding space using a shared classifier. However, these methods often fail to clearly model the key interactions between visual and textual features at the identity level, resulting in persistent modal differences within the feature space corresponding to the same identity. To address this, this invention fuses the feature representations of the two modalities, learns complementary cross-modal features, and strengthens their intrinsic correlation. Simultaneously, it ensures overall semantic consistency and improves the robustness of model matching through global semantic similarity distribution alignment.

[0046] The design motivation for aligning locally salient regions stems from the observation that raw fine-grained features are often dominated by irrelevant noise, such as cluttered background areas in pedestrian images, punctuation marks in text descriptions, or common function words—all non-discriminative tokens. This noise severely weakens the discriminative power of features, ultimately affecting retrieval accuracy. To address this issue, this invention employs an attention-driven token selection mechanism to filter and retain the most salient tokens for mining discriminative local feature representations.

[0047] To address part-of-speech-aware local region alignment, this invention improves Masked Language Modeling (MLM) to implicitly establish a stronger association between textual concepts and their visual counterparts. This process enhances fine-grained implicit alignment while guiding the model to learn hierarchical discriminative feature representations. Masked Language Modeling, a core pre-training objective in language models, has been used as an auxiliary task in the fine-tuning stage of text-image person re-identification to enhance fine-grained cross-modal interactions. By recovering the masked tokens, the model is forced to integrate visual cues with linguistic context, implicitly promoting fine-grained cross-modal matching. However, standard Masked Language Modeling methods randomly select tokens to be masked with uniform probability, which is inefficient for the framework of this invention because it forces the model to allocate significant computational resources to predicting visually meaningless words, resulting in insufficient training signals for fine-grained visual attributes crucial for accurate cross-modal matching. To address this issue, this invention observes that nouns, adjectives, and verbs constitute the semantically rich core vocabulary in person descriptions, carrying key discriminative information. To address this, this invention proposes a language-prior-guided masking strategy. This strategy assigns higher masking probabilities to descriptive words related to appearance, while maintaining standard masking probabilities for other tokens. Through this strategy, the model focuses more attention on key visual attributes in the text; simultaneously, it forces the model to rely more on visual cues to reconstruct these high-value tokens, thereby establishing a stronger association between textual concepts and their visual counterparts, achieving implicit fine-grained region alignment.

[0048] The hierarchical alignment of global features, local salient regions, and part-of-speech tagging (POS)-aware local regions constitutes a coherent multi-granularity learning paradigm. However, the static combination of these different learning objectives may lead to continuous optimization conflicts, limiting the potential of each alignment module. To address this core optimization challenge, this invention further proposes a dynamic loss scheduling mechanism. By adaptively balancing the interactions between different losses, it ensures that each learning objective can most effectively guide the model's learning at different stages of training. Attached Figure Description

[0049] Figure 1 A flowchart illustrating a hierarchical semantic alignment cross-modal person re-identification method based on part-of-speech guidance provided in an embodiment of the present invention;

[0050] Figure 2 This is a schematic diagram of the structure of a multimodal interactive encoder based on cross-attention provided in an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of the structure of a hierarchical semantic alignment cross-modal person re-identification system based on part-of-speech guidance provided in an embodiment of the present invention;

[0052] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] Text-image person re-identification technology faces a dual challenge: on the one hand, it needs to bridge the significant modal differences between visual and linguistic approaches; on the other hand, it needs to distinguish subtle appearance differences between pedestrians of different identities. This invention argues that the limitations of existing technologies do not stem from the quality of feature extraction, but rather from the lack of collaborative learning strategies capable of achieving multi-level semantic alignment. To address this issue, this invention provides a part-of-speech-guided hierarchical semantic alignment cross-modal person re-identification method, comprising the following steps:

[0055] S101: Obtain the pedestrian image set and the text set describing each pedestrian image;

[0056] S102: Construct a pedestrian re-identification model;

[0057] Specifically, this invention proposes a language-prior-guided hierarchical semantic alignment framework as a person re-identification model. The structure of this semantic alignment framework is as follows: Figure 1 As shown, the semantic alignment framework includes a modality-specific feature embedding module, a global cross-modal alignment module, a local salient region alignment module, and a part-of-speech-aware implicit fine-grained alignment module. The hierarchical structure of this semantic alignment framework helps extract discriminative multi-granularity identity features across two modalities.

[0058] The modality-specific feature embedding module is used to extract feature representations of the input pedestrian images and text, and generate image affinity matrices and text affinity matrices. The global cross-modal alignment module is used to simultaneously perform cross-modal complementary feature learning and global semantic similarity alignment based on image feature representations and text feature representations. The local salient region alignment module is used to filter and retain the most salient tokens based on the image affinity matrix and text affinity matrix using an attention-driven token selection mechanism, which is used to mine discriminative local feature representations. As a core module, the part-of-speech-aware implicit fine-grained alignment module is used to mask text tokens using part-of-speech-related language prior knowledge, and then guides the model to reconstruct text tokens through self-supervised learning. This process enhances fine-grained implicit alignment while guiding the model to learn hierarchical discriminative feature representations.

[0059] S103: Set the loss function of the pedestrian re-identification model, use the pedestrian image set and the text set as training datasets, and train the pedestrian re-identification model by minimizing the loss function;

[0060] S104: Input the retrieved text or pedestrian image into the trained pedestrian re-identification model, and output the matched pedestrian image.

[0061] The pedestrian re-identification method provided in this invention addresses global cross-modal alignment by fusing feature representations from two modalities to learn complementary cross-modal features and strengthen their intrinsic correlation. Simultaneously, it ensures overall semantic consistency and improves the robustness of model matching through global semantic similarity distribution alignment. For local salient region alignment, this invention employs an attention-driven token selection mechanism to filter and retain the most salient tokens for mining discriminative local feature representations. For part-of-speech-aware local region alignment, this invention observes that nouns, adjectives, and verbs constitute the semantically rich core vocabulary in pedestrian descriptions, carrying key discriminative information. Therefore, this invention proposes a language prior-guided masking strategy. Through this strategy, the model focuses more attention on key visual attributes in the text; simultaneously, it forces the model to rely more on visual cues to reconstruct these high-value tokens, thereby establishing a stronger correlation between textual concepts and their visual counterparts, achieving implicit fine-grained region alignment. The hierarchical alignment of global features, local salient regions, and part-of-speech-aware local regions constitutes a coherent multi-granularity learning paradigm.

[0062] In one embodiment, the modality-specific feature embedding module extracts modality-specific feature representations based on the CLIP model. The specific implementation is as follows:

[0063] The image encoder (ViT) in the CLIP model is used to process the input pedestrian images. Encoding is performed; where H is the image height, W is the image width, and C is the number of color channels. The image encoding process includes: first, segmenting image I into... A set of non-overlapping image patches, where P is the patch size; these patches are then flattened into a one-dimensional token sequence using a trainable fully connected layer. A learnable [CLS] token is added before the sequence as a global feature, and positional information is incorporated into the segmented image patches to represent the spatial location of each patch. The sequence is then input into an L-layer Transformer block to learn the relationships between the image patches, ultimately yielding the transformed image features. ,in These are global features of the image.

[0064] The CLIP text encoder is used to extract text feature representations. The text encoding process includes: for the input text description T, the sentence is first converted into a sub-word token sequence using the CLIP text tokenizer; the text is tokenized using byte-pair encoding (BPE) with a vocabulary size of 49152, and then special tokens [SOS] and [EOS] are added to both ends of the token sequence for labeling; this text token sequence is then input into the L-layer Transformer model of the text encoder to obtain the final text features. ={},in As a global feature of the text.

[0065] In one embodiment, the cross-modal complementary feature learning process is implemented as follows: To learn complementary features, this embodiment utilizes the fused features of two modalities for identity classification. Specifically, for global image features... and text global features The fused global features are defined as follows:

[0066] ,

[0067] in This indicates a concatenation operation; the fused features are then input into the classifier. Classifier Used to map fused features to pedestrian identity label space; calculate complementary identity (CID) loss:

[0068] ,

[0069] in, Represents cross-entropy loss, This indicates a real identity tag.

[0070] The core idea of ​​this invention is to achieve implicit cross-modal alignment by explicitly fusing information from two modalities; classifier Accurate identity prediction requires identifying and relying on consistent and complementary, highly distinctive features in both vision and language.

[0071] In one embodiment, to further achieve semantic alignment between the two modalities at the global level, this invention introduces a global similarity distribution matching loss. Specifically, the cosine similarity score of the global features of the image and text is first calculated. , The cosine similarity function is used; the score is then converted into a probability distribution using the softmax function. , This represents the softmax function; it minimizes the predicted probability distribution relative to the true label distribution. Global alignment is achieved using the KL divergence between them; where For one-hot vectors, the elements corresponding to the true matches are set to 1, and the rest are set to 0; the global similarity distribution matching loss is defined as:

[0072]

[0073] Where Q represents the number of image-text pairs in a batch, This refers to the temperature parameter.

[0074] To address the issue of distinguishing between different identities that are highly similar in overall appearance but differ in fine-grained attributes, the specific implementation of the local salient region alignment module proposed in this embodiment of the invention is as follows:

[0075] Features output by the image encoder Features of the text encoder output All are generated independently by L layers of Transformers; within each Transformer layer, the self-attention mechanism generates an affinity matrix. The affinity matrix output by the last Transformer layer. The attention weights given in the first row integrate high-level semantic information between the global token and all local tokens. To preserve salient information and filter noise, for each modality, only a few local tokens with the highest attention scores are retained, called the top local token set: for an input image containing N image patches... Filtering out the top local token set of vision from fine-grained features For input text containing M words Filter out the top local token set of the text. ;in , ;w represents the token selection ratio between images and text; in this scheme, w is set to 30%; "The overall representation represents the set of the top 30% of image features that are most informative," "The overall representation is the set of the top 30% of text features selected as the most informative. Subsequently, the visual token set is processed through a learnable linear layer." and text token sets Project the transformed features onto a common feature space; then, use max pooling to aggregate the transformed features, capturing the most salient information between tokens to obtain fine-grained locally salient embedding features. and The process is described as follows:

[0076]

[0077]

[0078] in, Represents the max pooling function. This represents a multilayer perceptron. This indicates a fully connected layer.

[0079] Local salient alignment loss By distributing the similarity of distinctive local regions With the true distribution Alignment is achieved, thereby improving the model's fine-grained matching and discrimination capabilities, and is defined as follows:

[0080]

[0081] To further enhance the fine-grained cross-modal alignment effect, this embodiment of the invention extracts implicit associations and strengthens the discriminative ability of feature embeddings based on Masked Language Modeling (MLM). Unlike existing technologies, this embodiment proposes a part-of-speech-aware implicit fine-grained alignment module that integrates prior language knowledge, guiding the model to focus its learning attention on the words with the highest semantic value (i.e., nouns, adjectives, and verbs representing key information about a person's appearance). The specific implementation is as follows:

[0082] (1) Part-of-speech analysis

[0083] Existing standard masking language modeling masks text words with uniform probability and treats all words equally. However, the information content of different categories of words differs significantly. This uniform masking probability seriously affects the performance of text-based pedestrian retrieval.

[0084] To address the aforementioned issues, this invention performs part-of-speech tagging before masking inference. The specific process is as follows: First, it verifies whether each token in the input text sequence belongs to a valid vocabulary; then, it uses the Natural Language Toolkit (NLTK) to perform part-of-speech tagging on the decoded tokens (i.e., decoding the token ID into the corresponding word string) and classifies all the tagged tokens by part of speech; based on the part-of-speech classification results, this invention adopts a differentiated masking strategy for tokens of different parts of speech: tokens carrying core content (including nouns, adjectives, and verbs) are assigned a higher masking probability of 25%, while functional words such as adverbs, articles, and prepositions maintain a conventional masking probability of 15%, consistent with the original masking language modeling; for selected masked tokens, the BERT protocol is followed: 80% are replaced with [MASK] tokens, 10% are replaced with random tokens, and 10% remain unchanged; finally, the masked text is obtained.

[0085] Specifically, for a given text segment, a text start marker (SOT) and a text end marker (EOT) are typically added at the beginning and end, respectively. For text sequences with a length of less than 77 tokens, zero padding is added. Since the above three markers do not require masking, this embodiment first determines whether each token is a valid vocabulary, that is, ensures that the token is not a start marker or an end marker, and that zero padding is applied.

[0086] The total vocabulary size of CLIP is typically 49408. Among them:

[0087] 0: Padding.

[0088] 1 ~ 49404: Valid vocabulary (actual words and sub-words).

[0089] 49405: <|startoftext|> (SOT).

[0090] 49406: <|endoftext|> (EOT).

[0091] Therefore, the "effective vocabulary" refers to the set of tokens that carry actual semantics after removing padding characters and special tokens, and only these tokens are masked.

[0092] (2) Implicit fine-grained alignment

[0093] For ease of description, the masked text sequence will be denoted as... The token output feature is obtained by inputting it into the CLIP text encoder. S is a text sequence The length of this feature; Image-specific features A common input based on cross-attention multimodal interactive encoder (as shown in Figure 2), where the masked text representation... As a query (Q), the image representation As keys (K) and values ​​(V), full interaction between the image and the masked text representation is achieved through:

[0094]

[0095]

[0096] Output It is a contextual representation corresponding to each text token, incorporating visual evidence; where Representation layer normalization, This represents a multi-head cross-attention mechanism, where d represents the embedding dimension of the mask token.

[0097] Features based on masked tokens Predict the original token using a multilayer perceptron classifier: The corresponding loss function is defined as follows:

[0098]

[0099] in, Represents the set of masked token indices. Indicates the size of the vocabulary. This represents the predicted probability of the j-th word in the vocabulary at the i-th mask position. This represents the actual label distribution on the vocabulary.

[0100] In summary, the loss function of the pedestrian re-identification model can be expressed as:

[0101]

[0102] Among them, complementary identity loss It tends to capture complementary information between modes, while the distribution alignment loss " This forces a direct correspondence between modalities; and For weights.

[0103] If the traditional static weight combination method is adopted (i.e. and Once set, these optimization objectives (which remain unchanged) can lead to continuous competition between them, potentially hindering model convergence and resulting in poor performance. Therefore, this invention further proposes a dynamic loss scheduling strategy for training the model. Specifically, in the initial training phase, to initially establish a discriminative spatial structure, a dynamic loss scheduling strategy is proposed. and Assign larger weights As the training progresses, gradually reduce... At the same time, gradually increase The weight β is used to enhance intra-class compactness. In this invention, β and β change linearly at a constant rate throughout the training process until they reach 0 and 1 respectively, as expressed below:

[0104] |

[0105] |

[0106] in, Indicates the total number of training rounds. and Indicates the initial value. and They represent and The rate of change. Through dynamic loss optimization, the model can learn powerful and discriminative feature representations in both modalities. In this scheme, and All are set to 1 / 50.

[0107] This invention ensures that each learning objective can most effectively guide the model's learning at different stages of training by adaptively balancing the interaction between different losses.

[0108] To demonstrate the effectiveness of this invention, extensive experiments were conducted on three benchmark datasets: CUHK-PEDES, ICFG-PEDES, and RSTPReid. On the publicly available large-scale person re-identification database CUHK-PEDES, the proposed solution achieved a 76.43% correct search rate for the first matching image and an average precision of 67.92%. On the large-scale person re-identification database ICFG-PEDES, the proposed solution achieved a 66.89% correct search rate for the first matching image and an average precision of 40.01%. On RSTPReid, the proposed solution achieved a 63.60% correct search rate for the first matching image and an average precision of 48.50%.

[0109] Based on the same inventive concept, such as Figure 3 As shown, this embodiment of the invention provides a hierarchical semantic alignment cross-modal person re-identification system based on part-of-speech tagging, including a dataset acquisition module, a model building module, a model training module, and a retrieval module;

[0110] The dataset acquisition module is used to acquire a set of pedestrian images and a set of text describing each pedestrian image;

[0111] The model building module is used to construct a pedestrian re-identification model, including a modality-specific feature embedding module, a global cross-modal alignment module, a local salient region alignment module, and a part-of-speech-aware implicit fine-grained alignment module. Specifically, the modality-specific feature embedding module extracts feature representations from the input pedestrian image and text, and generates image affinity matrices and text affinity matrices. The global cross-modal alignment module simultaneously performs cross-modal complementary feature learning and global semantic similarity alignment based on image and text feature representations. The local salient region alignment module uses an attention-driven token selection mechanism based on the image and text affinity matrices to filter and retain the most salient tokens. The part-of-speech-aware implicit fine-grained alignment module masks text tokens using part-of-speech-related linguistic prior knowledge, and then guides the model to reconstruct the text tokens through self-supervised learning.

[0112] The model training module is used to set the loss function of the pedestrian re-identification model, use the pedestrian image set and the text set as training datasets, and train the pedestrian re-identification model by minimizing the loss function;

[0113] The retrieval module is used to input retrieval text or pedestrian images into the trained pedestrian re-identification model and output the matched pedestrian images.

[0114] It should be noted that the part-of-speech guided hierarchical semantic alignment cross-modal pedestrian re-identification system provided in this embodiment of the invention is for implementing the above method. Its specific functions can be referred to the above method embodiments, and will not be repeated here.

[0115] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include: a processor 401, a communication interface 402, a memory 403, and a communication bus 404. The processor 401, communication interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions in the memory 403 to execute a part-of-speech-guided hierarchical semantic alignment cross-modal pedestrian re-identification method. This method includes: acquiring a set of pedestrian images and a set of text describing each pedestrian image; constructing a pedestrian re-identification model, including a modality-specific feature embedding module, a global cross-modal alignment module, a local salient region alignment module, and a part-of-speech-aware implicit fine-grained alignment module; wherein the modality-specific feature embedding module is used to extract feature representations of the input pedestrian images and text, and generate image affinity matrices and text affinity matrices; the global cross-modal alignment module is used to simultaneously execute cross-modal complementary features based on image feature representations and text feature representations. The learning process aligns with global semantic similarity; the local salient region alignment module uses an attention-driven token selection mechanism based on image affinity and text affinity matrices to filter and retain the most salient tokens; the part-of-speech-aware implicit fine-grained alignment module uses part-of-speech-related language prior knowledge to mask text tokens, and then guides the model to reconstruct text tokens through self-supervised learning; a loss function for the pedestrian re-identification model is set, and the pedestrian image set and the text set are used as training datasets, and the pedestrian re-identification model is trained by minimizing the loss function; the retrieved text or pedestrian image is input into the trained pedestrian re-identification model, and the matched pedestrian image is output.

[0116] Furthermore, when the logical instructions in the aforementioned memory 403 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] This invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer can execute a part-of-speech-guided hierarchical semantic alignment cross-modal pedestrian re-identification method provided in the above-described method embodiments.

[0118] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the part-of-speech-guided hierarchical semantic alignment cross-modal pedestrian re-identification method provided in the above-described method embodiments.

[0119] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hierarchical semantic alignment cross-modal person re-identification method based on part-of-speech tagging, characterized in that, include: Obtain a set of pedestrian images and a set of text describing each pedestrian image; A pedestrian re-identification model is constructed, including a modality-specific feature embedding module, a global cross-modal alignment module, a local salient region alignment module, and a part-of-speech-aware implicit fine-grained alignment module; The modality-specific feature embedding module is used to extract feature representations of the input pedestrian image and text, and generate image affinity matrix and text affinity matrix; the global cross-modal alignment module is used to simultaneously perform cross-modal complementary feature learning and global semantic similarity alignment based on image feature representation and text feature representation; the local salient region alignment module is used to filter and retain the most salient tokens based on the image affinity matrix and text affinity matrix using an attention-driven token selection mechanism; the part-of-speech-aware implicit fine-grained alignment module is used to mask text tokens using part-of-speech-related language prior knowledge, and then guide the model to reconstruct text tokens through self-supervised learning; Set a loss function for the pedestrian re-identification model, use the pedestrian image set and the text set as training datasets, and train the pedestrian re-identification model by minimizing the loss function; Input the retrieved text or pedestrian image into the trained pedestrian re-identification model, and output the matched pedestrian image.

2. The method for cross-modal person re-identification based on part-of-speech guided hierarchical semantic alignment according to claim 1, characterized in that, The modality-specific feature embedding module uses the CLIP model to extract feature representations of pedestrian images and text.

3. The method for cross-modal person re-identification based on part-of-speech guided hierarchical semantic alignment according to claim 1, characterized in that, The cross-modal complementary feature learning includes: The global features in the feature representations of pedestrian images and text are fused to obtain the fused global features; The fused global features are input into the classifier to obtain the predicted identity, and the complementary identity loss between the predicted identity and the real identity is calculated.

4. The method for cross-modal person re-identification based on part-of-speech guided hierarchical semantic alignment according to claim 1, characterized in that, The global semantic similarity alignment includes: Calculate the cosine similarity score between global features in the feature representations of pedestrian images and text, and convert the cosine similarity score into a probability distribution using the softmax function, denoted as the prediction distribution; calculate the KL divergence between the prediction distribution and the true label distribution and use it as the global similarity distribution matching loss.

5. The method for cross-modal person re-identification based on part-of-speech guided hierarchical semantic alignment according to claim 1, characterized in that, The filtering process performed by the local salient region alignment module includes: Based on the attention scores between the global image tokens and all local image tokens given by the image affinity matrix, the set of local image tokens with the highest attention scores is retained and denoted as the visual token set; based on the attention scores between the global text tokens and all local text tokens given by the text affinity matrix, the set of local text tokens with the highest attention scores is retained and denoted as the text token set. Visual token sets and text token sets are projected into a common feature space through a learnable linear layer, and then max pooling is used to aggregate the transformed features to obtain local image salient embedding features and local text salient embedding features. Calculate the cosine similarity score between local image salient embedding features and local text salient embedding features. Convert the cosine similarity score into a probability distribution using the softmax function, denoted as the similarity distribution. Calculate the KL divergence between the similarity distribution and the true label distribution and use it as the local salient alignment loss.

6. The method for cross-modal person re-identification based on part-of-speech guided hierarchical semantic alignment according to claim 1, characterized in that, The masking process of the part-of-speech-aware implicit fine-grained alignment module includes: first, verifying whether each token in the input text sequence belongs to the valid vocabulary; then, tagging and classifying the decoded tokens according to their parts of speech; next, based on the part-of-speech classification results, employing a differentiated masking strategy for tokens of different parts of speech, including: assigning higher masking probabilities to noun, adjective, and verb tokens, and assigning lower masking probabilities to adverb, article, and preposition tokens; finally, for the selected masked tokens, replacing 80% with [MASK] tokens, replacing 10% with random tokens, and leaving the remaining 10% unchanged, thus obtaining the masked text; The reconstruction process of the part-of-speech awareness implicit fine-grained alignment module includes: denoting the masked text as... Extract text Feature representation; in text The feature representations of the pedestrian images are used as query vectors, and the feature representations of the pedestrian images are used as key and value vectors. The query, key, and value vectors are jointly input into a multimodal interactive encoder based on cross-attention; the output features of the multimodal interactive encoder are then analyzed. The predicted text is obtained through a multilayer perceptron classifier; the loss between the predicted text and the original text is calculated; wherein, the processing procedure of the multimodal interactive encoder is represented as follows: in, The output feature of the multimodal interactive encoder is S, where S represents text. Length; Representation layer normalization, This represents a multi-head cross-attention mechanism, where d represents the feature dimension. in, This represents the loss between the predicted text and the original text. Represents the set of masked token indices. Indicates the size of the vocabulary. This represents the predicted probability of the j-th word in the vocabulary at the i-th mask position. This represents the actual label distribution on the vocabulary.

7. The method for cross-modal person re-identification based on part-of-speech guided hierarchical semantic alignment according to claim 1, characterized in that, The loss function of the pedestrian re-identification model is: in, This represents the loss corresponding to global semantic similarity alignment in the global cross-modal alignment module. The loss corresponds to the alignment module for locally significant regions. The loss is used for learning cross-modal complementary features in the global cross-modal alignment module. This represents the loss corresponding to the part-of-speech awareness implicit fine-grained alignment module. and As weight; Correspondingly, it also includes: training the pedestrian re-identification model by combining a dynamic loss scheduling strategy; the dynamic loss scheduling strategy refers to dynamically adjusting the weights according to the following formula. and : | | in, Indicates the total number of training rounds. and Indicates the initial value. and They represent and The rate of change.

8. A hierarchical semantic alignment cross-modal person re-identification system based on part-of-speech tagging, characterized in that, include: The dataset acquisition module is used to acquire a set of pedestrian images and a set of text describing each pedestrian image; The model building module is used to construct a pedestrian re-identification model, including a modality-specific feature embedding module, a global cross-modal alignment module, a local salient region alignment module, and a part-of-speech-aware implicit fine-grained alignment module. Specifically, the modality-specific feature embedding module extracts feature representations from the input pedestrian image and text, and generates image affinity matrices and text affinity matrices. The global cross-modal alignment module simultaneously performs cross-modal complementary feature learning and global semantic similarity alignment based on image and text feature representations. The local salient region alignment module uses an attention-driven token selection mechanism based on the image and text affinity matrices to filter and retain the most salient tokens. The part-of-speech-aware implicit fine-grained alignment module masks text tokens using part-of-speech-related linguistic prior knowledge, and then guides the model to reconstruct the text tokens through self-supervised learning. The model training module is used to set the loss function of the pedestrian re-identification model, use the pedestrian image set and the text set as training datasets, and train the pedestrian re-identification model by minimizing the loss function; The retrieval module is used to input retrieval text or pedestrian images into the trained pedestrian re-identification model and output the matched pedestrian images.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images