A Person Re-identification Method Based on Visual-Language Information Interaction Gain

Abstract

This invention discloses a pedestrian re-identification method based on visual-linguistic information interaction gain, belonging to the field of cross-modal pedestrian recognition technology. The invention proposes a data processing optimization and deep interaction fusion scheme, mainly comprising four processing stages: In the data processing stage, a heatmap is generated using a saliency network, and non-uniform sampling is achieved using a grid module to amplify key region features; in the attribute extraction stage, a Grounded SAM model is used to generate a visual attribute mask, while a GPT2 model is used to parse text attributes; in the feature fusion stage, the visual attribute mask and original image features are weighted and fused, and cross-modal encoding is achieved using a CLIP model; in the loss optimization stage, a hierarchical alignment loss function is designed, and sample pairs are subdivided based on similarity to optimize the alignment of difficult samples. This invention significantly improves the accuracy of pedestrian re-identification in complex scenes through saliency sampling to enhance key feature expression, deep interaction of dual-modal attributes, and hierarchical loss optimization.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a pedestrian re-identification method based on visual-linguistic information interaction gain. Background Technology

[0002] Text-based image person re-identification has proven to have significant application value in the fields of security and surveillance, providing a novel method for identifying pedestrians through text descriptions. In practical applications, text-based image person re-identification can be used for tasks such as suspect identification and locating missing children.

[0003] However, the joint retrieval of textual and visual information not only introduces instability but also presents traditional cross-modal challenges. Therefore, designing new text-image person re-identification technologies is of great significance.

[0004] Since its emergence, the Transformer technology has been widely applied across various fields. Due to the excellent performance of VisionTransformer (ViT) and Transformer-based Bidirectional Encoder Representation (BERT), most retrieval models use them as backbone networks. Furthermore, researchers continue to explore the potential of Visual Language Pre-trained (VLP) models.

[0005] In reality, data processing is an easily overlooked aspect. In the field of person re-identification, input images are generally required to be normalized to a fixed size, such as 384×128 or 256×128. If the input image size is larger than the specified size, existing methods often use uniform downsampling to reduce the image size to the specified size. However, this may be lossy for person re-identification tasks that require information from different spatial resolutions and locations.

[0006] Meanwhile, most existing methods simply concatenate text and image features or employ shallow fusion strategies, failing to fully explore the inherent semantic relationships between the two modalities. Due to the lack of deep interaction mechanisms, the model cannot deeply guide the network to extract key image features based on text information, nor can it utilize image feature feedback to optimize the understanding of text semantics. This results in poor robustness and adaptability when facing complex situations such as occlusion and changes in viewpoint.

[0007] This invention designs a novel modality fusion loss optimization strategy and applies an innovative saliency modality interaction enhancement network structure to solve the above problems. Summary of the Invention

[0008] This invention, based on the optimization of the data sampling process for the dataset and the need to enhance the interactivity between text and images, designs a pedestrian re-identification method with enhanced visual-linguistic information interaction. In the data processing stage, this method effectively amplifies the key features of pedestrian images in the dataset. In the subsequent feature extraction and post-processing, attribute information from the visual and text inputs is extracted through a visual converter and a text converter, and positive and negative samples are further subdivided to optimize the loss function. The technical framework diagram of this invention is shown below. Figure 1 As shown.

[0009] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solution: In the data processing stage, embodiments of the present invention provide a key region magnification sampling method, which aims to solve the information loss problem caused by uniform downsampling of input images. Specifically, this is achieved through the following technical solution: In the first sampling stage, a convolutional neural network is used to generate a saliency image. This saliency image is specific to the pedestrian re-identification task because different tasks may require attention to different image target regions. In the second sampling stage, the most important image regions are sampled based on the saliency image. The sampled images are normalized to a uniform target size, and the sampled images and their corresponding text descriptions are fed back into a text-image cross-modal pedestrian re-identification network based on a contrastive language-image pre-trained model for feature extraction.

[0010] In the data processing stage, the saliency image Z is extracted by a saliency network, such as... Figure 2 As shown, the key region magnification sampling method simultaneously inputs the input image I and the saliency image Z into the sampler S. The sampler S is designed to calculate the sampled image G derived from the input image I and the saliency image Z. The mapping relationship can be expressed as: G = S(I, Z).

[0011] The sampler S is a grid segmenter. Based on the original size of the input image I, the sampler generates a grid matrix of size [batch_size, height, width, (x, y)], where (x, y) represents the pixel position on the saliency heatmap. The saliency image Z contains the pixel saliency of the input image I. By multiplying element-wise, a weighted grid is generated, and then a Gaussian filter is used to process the weighted grid. The weight represents the offset of each position in the sampled image, and regions with high saliency are assigned a larger sampling range. This method can achieve smooth saliency non-uniform sampling.

[0012] After the above non-uniform saliency sampling operation, the input image I is resampled into a sampled image I with magnified important regions. sThe reconstructed dataset is fed back into a network for text-image person re-identification to perform the person re-identification task.

[0013] The text-image person re-identification network consists of an attribute extractor, an image encoder, a text encoder, and a classification comparison optimization module. To enhance the interactivity between the input text and the input image, this invention constructs a fine-grained cross-modal relationship correspondence pattern from the perspective of the attribute correspondence between text and image.

[0014] This invention addresses the challenges of cross-modal alignment and poor interactivity in text-image person re-identification through a dual-interaction method combining visual and textual attributes. Visual attributes are obtained by semantically segmenting the image input using the image attribute extractor of this invention, while textual attribute information is generated by word segmentation and prompting using the text attribute extractor of this invention.

[0015] Specifically, such as Figure 1 As shown, the visual attributes are obtained using the semantic segmentation pre-trained model Grounded SAM, which unifies the functions of Grounding DINO and SAM. Grounding DINO processes the input textual prompts and uses pre-trained weights trained on a large-scale dataset to obtain the target detection boxes and their confidence scores indicated by the textual attribute prompts. SAM, as a large-scale semantic segmentation pre-trained model capable of "segmenting everything," excels at combining input target detection boxes for high-precision semantic segmentation. Through the joint construction of Grounding DINO and SAM, this invention can obtain highly confident semantic segmentation results using textual attribute prompts.

[0016] Text attribute segmentation was performed using the large-scale language model GPT2. As a large-scale language model, GPT2 can handle text-related operations well given prompts and commands. Furthermore, by utilizing a large-scale language model, attribute descriptions can retain phrase structure, ensuring that attribute words are not isolated noun combinations but meaningful descriptive phrases, thus reducing redundant word interference.

[0017] In the subsequent feature extraction post-processing, this invention employs a contrastive language-image pre-trained model as both an image encoder and a text encoder to extract features from the input image and text. The contrastive language-image pre-trained model is a model that, after pre-training on a large-scale image-text dataset, can be fine-tuned for other specific tasks. This invention uses CLIP as the feature extractor, where image feature extraction utilizes a visual converter with pre-trained weights within CLIP, and text feature extraction utilizes a text converter with pre-trained weights within CLIP.

[0018] This invention describes the classification and processing of text and visual attribute features. Specifically, as shown in the embodiments of the invention... Figure 3 As shown, based on the similarity metric, we categorize text-visual attribute feature matching pairs into 12 cases, resulting in 6 class pairs. The similarity metric used is cosine similarity Si(·), with the formula:

[0019]

[0020] Among them, V attr_i For the feature representation of the i-th attribute of vision, similarly, T attr_j Let be the text representation of the j-th attribute of the text. Pass the positive sample similarity threshold θ. pos Similarity threshold θ with negative samples neg The text visual attribute sample pairs are divided into 12 cases. The visual attribute segmentation and text attribute word segmentation divide the samples of the two modalities into corresponding attribute features and assign labels. The attribute features are then measured one by one for similarity. Similarity values ​​higher than a threshold are considered similar feature pairs, otherwise they are considered dissimilar feature pairs.

[0021] Based on this, for images m and n, and text x and y, when the attribute features belong to the same pedestrian ID, they can be divided into 6 types according to the attribute similarity: (1) and Similarity less than θ pos (2) and Similarity greater than θ pos (3) and Similarity less than θ pos (4) and Similarity greater than θ pos (5) and Similarity less than θ pos (6) and Similarity greater than θ pos When the attribute features belong to different pedestrian IDs, they can also be divided into the above 6 cases, namely: (7) and Similarity less than θ neg (8) and Similarity greater than θ neg (9) and Similarity less than θ neg (10) and Similarity greater than θ neg(11) and Similarity less than θ neg (12) and Similarity greater than θ neg .

[0022] The similarity judgment can be performed on the input image-text pair with attribute q. There are 6 categories for subsequent classification and comparison optimization modules. Specifically, cases (1) and (5) are false negative samples within a modality, case (3) is false negative samples between modalities, cases (8) and (12) are false positive samples within a modality, case (10) is false positive samples between modalities, cases (2), (4) and (6) are positive samples, and cases (7), (9) and (11) are negative samples.

[0023] Based on the aforementioned classification method, this invention designs a hierarchical alignment loss function to optimize the network's ability to distinguish different sample types. For cases where false positive and false negative samples within and between modalities are considered hard samples, this invention increases the penalty for these hard samples to ensure that features with the same ID and attribute are close in cross-modal space, while features with different IDs and attributes are far apart.

[0024] The core idea of ​​the loss function is to combine instance-level similarity loss, attribute-level triplet loss and contrast loss, and constrain the model from the perspectives of global feature alignment and local attribute alignment, respectively. The loss weight is increased for difficult samples, prompting the model to prioritize learning the feature differences of samples that are difficult to identify and easily confused. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a framework diagram of a pedestrian re-identification method that leverages visual-linguistic information interaction gains.

[0027] Figure 2 The present invention employs a saliency network diagram.

[0028] Figure 3 This is a hierarchical alignment structure diagram that combines global alignment and attribute alignment.

[0029] Figure 4 A visualization of attribute segmentation generated by a segmentation network and a large-scale language model.

[0030] Figure 5 This is a partial search result of the present invention on the CUHK-PEDES dataset. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] First see Figure 1 , Figure 1 This is a schematic flowchart of a saliency sampling method provided in an embodiment of the present invention. The embodiment uses a saliency network to extract a saliency image Z, and provides a sampler S to establish a mapping relationship between the pre-sampling image I, the saliency heatmap Z, and the post-sampling image G: G = S(I, Z).

[0033] In the saliency network structure, such as Figure 2 As shown, after the image is input, it first passes through a 7×7 convolutional layer. This convolutional layer uses a stride of 2 and padding of 3, and its core function is to initially extract image features. The data output from the first convolutional layer undergoes batch normalization, a non-linear transformation using the ReLU activation function, and then passes through a 3×3 max pooling layer. The max pooling layer can reduce the dimensionality of the feature map space while preserving key feature information of the image.

[0034] After the above processing, the data will be fed into a structure consisting of four consecutive residual join blocks, the specific architecture of which can be found in [reference needed]. Figure 2 After the residual blocks complete feature processing, the saliency network further mines salient features: through a 1×1 convolutional layer, the number of channels in the feature map is compressed to 1 channel, thereby generating a single-channel saliency image. To ensure the output size meets the preset standard, the network uses bilinear interpolation for upsampling, and finally uses the sigmoid activation function to normalize the data, ultimately obtaining the complete saliency heatmap output.

[0035] The saliency non-uniform sampler S consists of two core components: a saliency grid generation module and a sampling execution module. The grid generation module's workflow begins with filling the saliency heatmap, a step that ensures boundary continuity in subsequent grid generation. Next, the sampler constructs a basic grid structure with dimensions configured as [batch_size, height, width, 2], where the last dimension's value of 2 corresponds to the normalized (x, y) coordinates. By performing element-wise multiplication between the saliency heatmap and the basic grid, a weighted grid structure is obtained. Subsequently, a Gaussian filtering algorithm is used to smooth this weighted grid, achieving saliency non-uniform sampling with smoothness. Finally, this sampled image is fed into an image recognition network to complete the type identification task.

[0036] The dataset used for the text image person re-identification task is sequentially fed into the saliency non-uniform sampler S and the attribute extractor. The attribute extractor is a text attribute extractor and an image attribute extractor. Specifically, the text attribute extractor is the language large model GPT2, and the image attribute extractor is the semantic segmentation large model GroundedSAM.

[0037] like Figure 3 As shown, the extended dataset after attribute extraction is input into a text-image person re-identification network for feature extraction and loss optimization. The extended dataset includes a visual attribute mask M obtained based on specified attribute cues. att With text T with added attribute fields att Feature extraction uses CLIP-ViT / B-16 as the backbone network, combining the image target attribute mask with the original image features F. v The fused image attribute feature representation is obtained after fusing at a ratio r = 0.9. That is,

[0038] Similarly, the text T with added attribute fields att After passing through the text feature extractor, the text attribute feature representation is obtained. Subsequently, and Image-text feature pairs with attributes q, q∈A (A is the attribute set) in the attribute space. The samples are fed into the attribute-level alignment optimization process. After similarity measurement and the above classification strategy, false positive and false negative sample pairs within and between modes in this batch can be obtained. These hard sample pairs are constrained by the following formulas:

[0039] At the local attribute level, for image and text features of attribute q, this invention employs intra-modal contrastive learning loss respectively. Intermodal contrastive learning loss and attribute-level triplet loss Combinatorial optimization. The formula is shown below:

[0040]

[0041] in, The contrastive learning loss is the in-modal attribute q of the image. The contrastive learning loss for the in-modal attribute q of the text is shown in the following formula:

[0042]

[0043]

[0044] Among them, B q This represents the number of valid samples for attribute q. α(·) represents the Modified Linear Unit (RELU) activation function. F I q [i] represents the feature of attribute q in the i-th image. Let FN represent the feature of attribute q in the i-th text. All features are processed by self-attention pooling and feature fusion. i FP represents the set of false negative samples with the same ID as sample i. i Let Si(·) represent the set of false positive samples, and let Si(·) represent the cosine similarity.

[0045] Similarly, inter-modal contrastive learning loss The formula is as follows, and the variable definitions follow the meanings described above.

[0046]

[0047] After implementing hard-sample contrastive learning, the remaining false positive and false negative attribute pairs are used in the triplet loss of perceptual attributes to strengthen cross-modal alignment. For image sample i, when classifying text sample n (belonging to different identities but with high similarity scores, as a false positive sample pair) and text sample p (having the same identity as i but with low similarity scores, as a false negative sample pair), we apply triplet loss to strengthen the association between image sample i and text sample p, while distinguishing sample i from n, where ∈ is the triplet margin. The loss function is defined as follows:

[0048]

[0049] For loss optimization at the global instance and identity levels, this invention example uses instance loss L. ins Loss of pedestrian identity L id .

[0050]

[0051]

[0052]

[0053] Where y is a vector of length B (batch size), and y i =1 indicates that the i-th image-text pair belongs to the same identity; otherwise, y = 1. i =0. S I2T This is the image-to-text similarity matrix. This is the text-to-image similarity matrix, S. I2T The transpose of . τ represents the temperature parameter, set to 0.07. f I g and These represent the global features of the image and text, respectively. S i，j This represents the similarity between the i-th image and the j-th text. This represents the similarity score between the i-th sample and its direct counterpart.

[0054]

[0055] Where N represents the number of samples, This indicates that the i-th sample predicted by the network belongs to its real identity y. i The probability of.

[0056] After obtaining the losses at the local attribute level and the global instance identity level, the overall loss L adopted in this invention is... total The weighted sum of the above losses is composed of:

[0057]

[0058]

[0059]

[0060] λ ins With λ tri The instance loss L is respectively ins Loss compared to triples of attribute q Fixed weight parameters, and This is the contrastive learning loss between attribute q modes. Intramodal contrast loss The weights are dynamically adjusted. N attr This represents the number of valid attributes. A represents the set of attributes, where η is a small value used to prevent division by zero, set to 1×10. -8 Var(·) represents the variance of the loss value.

[0061] To verify the effectiveness of the present invention in the field of person re-identification in text images, the technique was tested on three benchmark datasets with different characteristics (CUHK-PEDES, ICFG-PEDES, and RSTPReid), and compared with traditional methods and state-of-the-art methods. The results show that the present invention achieves optimal or near-optimal performance on these three benchmark datasets. Figure 4 This paper presents an example of segmentation results obtained by processing a dataset using the pre-trained semantic segmentation model Grounded SAM and the large-scale language model GPT2 employed in this invention. Figure 5 This paper presents some retrieval results of the present invention on the CUHK-PEDES dataset. Through an innovative design that integrates data processing optimization and deep interactive methods, the present invention effectively solves the technical bottlenecks of key feature loss and insufficient cross-modal semantic association in text image person re-identification tasks.

[0062] The method proposed in this invention has significant application prospects in fields such as intelligent security, suspect tracking, and missing persons retrieval. Field tests demonstrate its excellent performance, providing an efficient solution for cross-modal pedestrian recognition tasks. In the future, it can be further extended to scenarios such as multimodal fusion video pedestrian re-identification and cross-camera collaborative tracking. Furthermore, it is expected to enhance feature completion capabilities under extreme conditions by introducing generative AI models, providing strong support for the technological upgrade of intelligent visual surveillance systems.

[0063] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0064] It should also be noted that, in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A pedestrian re-identification method of visual-linguistic information interaction gain, characterized in that, Comprising the following steps: First, in the data processing step, the pedestrian saliency image is generated by the saliency network, and the image is non-uniformly sampled based on the saliency image to obtain a key region enlarged sampled image; Then, in the attribute extraction step, the visual attribute mask of the sampled image and the text with added attribute field corresponding to the text description are extracted by the image attribute extractor and the text attribute extractor respectively, wherein the image attribute extractor adopts the Grounded SAM model to generate high-confidence semantic segmentation results through external text attribute prompt words; the text attribute extractor adopts the GPT2 model to perform phrase-level attribute extraction on the text description according to the attribute tokenization instruction; the image attribute, the text attribute, the sampled image and the text description are input into the cross-modal feature encoding step for feature extraction; Then, in the cross-modal feature encoding step, the sampled image and the text description are encoded by using the contrastive language-image pre-training model to obtain image global features and text global features, wherein the image global features are extracted using the visual transformer of the contrastive language-image pre-training model, and the input is the sampled image; the text global features are extracted using the text transformer of the contrastive language-image pre-training model, and the input is the text description; the image global features and the text global features are fused with the corresponding attribute features through feature fusion, and the feature fusion formula is M att is a visual attribute mask, F v is an image global feature, is a visual attribute feature, and then, the text attribute feature is obtained through the text added with the attribute field Finally, in the feature fusion processing step, based on the similarity classification result of the visual attribute feature and the text attribute feature, the model parameters are optimized by a hierarchical alignment loss function, wherein the loss function includes attribute-level intra-modal contrast loss, inter-modal contrast loss, three tuple loss, and global-level instance loss and pedestrian identity loss.

2. The visual-linguistic information interaction gain pedestrian re-identification method of claim 1, wherein, In the data processing step, the saliency image is generated by the saliency network to obtain a non-uniform sampling result, and the saliency network comprises a 7x7 convolution layer, a batch normalization layer, a ReLU activation function, a 3x3 max pooling layer, four residual connection blocks, a 1x1 convolution layer and a Sigmoid activation function connected in turn, wherein the 1x1 convolution layer compresses the number of feature map channels to 1 channel and is up-sampled to the input image size by bilinear interpolation.

3. The visual-linguistic information interaction gain based pedestrian re-identification method of claim 2, wherein, The non-uniform sampling result is realized by a saliency grid generation module, and the module comprises: Performing boundary padding processing on the saliency image; Constructing a basic grid with a dimension of [batch_size, height, width, 2], wherein the last dimension is the normalized coordinates (x, y); multiplying the saliency image and the basic grid element by element to generate a weighted grid, and performing sampling after Gaussian filtering.

4. The visual-linguistic information interaction gain pedestrian re-identification method of claim 1, wherein, A classification strategy for sample division according to cosine similarity is specified, and the similarity classification result of the visual attribute feature and the text attribute feature is obtained by cosine similarity Si(·), and the formula is: Wherein, V attr_i is the i-th visual attribute feature, T attr_j is the j-th text attribute feature, and the attribute sample pair is divided into 12 similarity cases by a pre-set positive and negative sample threshold θ pos and θ neg .

5. The visual-linguistic information interaction gain based pedestrian re-identification method of claim 4, wherein, According to the classification strategy, a hierarchical alignment loss function L is defined total is represented as: where L ins is the instance loss, L id is the pedestrian identity loss, L attr is the attribute-level total loss, λ ins is the instance loss weight, N attr is the number of valid attributes.

6. The visual-linguistic information interaction gain pedestrian re-identification method of claim 5, wherein, The attribute-level total loss L attr weighted by the following losses Composition: where, and are the dynamic weights of inter-modal contrast loss and intra-modal contrast loss respectively, is the attribute-level triple loss, λ tri is the fixed weight of triple loss, and q is the attribute unit in the attribute set A.

7. The visual-linguistic information interaction gain based pedestrian re-identification method of claim 5, wherein, The example loss L ins The calculation is: where S I2T is an image-text similarity matrix, is its transpose matrix, y is the identity label vector, and CE(·) is the cross-entropy loss function.

Images