Vision-language information interaction gain pedestrian re-identification method and system

Abstract

The invention discloses a pedestrian re-identification method and system based on vision-language information interaction gain, and relates to the technical field of cross-modal pedestrian identification. The invention provides a data processing optimization and deep interactive fusion scheme which mainly comprises four processing stages: in the data processing stage, a thermodynamic diagram is generated through a saliency network, and non-uniform sampling is realized by combining a grid module to amplify key region features; in the attribute extraction stage, a Ground SAM model is adopted to generate a visual attribute mask, and meanwhile, a GPT2 model is utilized to analyze text attributes; in the feature fusion stage, the visual attribute masks and original image features are subjected to weighted fusion, and cross-modal coding is realized through a CLIP model. In the loss optimization stage, a hierarchical alignment loss function is designed, sample pairs are subdivided according to similarity, and difficult sample alignment is optimized. According to the method, key feature expression, bimodal attribute deep interaction and hierarchical loss optimization are enhanced through saliency sampling, so that the pedestrian re-recognition accuracy in a complex scene is remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and system for pedestrian re-identification based on visual-language information interaction gain. Background Technology

[0002] Text-based image person re-identification has proven to have significant application value in the fields of security and surveillance, providing a novel method for identifying pedestrians through text descriptions. In practical applications, text-based image person re-identification can be used for tasks such as suspect identification and locating missing children.

[0003] However, the joint retrieval of textual and visual information not only introduces instability but also presents traditional cross-modal challenges. Therefore, designing new text-image person re-identification technologies is of great significance.

[0004] Since its emergence, the Transformer technology has been widely applied across various fields. Due to the excellent performance of VisionTransformer (ViT) and Transformer-based Bidirectional Encoder Representation (BERT), most retrieval models use them as backbone networks. Furthermore, researchers continue to explore the potential of Visual Language Pre-trained (VLP) models.

[0005] In reality, data processing is an easily overlooked aspect. In the field of pedestrian re-identification, input images are generally required to be normalized to a certain fixed size, such as... or If the input image size is larger than the specified size, existing methods often use uniform downsampling to reduce the image size to the specified size. However, this may be lossy for pedestrian re-identification tasks that require information from different spatial resolutions and locations.

[0006] Meanwhile, most existing methods simply concatenate text and image features or employ shallow fusion strategies, failing to fully explore the inherent semantic relationships between the two modalities. Due to the lack of deep interaction mechanisms, the model cannot deeply guide the network to extract key image features based on text information, nor can it utilize image feature feedback to optimize the understanding of text semantics. This results in poor robustness and adaptability when facing complex situations such as occlusion and changes in viewpoint.

[0007] This invention designs a novel modality fusion loss optimization strategy and applies an innovative saliency modality interaction enhancement network structure to solve the above problems. Summary of the Invention

[0008] This invention, based on the optimization of the data sampling process of the dataset and the need to enhance the interactivity of text and images, designs a pedestrian re-identification method and system with enhanced visual-linguistic information interaction. In the data processing stage, this method and system can effectively amplify the key features of pedestrian images in the dataset. In the subsequent feature extraction and post-processing, attribute information from the visual and text inputs is extracted through a visual converter and a text converter, and positive and negative samples are further subdivided to optimize the loss function. The technical framework diagram of this invention is shown below. Figure 1 As shown.

[0009] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solution: In the data processing stage, embodiments of the present invention provide a key region magnification sampling method, which aims to solve the information loss problem caused by uniform downsampling of input images. Specifically, this is achieved through the following technical solution: In the first sampling stage, a convolutional neural network is used to generate a saliency image. This saliency image is specific to the pedestrian re-identification task because different tasks may require attention to different image target regions. In the second sampling stage, the most important image regions are sampled based on the saliency image. The sampled images are normalized to a uniform target size, and the sampled images and their corresponding text descriptions are fed back into a text-image cross-modal pedestrian re-identification network based on a contrastive language-image pre-trained model for feature extraction.

[0010] In the data processing stage, the saliency image Extracted from a saliency network, such as... Figure 2 As shown. The key region magnification sampling method will input the image. With saliency image Simultaneously input sampler The sampler The purpose is to calculate the input image With saliency image Calculated sampled image The mapping relationship can be expressed as: .

[0011] The sampler As a grid segmenter, based on the input image The sampler will generate a size of the original size. The grid matrix, The pixel location represents the saliency heatmap. The saliency image... The input image is included The pixel saliency is calculated by multiplying element-wise to generate a weighted grid, which is then processed using a Gaussian filter. The weights represent the offset of each location in the sampled image, and regions with higher saliency are assigned a larger sampling range. This method can achieve smooth saliency non-uniform sampling.

[0012] After the above-mentioned non-uniform saliency sampling operation, the input image Resampled to enlarge the important regions The reconstructed dataset is fed back into a network for text-image person re-identification to perform the person re-identification task.

[0013] The text-image person re-identification network consists of an attribute extractor, an image encoder, a text encoder, and a classification comparison optimization module. To enhance the interactivity between the input text and the input image, this invention constructs a fine-grained cross-modal relationship correspondence pattern from the perspective of the attribute correspondence between text and image.

[0014] This invention addresses the challenges of cross-modal alignment and poor interactivity in text-image person re-identification through a dual-interaction method combining visual and textual attributes. Visual attributes are obtained by semantically segmenting the image input using the image attribute extractor of this invention, while textual attribute information is generated by word segmentation and prompting using the text attribute extractor of this invention.

[0015] Specifically, such as Figure 1 As shown, the visual attributes are obtained using the semantic segmentation pre-trained model Grounded SAM, which unifies the functions of Grounding DINO and SAM. Grounding DINO processes the input textual prompts and uses pre-trained weights trained on a large-scale dataset to obtain the target detection boxes and their confidence scores indicated by the textual attribute prompts. SAM, as a large-scale semantic segmentation pre-trained model capable of "segmenting everything," excels at combining input target detection boxes for high-precision semantic segmentation. Through the joint construction of Grounding DINO and SAM, this invention can obtain highly confident semantic segmentation results using textual attribute prompts.

[0016] Text attribute segmentation was performed using the large-scale language model GPT2. As a large-scale language model, GPT2 can handle text-related operations well given prompts and commands. Furthermore, by utilizing a large-scale language model, attribute descriptions can retain phrase structure, ensuring that attribute words are not isolated noun combinations but meaningful descriptive phrases, thus reducing redundant word interference.

[0017] In the subsequent feature extraction post-processing, this invention employs a contrastive language-image pre-trained model as both an image encoder and a text encoder to extract features from the input image and text. The contrastive language-image pre-trained model is a model that, after pre-training on a large-scale image-text dataset, can be fine-tuned for other specific tasks. This invention uses CLIP as the feature extractor, where image feature extraction utilizes a visual converter with pre-trained weights within CLIP, and text feature extraction utilizes a text converter with pre-trained weights within CLIP.

[0018] This invention describes the classification and processing of text and visual attribute features. Specifically, as shown in the embodiments of the invention... Figure 3 As shown, based on the similarity metric, we categorize text-visual attribute feature matching pairs into 12 cases, resulting in 6 class pairs. The similarity metric used is cosine similarity. The formula is:

[0019]

[0020] in, For the first visual Similarly, the feature representation of each attribute... For the first part of the text Textual representation of each attribute. Passed through a positive sample similarity threshold. Similarity threshold with negative samples The text visual attribute sample pairs are divided into 12 cases. The visual attribute segmentation and text attribute word segmentation divide the samples of the two modalities into corresponding attribute features and assign labels. The attribute features are then measured one by one for similarity. Similarity values ​​higher than a threshold are considered similar feature pairs, otherwise they are considered dissimilar feature pairs.

[0021] Based on this, the image With images ,text With text When attribute features belong to the same pedestrian ID, they can be divided into 6 types based on attribute similarity: (1) and Similarity less than (2) and Similarity greater than (3) and Similarity less than (4) and Similarity greater than (5) and Similarity less than (6) and Similarity greater than When the attribute features belong to different pedestrian IDs, they can also be divided into the above 6 cases, namely: (7) and Similarity less than (8) and Similarity greater than (9) and Similarity less than (10) and Similarity greater than (11) and Similarity less than (12) and Similarity greater than .

[0022] The similarity judgment can be based on the input attributes. Image text pairs There are 6 categories for subsequent classification and comparison optimization modules. Specifically, cases (1) and (5) are false negative samples within a modality, case (3) is false negative samples between modalities, cases (8) and (12) are false positive samples within a modality, case (10) is false positive samples between modalities, cases (2), (4) and (6) are positive samples, and cases (7), (9) and (11) are negative samples.

[0023] Based on the aforementioned classification method, this invention designs a hierarchical alignment loss function to optimize the network's ability to distinguish different sample types. For cases where false positive and false negative samples within and between modalities are considered hard samples, this invention increases the penalty for these hard samples to ensure that features with the same ID and attribute are close in cross-modal space, while features with different IDs and attributes are far apart.

[0024] The core idea of ​​the loss function is to combine instance-level similarity loss, attribute-level triplet loss and contrast loss, and constrain the model from the perspectives of global feature alignment and local attribute alignment, respectively. The loss weight is increased for difficult samples, prompting the model to prioritize learning the feature differences of samples that are difficult to identify and easily confused. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the prior art and embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a framework diagram of a pedestrian re-identification method and system with visual-linguistic information interaction gain.

[0028] Figure 2 The present invention employs a saliency network diagram.

[0029] Figure 3 This is a hierarchical alignment structure diagram that combines global alignment and attribute alignment.

[0030] Figure 4 A visualization of attribute segmentation generated by a segmentation network and a large-scale language model.

[0031] Figure 5 This is a partial search result of the present invention on the CUHK-PEDES dataset. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] First see Figure 1 , Figure 1 This is a flowchart illustrating a saliency sampling method provided in an embodiment of the present invention. The embodiment of the present invention uses a saliency network to extract a saliency image. And provide a sampler Image before sampling Significance heatmap Compared with the sampled image Mapping relationship: .

[0035] In the saliency network structure, such as Figure 2As shown, after the image is input, it first passes through a 7×7 convolutional layer. This convolutional layer uses a stride of 2 and padding of 3, and its core function is to initially extract image features. The data output from the first convolutional layer undergoes batch normalization, a non-linear transformation using the ReLU activation function, and then passes through a 3×3 max pooling layer. The max pooling layer can reduce the dimensionality of the feature map space while preserving the key feature information of the image.

[0036] After the above processing, the data will be fed into a structure consisting of four consecutive residual join blocks, the specific architecture of which can be found in [reference needed]. Figure 2 After the residual blocks complete feature processing, the saliency network further mines salient features: through a 1×1 convolutional layer, the number of channels in the feature map is compressed to 1 channel, thus generating a single-channel saliency image. To ensure the output size meets the preset standard, the network uses bilinear interpolation for upsampling, and finally uses the sigmoid activation function to normalize the data, ultimately obtaining the complete saliency heatmap output.

[0037] Significant Non-Uniform Sampler It consists of two core components: a saliency grid generation module and a sampling execution module. The grid generation module's workflow begins with filling the saliency heatmap, a step that ensures boundary continuity in subsequent grid generation processes. Next, the sampler constructs a basic grid structure with dimensions configured as follows: The value 2 in the last dimension corresponds to the normalized value. Coordinate values. A weighted grid structure is obtained by performing element-wise multiplication between the saliency heatmap and the base grid. Subsequently, a Gaussian filtering algorithm is used to smooth this weighted grid, thus achieving saliency non-uniform sampling with smoothing properties. Finally, this sampled image is fed into an image recognition network to complete the type identification task.

[0038] The dataset used for the text image person re-identification task is sequentially fed into the saliency non-uniform sampler. And attribute extractors. The attribute extractors are a text attribute extractor and an image attribute extractor. Specifically, the text attribute extractor is the language large-scale model GPT2, and the image attribute extractor is the semantic segmentation large-scale model Grounded SAM.

[0039] like Figure 3 As shown, the extended dataset after attribute extraction is input into a text-image person re-identification network for feature extraction and loss optimization. The extended dataset includes image target attribute masks obtained based on specified attribute cues. With text that has added attribute fields Feature extraction uses CLIP-ViT / B-16 as the backbone network, combining the image target attribute mask with the original image features. In proportion After fusion, the attribute feature representation of the fused image is obtained. , that is, .

[0040] Similarly, text with added attribute fields After passing through the text feature extractor, the text attribute feature representation is obtained. Subsequently, and In the attribute space as an attribute ( Image text feature pairs (for attribute sets) The samples are fed into the attribute-level alignment optimization process. After similarity measurement and the above classification strategy, false positive and false negative sample pairs within and between modes in this batch can be obtained. These hard sample pairs are constrained by the following formulas:

[0041] At the local attribute level, for attributes Based on image and text features, this invention employs intra-modal contrastive learning loss respectively. Intermodal contrastive learning loss and attribute-level triplet loss Combinatorial optimization. The formula is shown below:

[0042]

[0043] in, Intramodal properties of an image Contrastive learning loss, Modal attributes of text The contrastive learning loss is calculated as follows:

[0044]

[0045]

[0046] in, Represents attributes The number of valid samples. This represents the activation function of the Modified Linear Unit (RELU). Indicates the first Attributes in a single image Features Indicates the first Attributes in text The features are all processed by self-attention pooling and feature fusion. Representation and Sample A set of false negative samples with the same ID. Denotes the set of false positive samples. This represents the cosine similarity.

[0047] Similarly, inter-modal contrastive learning loss The formula is as follows, and the variable definitions follow the meanings described above.

[0048]

[0049] After implementing hard-sample contrastive learning, the remaining false positive and false negative attribute pairs are used in the perceptual attribute triple loss to enhance cross-modal alignment. For image samples... In the text sample (Pairs of samples with different identities but high similarity scores are used as false positive samples) and text samples (and When classifying images with the same identity but low similarity scores (used as false negative pairs), we apply triplet loss to enhance the image samples. With text samples The correlation between them, and at the same time the samples and Distinguish, It is a triplet margin. The loss function is defined as follows:

[0050]

[0051] For loss optimization at the global instance and identity levels, this invention example uses instance loss. Loss of pedestrian identity .

[0052]

[0053]

[0054]

[0055] in, It is a length of A vector of (batch size), and Indicates the first If the image-text pairs belong to the same identity, otherwise . This is the image-to-text similarity matrix. This is the text-to-image similarity matrix, which is The transpose of . This represents the temperature parameter, set to 0.07. and These represent the global features of the image and text, respectively. Indicates the first The image and the first Similarity between texts. Indicates the first The similarity score between the front and back of each sample.

[0056]

[0057] in, Indicates the number of samples. This indicates the first prediction made by the network. Each sample belongs to its true identity. The probability of.

[0058] After obtaining the losses at the local attribute level and the global instance identity level, the overall loss method adopted in this invention... The weighted sum of the above losses is composed of:

[0059]

[0060]

[0061]

[0062] and Instance loss respectively With attributes Triple contrast loss Fixed weight parameters, and Then it is an attribute Intermodal contrastive learning loss Intramodal contrast loss The weights are dynamically adjusted. Indicates the number of valid attributes. Represents a collection of attributes, where It is a small value used to prevent division by zero, set to , This represents the variance of the loss value.

[0063] To verify the effectiveness of the present invention in the field of person re-identification in text images, the technique was tested on three benchmark datasets with different characteristics (CUHK-PEDES, ICFG-PEDES, and RSTPReid), and compared with traditional methods and state-of-the-art methods. The results show that the present invention achieves optimal or near-optimal performance on these three benchmark datasets. Figure 4 This paper presents an example of segmentation results obtained by processing a dataset using the pre-trained semantic segmentation model Grounded SAM and the large-scale language model GPT2 employed in this invention. Figure 5 This paper presents some retrieval results of the present invention on the CUHK-PEDES dataset. Through an innovative design that integrates data processing optimization and deep interactive methods, the present invention effectively solves the technical bottlenecks of key feature loss and insufficient cross-modal semantic association in text image person re-identification tasks.

[0064] The method proposed in this invention has significant application prospects in fields such as intelligent security, suspect tracking, and missing persons retrieval. Field tests demonstrate its excellent performance, providing an efficient solution for cross-modal pedestrian recognition tasks. Future applications can be further expanded to scenarios such as multimodal fusion video pedestrian re-identification and cross-camera collaborative tracking. Furthermore, it is expected to enhance feature completion capabilities under extreme conditions by introducing generative AI models, providing strong support for the technological upgrade of intelligent visual surveillance systems.

[0065] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0066] It should also be noted that, in this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method and system for pedestrian re-identification based on visual-linguistic information interaction gain, characterized in that, Includes the following steps: First, in the data processing step, a pedestrian saliency image is generated through a saliency network. Based on the saliency image, the image is non-uniformly sampled to obtain a sampled image with magnified key regions. Then, in the attribute extraction step, the image attribute extractor and the text attribute extractor are used to extract the visual attribute features of the sampled image and the text attribute features of the corresponding text description, respectively. Next, in the cross-modal feature encoding step, a contrastive language-image pre-trained model is used to encode the features of the sampled image and text description to obtain global image features and global text features; Finally, in the feature fusion processing step, based on the similarity classification results of visual attribute features and text attribute features, the model parameters are optimized through a hierarchical alignment loss function. The loss function includes attribute-level intra-modal contrast loss, inter-modal contrast loss, triplet loss, and global-level instance loss and pedestrian identity loss.

2. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 1, characterized in that, In the data processing step, the saliency image is generated by a saliency network that produces non-uniform sampling results. The saliency network comprises sequentially connected... Convolutional layers, batch normalization layers, ReLU activation function Max pooling layer, four residual connection blocks, Convolutional layers and the Sigmoid activation function, the The convolutional layer compresses the number of feature map channels to 1 channel and upsamples it to the input image size using bilinear interpolation.

3. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 2, characterized in that, The non-uniform sampling results are achieved through a saliency grid generation module, which includes: Perform boundary padding on the saliency image; Construction dimension The basic grid, where the last dimension is a normalized coordinate. The saliency image is multiplied element-wise with the base grid to generate a weighted grid, which is then sampled after Gaussian filtering.

4. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 1, characterized in that, In the attribute extraction step: The image attribute extractor uses the Grounded SAM model to generate high-confidence semantic segmentation results through external text attribute prompts; the text attribute extractor uses the GPT2 model to extract phrase-level attributes from the text description according to the attribute word segmentation instructions; and the feature extraction is performed through a cross-modal feature encoding step using image attributes, text attributes, original images, and original text inputs.

5. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 4, characterized in that, The cross-modal feature encoding step uses a contrastive language-image pre-trained model as the encoder, wherein: Image feature extraction uses a visual converter of a contrastive language-image pre-trained model, with the input being the sampled image; Text feature extraction uses a text converter from a contrastive language-image pre-trained model, with text descriptions as input; Image global features and text global features are fused with corresponding attribute features through self-attention pooling and feature fusion.

6. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 5, characterized in that, The fusion formula for the feature fusion processing is: ； in, For image target attribute mask, Original image features The combined visual features.

7. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 6, characterized in that, A classification strategy based on cosine similarity is specified, wherein the classification results of the image fusion features and text fusion features are classified using cosine similarity. The formula is as follows: ； in, For the first A visual attribute feature, For the first Each text attribute feature is evaluated using a preset positive and negative sample threshold. and The attribute sample pairs were divided into 12 similarity categories.

8. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 7, characterized in that, Based on the classification strategy, a hierarchical alignment loss function is defined. Represented as: ； in, For instance loss, For the loss of pedestrian identity, This represents the total attribute-level loss. For instance loss weights, This represents the number of valid attributes.

9. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 8, characterized in that, The total loss at the attribute level Weighted by the following losses composition: ； in, and Intermodal contrast loss Intramodal contrast loss Dynamic weights, For attribute-level triplet loss, The triplet loss is given a fixed weight. Attribute level The attribute unit in the text.

10. The pedestrian re-identification method and system with visual-linguistic information interaction gain as described in claim 8, characterized in that, The instance loss The calculation method is as follows: ； in, For image-text similarity matrix, Let it be its transpose matrix. For identity tag vectors, This is the cross-entropy loss function.

Images