An asymmetric cost-aware based text-to-image pedestrian re-identification method

Abstract

This invention discloses a text-to-image person re-identification method based on asymmetric cost awareness. First, a dataset containing pedestrian images, text descriptions, and identity labels is acquired and preprocessed, then divided into training, validation, and test sets. A high-risk attribute label set is constructed based on the identity labels, adding risk attribute labels to each sample. A model is constructed containing a two-stream feature extraction network and a three-branch re-identification network. The two-stream network extracts image and text features; the three-branch network calculates the global opposite label triplet loss through an opposite label triplet loss branch, generates interactive features and calculates the cost-weighted identity loss through a feature interaction branch, and selects fine-grained local features and calculates the local opposite label triplet loss through a TopK feature selection branch. Multiple losses are fused to construct a total loss function. The model is trained and validated using the training and validation sets. Test text is input into the trained model, and the output is the matched person re-identification result.

Description

Technical Field

[0001] This invention relates to a text-to-image pedestrian re-identification method based on asymmetric cost perception, belonging to the field of pedestrian re-identification technology. Background Technology

[0002] The core objective of Text-to-Image Person Re-identification (TIPR) is to retrieve target pedestrians from large-scale image databases using natural language descriptions provided by witnesses as query conditions. This has practical value in high-risk applications such as intelligent security and criminal investigation. However, most existing TIPR methods focus on the global or local alignment of multimodal features, with their core optimization objective being to maximize overall retrieval matching accuracy. These methods generally follow the traditional principle of minimizing empirical risk, implicitly assuming that the misclassification costs between different identity categories and attributes are absolutely equivalent when constructing the cross-modal attitude space.

[0003] The TIPR method, based on the assumption of cost equivalence, has significant limitations in practical applications: it severely neglects the fact that different types of misclassification can lead to drastically different consequences in real-world high-risk monitoring environments. Existing technologies lack modeling and constraint mechanisms for this asymmetric cost, making it difficult to meet the needs of practical applications in high-risk environments. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a text-to-image pedestrian re-identification method based on asymmetric cost perception, which enhances the model's ability to perceive high-cost misjudgments in cross-modal contexts through an asymmetric cost perception loss function and feature interaction module.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A text-to-image person re-identification method based on asymmetric cost awareness includes the following steps: Step 1: Obtain the text-to-image pedestrian re-identification dataset. Each sample in the dataset includes a pedestrian image and a text description pair, as well as the corresponding identity label. After preprocessing the dataset, it is divided into a training set, a validation set, and a test set according to a preset ratio. Step 2: Based on the identity labels in the entire dataset, construct a high-risk attribute label set and add a risk attribute label to each sample in the dataset; Step 3: Construct a text-to-image person re-identification model, including a two-stream feature extraction network and a three-branch person re-identification network. The two-stream feature extraction network is used to extract image features and text features from the input samples. The three-branch person re-identification network includes an adversarial label triplet loss branch, a feature interaction branch, and a TopK feature selection branch. The adversarial label triplet loss branch is used to calculate the global adversarial label triplet loss based on global image features in image features and global text features in text features. The feature interaction branch is used to obtain interactive image features and interactive text features through the feature interaction module, and further calculate the cost-weighted identity loss. The TopK feature selection branch is used to obtain fine-grained local features through feature selection and calculate the local adversarial label triplet loss. Step 4: Construct the overall loss function based on the global and local opposition label triplet loss and the cost-weighted identity loss, and train and validate the text-to-image person re-identification model based on the training set and validation set to obtain the trained text-to-image person re-identification model. Step 5: Input the text descriptions from the test set into the trained text-to-image pedestrian re-identification model to obtain the pedestrian image re-identification results corresponding to the text descriptions.

[0006] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: 1. Unlike previous methods that only focused on improving retrieval accuracy, this invention is the first to recognize the industry pain point that different types of misclassification can lead to significantly different security consequences in practical applications. It is the first to view text-to-image tasks from a cost-aware perspective and designs cost-aware loss functions such as the adjoint label triplet loss and cost-weighted identity loss.

[0007] 2. This invention breaks through the limitation of traditional methods that assume equal costs for misjudgments, accurately penalizing and suppressing high-cost errors. It effectively overcomes the technical barrier of easily missing key targets in high-risk security scenarios, providing effective security guarantees for practical applications. Attached Figure Description

[0008] Figure 1 This is a flowchart of the text-to-image pedestrian re-identification method based on asymmetric cost perception according to the present invention; Figure 2 This is a network architecture diagram of the text-to-image person re-identification method based on asymmetric cost awareness, as presented in this invention. Figure 3 This is a schematic diagram of the opposing label triplet loss proposed in this invention; Figure 4 This is a schematic diagram of the feature interaction module proposed in this invention; Figure 5This is a schematic diagram of the cost-weighted identity loss proposed in this invention; Figure 6 This is a schematic diagram of the t-SNE model feature distribution of the present invention. Detailed Implementation

[0009] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0010] like Figure 1 As shown, this invention proposes a text-to-image person re-identification method based on asymmetric cost perception, the specific steps of which are as follows: (1) Divide the dataset into training set, validation set and test set, and perform data preprocessing on the input images and text; The system reads pedestrian images, text descriptions, and identity labels from existing text-to-image pedestrian re-identification datasets CUHK-PEDES, ICFG-PEDES, and RSTPReid, and divides them into training, validation, and test sets according to a preset ratio. Data preprocessing includes cropping pedestrian images to a preset size of 384×128 pixels, performing random horizontal flipping, random cropping, and random erasing to enhance training images, and simultaneously performing random masking, replacement, and removal on text descriptions to augment the data.

[0011] (2) Process the identity labels of the dataset to obtain the high-risk attribute label set R, and add the corresponding risk attribute label to each training sample; In a complete dataset containing training, validation, and test sets, to simulate high-risk real-world environments, identity labels across the entire dataset are uniformly and randomly sampled at a ratio of 70%. These sampled identity labels are designated as anomalous risk labels, and a high-risk attribute label set R is constructed. Based on the high-risk identity label set R, corresponding risk attribute labels are added to the samples in the entire dataset. (Samples with the identity label R are considered high-risk abnormal samples.) Samples whose identity labels do not belong to R are considered normal samples. ).

[0012] (3) The pre-trained CLIP model is used as the dual-stream feature extraction network. The pre-processed text and image data are input into the dual-stream feature extraction network to extract the corresponding image features. and text features ; The pre-trained CLIP model was selected as the dual-stream feature extraction network, and fine-tuned on the text-to-image person re-identification datasets CUHK-PEDES, ICFG-PEDES, and RSTPReid. The input training set data from step (1) was used, along with the image features extracted by CLIP. and text features The three-branch person re-identification network comprises three branches: opposing label triplet loss, feature interaction, and TopK feature selection. The two-stream feature extraction network and the three-branch person re-identification network together form a text-to-image person re-identification model, such as... Figure 2 As shown, SOS and ESO represent the prefix and suffix of a word, respectively.

[0013] (4) Calculate the global opposite label triplet loss OLT using the global features of the image text and the risk attribute labels. In the global feature space, differentially increase the feature distance between the query sample and the negative sample pairs of different risk attribute labels. like Figure 3 As shown, based on the risk attribute labels of the input samples The negative sample set with different identity labels from the input sample is decoupled into a key negative sample subset with opposing risk attribute labels and a regular negative sample subset with the same risk attribute labels. The global features of the image text features extracted in step (3) are then used to... The input is fed into the opposite-label triplet loss for global opposite-label triplet loss calculation, as shown in the following formula: , in, The batch size of the input samples. , or , For safety boundary values, The asymmetric penalty weights are for the key negative text description subset and the key negative pedestrian image subset. The asymmetric penalty weights are for the subsets of ordinary negative text descriptions and ordinary negative pedestrian images. and Set them to 0.9 and 0.1 respectively. This represents the weighted average similarity of positive samples from global text features to global image features. and , respectively, represent the maximum smooth similarity penalty terms calculated for the key negative pedestrian image subset and the ordinary negative pedestrian image subset; The weighted average similarity of positive samples from global image features to global text features is calculated using the following formula: , in, Representing global image features With global text features Similarity between them It can be calculated in the following ways: , This indicates the first number extracted from a batch. Global text features among text features All are binary matching tags. This refers to temperature hyperparameters.

[0014] and These represent the maximum smooth similarity penalty terms calculated for the key negative text description subset and the ordinary negative text description subset, respectively, and can be calculated in the following way: , , in, For the first A subset of key negative text descriptions for each image feature For the first A subset of ordinary negative text descriptions of image features. for The Middle Global text features described by a text for The Middle Global text features described by a text.

[0015] (5) such as Figure 4 As shown, cross-modal interaction features are obtained through the feature interaction module FFM. The cost-weighted identity loss (CSI) is calculated using the cross-modal interaction features, risk attribute labels, and a pre-defined asymmetric cost weight matrix. Differentiated penalty constraints are applied to different types of misclassification. The image features extracted in step (3) and text features First, information is aggregated within each modality to obtain features rich in contextual information of each modality. The specific calculation method is as follows: , , in, Presentation layer normalization operation, This represents a multi-head self-attention operation. Then, cross-modal deep aggregation is performed to obtain the interactive image features. and interactive text features : , , in, This represents a multi-head cross-attention operation. Subsequent feature aggregation yields cross-modal interaction features. This process can be represented as: , in, This indicates L2 norm normalization. is the learnable linear mapping parameter matrix.

[0016] Furthermore, cross-modal interaction features A pre-defined 2×2 asymmetric cost weight matrix Together with the risk attribute labels and identity labels of the input samples, they are used to calculate the cost-weighted identity loss. ,like Figure 5 As shown, the calculation process is as follows: , in, The batch size of the input samples. For real identity tags, For the standard cross-entropy function, and These are the risk attribute labels for the actual values ​​and the risk attribute labels for the model predicted values, respectively. True risk attribute labels for samples Risk attribute labels predicted by the model The corresponding asymmetric cost weight matrix value, It is a pre-defined 2×2 asymmetric cost weight matrix ( : : : = 20:2:1:1).

[0017] Figure 5 This diagram illustrates the cost-weighted identity loss. Different colors represent different risk attribute labels: green represents normal labels, and orange-red represents abnormal labels. This loss uses a pre-defined cost matrix to apply asymmetric penalties to different types of misclassification errors, aiming to preferentially increase the distance between features with different risk attribute labels within the feature space, constructing robust feature boundaries. Based on this, the distribution of samples within the same risk attribute label is further aligned.

[0018] (6) Select local salient features through TopK feature selection, calculate the local opposite label triplet loss OLT, and differentially increase the spacing between negative sample features of different risk attributes in the fine-grained local feature space. Use the image features extracted in step (3) and text features To perform TopK feature selection, taking image features as an example, the specific steps are as follows: For image features... The sequence of local salient image features with contextual information is selected based on the feature scores between the local and global image features of the last attention map in the image encoder. The same operation can yield locally salient text feature sequences. Then use The key negative text description subsets and ordinary negative text description subsets corresponding to the salient image features in each local area, and For each local salient text feature, the key negative pedestrian image subset and the ordinary negative pedestrian image subset are used to perform fine-grained opposition label triplet loss calculation. The specific calculation process is as follows: , Finally, the overall training loss function for: .

[0019] Every few training epochs, the network weights are frozen, and the model's cost metrics and accuracy are validated using a validation set. The model weights with the best results are saved for testing on the validation set. During testing, the optimal model weights are loaded, features of the query text and image samples in the test set are extracted, and similarity scores between the text and images are calculated using cosine similarity. The results are then sorted according to similarity and displayed.

[0020] Tables 1-3 compare the performance of the method of this invention with other text image person re-identification methods on three mainstream text image person re-identification datasets: CUHK-PEDES, ICFG-PEDES, and RSTPReid.

[0021] Table 1 Performance Comparison on CHUK-PEDES

[0022] Table 2 Performance Comparison on ICFG-PEDES

[0023] Table 3 Performance Comparison on RSTPReid

[0024] As can be seen from Tables 1-3, this invention achieves the best results in terms of the most important classification cost and the most serious false negative rate. Compared to classic methods like RDE, it significantly outperforms them, and even when compared to Bi-IRRA methods which use stronger feature encoders, it still outperforms Cost despite having a lower accuracy. The indicators demonstrate that the present invention can effectively improve the anomaly detection capability of text-to-image pedestrian re-identification systems in high-risk environments.

[0025] Figure 6 This diagram illustrates the visualization comparison results of the t-SNE model feature distribution. Different colored feature points represent different identity labels. Compared to traditional methods that only aim to align identity labels, the cost-aware method of this invention constructs robust feature boundaries. While achieving clustering with identity labels as cluster centers, it forcibly widens the feature distance between normal risk attribute label regions and high-risk attribute label regions, effectively preventing high-risk features from crossing boundaries and significantly reducing serious misjudgments caused by mistaking abnormal samples for normal samples.

[0026] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned text-to-image pedestrian re-identification method based on asymmetric cost perception.

[0027] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the aforementioned text-to-image pedestrian re-identification method based on asymmetric cost perception.

[0028] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0029] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0030] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0031] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0032] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A text-to-image person re-identification method based on asymmetric cost perception, characterized in that, Includes the following steps: Step 1: Obtain the text-to-image pedestrian re-identification dataset. Each sample in the dataset includes a pedestrian image and a text description pair, as well as the corresponding identity label. After preprocessing the dataset, it is divided into training set, validation set and test set according to a preset ratio; Step 2: Based on the identity labels in the entire dataset, construct a high-risk attribute label set and add a risk attribute label to each sample in the dataset; Step 3: Construct a text-to-image person re-identification model, including a two-stream feature extraction network and a three-branch person re-identification network. The two-stream feature extraction network is used to extract image features and text features from the input samples. The three-branch person re-identification network includes an adversarial label triplet loss branch, a feature interaction branch, and a TopK feature selection branch. The adversarial label triplet loss branch is used to calculate the global adversarial label triplet loss based on global image features in image features and global text features in text features. The feature interaction branch is used to obtain interactive image features and interactive text features through the feature interaction module, and further calculate the cost-weighted identity loss. The TopK feature selection branch is used to obtain fine-grained local features through feature selection and calculate the local adversarial label triplet loss. Step 4: Construct the overall loss function based on the global and local opposition label triplet loss and the cost-weighted identity loss, and train and validate the text-to-image person re-identification model based on the training set and validation set to obtain the trained text-to-image person re-identification model. Step 5: Input the text descriptions from the test set into the trained text-to-image pedestrian re-identification model to obtain the pedestrian image re-identification results corresponding to the text descriptions.

2. The text-to-image person re-identification method based on asymmetric cost perception according to claim 1, characterized in that, In step 1, the preprocessing of the dataset includes: cropping the pedestrian images to a preset size, and performing random horizontal flipping, random cropping, and random erasing operations on the cropped pedestrian images, while performing random masking, replacement, and removal operations on the text descriptions.

3. The text-to-image person re-identification method based on asymmetric cost perception according to claim 1, characterized in that, In step 2, identity labels from the entire dataset are uniformly and randomly sampled at a rate of 70%. The selected identity labels are designated as anomalous risk labels. A high-risk attribute label set R is constructed based on these anomalous risk labels. A corresponding risk attribute label is added to each sample in the entire dataset based on R. Samples whose identity labels belong to R are considered anomalous high-risk samples, and the added risk attribute label is... ; Samples whose identity labels do not belong to R are considered normal samples. The added risk attribute label is... .

4. The text-to-image person re-identification method based on asymmetric cost perception according to claim 1, characterized in that, In step 3, the dual-stream feature extraction network is a pre-trained CLIP model, which includes an image encoder and a text encoder. The image encoder is used to extract global and local features of the image from the input sample, and the text encoder is used to extract global and local features of the text from the input sample.

5. The text-to-image person re-identification method based on asymmetric cost perception according to claim 4, characterized in that, In step 3, the specific process of the contrastive label triplet loss branch is as follows: For any pedestrian image in the input sample Based on the input sample and the pedestrian image A negative text description set is constructed based on text descriptions of different identity labels, according to pedestrian images. Risk attribute label of the sample Combine negative text descriptions with risk attribute labels Opposing text descriptions are divided into a subset of key negative text descriptions, and risk attribute labels within the negative text description set are then compared with... The same text descriptions are grouped into a subset of ordinary negative text descriptions; For any text description in the input sample Based on the input sample and the text description A negative pedestrian image set is constructed from pedestrian images with different identity labels, based on text descriptions. Risk attribute label of the sample Combine negative pedestrian images with risk attribute labels Opposing pedestrian images are segmented into a key negative pedestrian image subset, and the risk attribute labels in the negative pedestrian image subset are then compared with... Identical pedestrian images are grouped into a subset of ordinary negative pedestrian images; In a batch, the first feature extracted by the dual-stream feature extraction network is... Global image features among image features and the Global text features among text features The input is fed into the contrast label triplet loss branch for global contrast label triplet loss. The calculation formula is as follows: ， in, The batch size of the input samples. , or , For safety boundary values, The asymmetric penalty weights are for the key negative text description subset and the key negative pedestrian image subset. The asymmetric penalty weights are for the subsets of ordinary negative text descriptions and ordinary negative pedestrian images. This represents the weighted average similarity of positive samples from global text features to global image features. and , respectively, represent the maximum smooth similarity penalty terms calculated for the key negative pedestrian image subset and the ordinary negative pedestrian image subset; The weighted average similarity of positive samples from global image features to global text features is calculated using the following formula: ， ， in, The weighted coefficient for positive sample similarity. Representing global image features With global text features Similarity between them These represent the first and second digits extracted from a batch. Global text features among text features All are binary matching tags. This refers to temperature hyperparameters. and These represent the maximum smooth similarity penalty terms calculated for the key negative text description subset and the ordinary negative text description subset, respectively. The calculation formula is as follows: ， ， in, For the first A subset of key negative text descriptions for each image feature For the first A subset of ordinary negative text descriptions of image features. for The Middle Global text features described by a text for The Middle Global text features described by a text.

6. The text-to-image person re-identification method based on asymmetric cost perception according to claim 1, characterized in that, In step 3, the specific process of the feature interaction branch is as follows: Image features extracted by a dual-stream feature extraction network and text features In the feature interaction module, intramodal information aggregation is first performed to obtain image features containing contextual information of each modality. and text features The calculation formula is as follows: ， in, Presentation layer normalization operation, This indicates a bullish self-attention strategy; Then, image features based on context information are used. and text features Cross-modal deep aggregation is performed to obtain interactive image features. and interactive text features The calculation formula is as follows: ， in, This indicates a multi-head cross-attention operation; subsequently, for and Feature aggregation is performed to obtain cross-modal interaction features. , represented as: ， in, This indicates L2 norm normalization. The parameter matrix of a learnable linear mapping; Cross-modal interaction features Preset asymmetric cost weight matrix Together with the risk attribute labels and identity labels of the input samples, they are used to calculate the cost-weighted identity loss. The calculation formula is as follows: ， in, The batch size of the input samples. For real identity tags, For the standard cross-entropy function, and These are the risk attribute labels for the actual values ​​and the risk attribute labels for the model predicted values, respectively. True risk attribute labels for samples Risk attribute labels predicted by the model The corresponding asymmetric cost weight matrix value.

7. The text-to-image person re-identification method based on asymmetric cost perception according to claim 5, characterized in that, In step 3, the specific process of the TopK feature selection branch is as follows: Image features extracted based on a two-stream feature extraction network and text features TopK feature selection is performed specifically as follows: for the extracted image features Global image features And local image features, combining the local image features output from the last layer attention map of the image encoder with... The feature scores are sorted in descending order, and the top K local image features are selected as the sequence of local salient image features with contextual information. K is a preset value; the same operation yields a sequence of locally salient text features with contextual information. ; use The key negative text description subsets and ordinary negative text description subsets corresponding to the salient image features in each local area, and For each local salient text feature, a subset of key negative pedestrian images and a subset of ordinary negative pedestrian images are used to perform a local contrastive label triplet loss. The calculation is as follows: ， in, This represents the weighted average similarity of positive samples from local image features to local text features. This represents the weighted average similarity of positive samples from local text features to local image features.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the text-to-image pedestrian re-identification method based on asymmetric cost awareness as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the text-to-image pedestrian re-identification method based on asymmetric cost awareness as described in any one of claims 1 to 7.

Images