An end-to-end pedestrian re-identification method based on index prediction
By employing an end-to-end index prediction method, we utilize the ViT model and Transformer architecture for joint optimization of feature extraction, enhancement, and ranking. We also design a multi-label cross-entropy loss function to address the issues of optimization inconsistency and error accumulation in person re-identification, thereby improving the model's performance and generalization ability.
Patent Information
- Application Number
- CN202510285742.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-11
- Publication Date
- 2025-12-05
- Estimated Expiration
- 2045-03-11
AI Technical Summary
Existing pedestrian re-identification methods suffer from problems such as inconsistent optimization, error accumulation, and limited generalization ability in the feature extraction, feature enhancement, and ranking stages. In particular, the traditional cross-entropy loss function is not applicable in the case of multiple matching images, leading to a decline in model performance.
An end-to-end approach based on index prediction is adopted, which utilizes the ViT model and Transformer architecture for joint optimization of feature extraction, enhancement and ranking, and designs a multi-label cross-entropy loss function. Index prediction is realized through an encoder-decoder architecture, and a dynamic codebook is constructed to adapt to the case of multiple matching images.
It significantly improves the performance and generalization ability of the pedestrian re-identification model, solves the joint optimization problem of feature extraction and ranking, enhances feature discrimination and model robustness, and adapts to the prediction of multiple matching images.
Smart Images
Figure CN120126181B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of computer vision, and particularly relates to an end-to-end pedestrian re-identification method based on index prediction. BACKGROUND
[0002] Person Re-Identification (ReID) is an important task in the field of computer vision, aiming to match images of the same person through a large-scale database. This technology has wide application prospects in public security, video surveillance, intelligent transportation and other fields. With the widespread deployment of surveillance cameras and the development of artificial intelligence technology, ReID has received increasing attention.
[0003] Traditional methods usually decompose the ReID task into multiple key sub-tasks, including feature extraction, feature enhancement and ranking. However, this method of decomposing the task into multiple independent sub-tasks has some problems. First, optimizing these sub-tasks independently may lead to inconsistent training and testing objectives. For example, the feature extraction stage may focus on learning discriminative features, but the ranking stage may be affected by the feature enhancement strategy, resulting in limited overall performance. Second, the feature enhancement and ranking stages usually rely on hand-designed strategies that may not adapt to complex scene changes such as occlusion, lighting changes or clothing changes, affecting the model's generalization and robustness capabilities. Third, due to the lack of joint optimization between sub-tasks, errors in the previous sub-task may be passed on to subsequent sub-tasks, leading to overall performance degradation.
[0004] Some research has also attempted to optimize the ReID model as a whole. For example, some research has used encoder-decoder architecture to handle the ReID task, but these methods mostly treat the ranking problem as an independent step without joint optimization with the feature extraction and enhancement steps. In addition, these methods often face the problem of non-unique correct prediction when dealing with multiple matching images in the database, which makes the traditional cross-entropy loss function no longer applicable.
[0005] In the field of pedestrian re-identification, although some research has attempted to handle the ReID task through an encoder-decoder architecture to achieve overall optimization, these methods mostly treat the ranking problem as an independent step without joint optimization with the feature extraction and enhancement steps. This separate processing approach leads to inconsistencies in optimization objectives, error accumulation, and limited generalization capabilities. In addition, existing methods often face the problem of non-unique correct prediction when dealing with multiple matching images in the database, which makes the traditional cross-entropy loss function no longer applicable, further limiting the optimization and performance improvement of the model. SUMMARY
[0006] In order to overcome the deficiencies of the prior art, the present application provides an end-to-end pedestrian re-identification method based on index prediction, a new ReID method-StreamReID model is constructed, the ReID task is redefined as an index prediction problem, and the encoder-decoder architecture is used to realize the joint optimization of feature extraction, feature enhancement and sorting steps, and a customized multi-label cross-entropy loss function is designed to adapt to the case of multiple matching images, thereby effectively solving the problems existing in the prior art, and significantly improving the performance and generalization ability of the ReID model.
[0007] The technical scheme adopted by the present application to solve its technical problems is as follows:
[0008] Step 1: hard sampling strategy;
[0009] The ViT model is used as a feature extractor;
[0010] The sample image is input into the ViT model to extract features;
[0011] The extracted features are resampled for training the subsequent encoder-decoder module;
[0012] Step 2: build an encoder to realize relationship information interaction;
[0013] Step 2-1: spatial average pooling is performed on the image features extracted from the feature extractor, and the image features are compressed into one-dimensional vectors; the pooled query features and database features are represented as F={f q ,f d0 ,f d1 ,...,f di ,....,f dn}, wherein f q represents the query feature, f di represents the database feature, and i is the index of the feature in the database; the database feature f di and the query feature wherein C is the channel dimension;
[0014] Step 2-2: input the pooled features into the encoder module, the encoder module is composed of 6 Transformer layers, each layer contains a self-attention layer followed by a feedforward layer; for the input feature F, the self-attention mechanism is represented as:
[0015] Projection: Q=FC Q (F), K=FC K (F), V=FC V (F)
[0016]
[0017] Aggregation:F a =S`V (1)
[0018] FC Q FC K FC V All represent fully connected layers, used to embed input features into query Q, key K, and value V vectors respectively; d represents the dimension of the feature vector; S is the calculated affinity measure, reflecting the relationship between the compared features; It is the output feature;
[0019] Then F a It is fed into the feedforward layer;
[0020] Step 2-3: Repeat step 2-2 six times to obtain the feature enhanced by the encoder, denoted as F. en ;
[0021] Step 3: Build a decoder to perform index prediction;
[0022] Step 3-1: Aggregate image features from the database Through a set of learnable positional embeddings L = {l0, l1, ..., l n ,l End The process of constructing a dynamic codebook can be formally represented as follows:
[0023]
[0024] Where φ={φ0,φ1,...,φ n ,φ End} represents the constructed dynamic codebook, n is the number of images in the database, and φ End It is an end marker;
[0025] Step 3-2: Use the query feature as the starting marker, denoted as... Then it is sent to the decoder module, which consists of two Transformer layers. This process is written as follows:
[0026] D 0 =Decoder(H 0 (3)
[0027] in It is the output of the decoder module;
[0028] Step 3-3: Based on S 0 The correlation between the last element in the codebook and each index feature in the dynamic codebook predicts the index position of the matching element. This process can be written as:
[0029] ψ 0 = softmax(η(S 0 · φ) (4)
[0030] where η is a function that selects the last element in the set S 0 ; the index of the maximum value in the vector ψ indicates the position of the current prediction matching image;
[0031] Step 3-4: assuming the predicted position is j, concatenate the jth element in the dynamic codebook with H 0 to construct the input for the next round After that, input H 1 to the decoder and iteratively apply equation (3) and equation (4) to generate the index positions of all matching images;
[0032] Step 4: loss function;
[0033] A multi-label cross-entropy loss function is used, written as:
[0034]
[0035] where ψ i is the ith element in ψ; pos represents the index of all matching images in the database;
[0036] By using the above loss function, the decoder model can be trained to predict the index of any matching image at each round.
[0037] Preferably, the step 1 is specifically:
[0038] Randomly select an image feature as a query and construct a database for it; specifically, use a memory bank to retain the features of the first 10,000 images, and select 100 images with the most similar but different identities from the query to construct the database; next, randomly insert the image features with the same identity as the query in the current small batch into different positions in the database; the constructed database together with the query feature is used as the input of the encoder-decoder model; finally, train the encoder-decoder model to predict the matching image position.
[0039] Preferably, when training the decoder model: to avoid the decoder repeatedly predicting, discard the index markers corresponding to the previous prediction from the codebook after each round; once all index markers corresponding to matching elements are discarded, the decoder is trained to predict the index of the end marker; in the training stage, in the decoder module, arrange all matching image index markers in the dynamic codebook in ascending order and input them into the decoder in parallel. Apply the mask attention mechanism to prevent forward markers from accessing subsequent markers, thereby achieving parallel training of the decoder.
[0040] A computer program, which makes a computer execute the above-mentioned end-to-end pedestrian re-identification method.
[0041] An electronic device, comprising: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the above-mentioned end-to-end pedestrian re-identification method.
[0042] A computer readable storage medium, which stores a computer program, the computer program is executed by a processor to realize the above-mentioned end-to-end pedestrian re-identification method.
[0043] A chip, comprising: a processor, used to call and run a computer program from a memory, so that a device installed with the chip executes the above-mentioned end-to-end pedestrian re-identification method.
[0044] A computer program product, comprising a computer storage medium, the computer storage medium stores a computer program, the computer program comprises instructions executable by at least one processor, when the instructions are executed by the at least one processor, the above-mentioned end-to-end pedestrian re-identification method is realized.
[0045] The beneficial effects of the present application are as follows:
[0046] The present application introduces a hard sampling strategy in input data, constructs more challenging training samples, and significantly improves the model generalization ability; in the model structure, the encoder-decoder architecture is adopted, the self-attention mechanism of the Transformer is used for feature enhancement, and the ranking problem is redefined as an index prediction task, realizing the joint optimization of feature extraction, enhancement and ranking; in the training process, a customized multi-label cross-entropy loss function is designed to adapt to the prediction problem of multiple matching images, and an end-to-end training method is used to avoid error accumulation and improve overall performance. In addition, the self-attention mechanism formula and the multi-label cross-entropy loss function formula are optimized, further enhancing the feature discrimination and the optimization ability of the model. BRIEF DESCRIPTION OF DRAWINGS
[0047] Figure 1 It is a StreamReID model schematic diagram of the present application. DETAILED DESCRIPTION
[0048] The present application will be further described below in combination with the drawings and examples.
[0049] The present application proposes a new method of integrating ReID tasks into an end-to-end framework that can unify the optimization of feature extraction, feature enhancement and ranking steps, aiming to improve the performance of ReID models through unified optimization objectives.
[0050] The present application proposes a brand-new ReID method-StreamReID model, which redefines the ReID task as an index prediction problem, and uses an encoder-decoder architecture to realize the joint optimization of feature extraction, feature enhancement and sorting steps, and at the same time designs a customized multi-label cross-entropy loss function to adapt to the case of multiple matching images, thereby effectively solving the problems existing in the prior art, and significantly improving the performance and generalization ability of the ReID model. The structure is as shown in Figure 1 The main work of the present application includes three components: feature extractor, Transformer encoder module and Transformer decoder module.
[0051] 1. Hard sampling strategy;
[0052] In each mini-batch, the present application extracts 128 images, containing 32 pedestrian instances, and 4 images for each instance. These images are then sent to the ViT model to extract basic features. Then, the extracted features are resampled to train the subsequent encoder-decoder module. In this step, a random image feature is selected as a query, and a challenging database is constructed for it. Specifically, an in-memory library is used to retain the features of the first 10,000 images, and 100 images with the most similar but different identities are selected from them to construct the database. Next, the image features with the same identity as the query in the current mini-batch are randomly inserted into different positions in the database. The constructed database and the query feature are used as the input of the encoder-decoder model. Finally, the encoder-decoder model is trained to predict the matching image position.
[0053] 2. Encoder: realize relationship information interaction;
[0054] In the StreamReID model, the self-attention mechanism of the encoder module is used to capture and exchange relationship information between the query image features and the database image features, thereby enhancing the discriminability of the features. Unlike some existing works that use handcrafted strategies (such as adjacency matrices) to model relationship information, the Transformer architecture used does not require such strategies, thereby enabling more effective optimization.
[0055] In fact, first, the image features extracted from the feature extractor (ViT) are spatially averaged and pooled into one-dimensional vectors. The pooled query feature and database feature are represented as F={f q ,f d0 ,f d1 ,...,f dn}, where f q represents the query feature, and f didenote database features, i is the index of features in the database. Database features f di and query features where C is the channel dimension. These pooled features are then fed into an encoder module, which consists of 6 Transformer layers, each containing a self-attention layer followed by a feed-forward layer. The self-attention mechanism within the Transformer layer is able to capture and exchange relationship information between compared features. For input features F, the self-attention mechanism can be represented as:
[0056] Projection: Q = FC Q (F), K = FC K (F), V = FC V (F)
[0057]
[0058] Aggregation: F a = S`V
[0059] where FC Q / K / V denotes a fully connected layer for embedding input features into query (Q), key (K), and value (V) vectors; S is the computed affinity measure reflecting the relationship between compared features; is the output feature. Thereafter, F a is fed into a feed-forward layer. By repeating the above operation 6 times, we obtain the encoder-enhanced features, denoted as F en .
[0060] 3. Decoder: to realize index prediction;
[0061] In the model of the present application, a decoder module is used to redefine the ranking problem as a prediction task of the index position of the matching image in the database. This method can realize the integration and joint optimization of the ranking process. In this step, the enhanced query features and database features F en are input into the decoder model, which is trained to predict the index position of the database image with the same identity as the query. In order to establish the association between the database image and its index, a dynamic codebook is first constructed for the decoder based on the database features.
[0062] In fact, given the enhanced features F en , the database image features are first aggregated by a set of learnable position embeddings L = {l0, l1,..., l n , l End} to construct a dynamic codebook. This process can be formally represented as:
[0063]
[0064] Where φ={φ0,φ1,...,φ n ,φ End} represents the constructed dynamic codebook, n is the number of images in the database, and φ End This is an end marker. Subsequently, to initialize the input to the decoder module, the query feature is used as a start marker, denoted as... Then it is fed into the decoder module, which consists of two Transformer layers. This process can be written as:
[0065] S 0 =Decoder(H 0 (3)
[0066] in This is the output of the decoder module. Then, based on S... 0 The index position of the matching element is predicted by the correlation between the last element in the codebook and each index feature in the dynamic codebook. This process can be written as:
[0067] ψ 0 =softmax(η(S) 0 )·φ) (4)
[0068] Where η is a function S used to select the last element in the set. 0 The index of the maximum value in vector ψ indicates the position of the currently predicted matching image. Assuming the predicted position is j, the j-th element in the codebook is then compared with H. 0 Connect them to build the input for the next round. After that, H 1 Input the decoder and repeatedly apply Equations 3 and 4 to iteratively generate the index positions of all matching images.
[0069] During training, it was noted that at each step, multiple database images share the same identity as the query image, leading to multiple potentially correct predictions. This makes the standard cross-entropy loss function (which assumes a unique prediction) ineffective for training the model. To address this issue, a custom multi-label cross-entropy loss function is proposed, which can be written as:
[0070]
[0071] Where ψ iis the i-th element in ψ; pos represents the index of all matching images in the database. By using this loss function, the decoder model can be trained to predict the index of any matching image in each round. To avoid the decoder repeatedly predicting, the index markers corresponding to the previously predicted indices are discarded from the codebook after each round. Once all index markers corresponding to the matching elements are discarded, the decoder is trained to predict the index of the end marker. In addition, during the training phase, all matching image index markers in the dynamic codebook are sorted in ascending order in the decoder module, and they are input into the decoder in parallel. A mask attention mechanism is applied to prevent forward markers from accessing subsequent markers, thereby enabling parallel training of the decoder.
[0072] Embodiments:
[0073] 1. Dataset selection;
[0074] In terms of dataset setting, the present application first uses three widely used person re-identification datasets, namely Market-1501, DukeMTMC-reID and MSMT17. In addition, an occluded person re-identification dataset Occluded Duke is also included to evaluate the robustness of the model of the present application to occlusion. In addition, two clothing change datasets PRCC and LTCC are also used by the present application to evaluate the performance of the model under different dressing conditions. This comprehensive evaluation can better understand the robustness of the method of the present application under different scenarios.
[0075] 2. Implementation details setting;
[0076] The present application adopts the visual encoder of the CLIP model as the feature extractor, and trains the model for 60 cycles with a batch size of 128. During training and testing, the images of each person are adjusted to a size of (256x128) to ensure consistency and facilitate direct comparison with previous work. The learning rate is set to 3.5x10 -4 During training, random horizontal flipping is performed with a probability of 0.5, and random erasing is used for data augmentation. To evaluate the trained model, the present application adopts the mainstream CMC and mAP indicators. All experiments are carried out on a single NVIDIA RTX 4090 GPU, which provides sufficient computing power to handle large-scale training and evaluation processes, ensuring that the model converges efficiently within a reasonable time.
[0077] 3. Training and inference steps;
[0078] During the training phase, the present application first inputs the image into the feature extractor to extract basic features. In this step, a metric loss such as triplet loss is used to train the feature extractor. The loss function used to train the extractor is represented as Subsequently, the extracted features are resampled, and the resampled features are sent to the encoder model. Then, an identity classification loss is applied The enhanced features are generated by the encoder module, which guides it to capture effective relationship information and enhance the discriminability of the extracted features. Then, the enhanced features are input into the decoder module, and the MCE loss is used for training. The overall loss function of StreamReID can be written as:
[0079]
[0080] In the testing stage, to ensure the inference efficiency, the feature extractor of the StreamReID model is used to extract the basic features of the query image and the database image. Based on their cosine similarity, 100 most difficult-to-distinguish candidate objects are preliminarily selected for each query, and then these candidate objects are input into the StreamReID model together with the query image for prediction. In the decoder module, the query feature is used as the initial label, and the previously predicted index label is gradually removed from the codebook in each round to avoid repeated prediction. In this step, the end label is discarded, and the decoder is required to continue prediction until the entire candidate object list is traversed, so as to construct the final ranking list according to the predicted order.
[0081] 4. Implementation environment
[0082] The present application is implemented using Sugon-W580-G20 servers and Linux Ubuntu 16.04.4LTS operating systems, trained using an NVIDIA GeForce GTX 3090Ti graphics processor, accelerated using the NVIDIA CUDA 10.2 platform for training, and configured using Python 3.8.13(GCC 7.5.0) and PyTorch1.11.0, as well as dependent libraries such as Numpy1.19.2 and Pillow 9.0.2 in terms of software configuration.
Claims
1. An end-to-end person re-identification method based on index prediction, characterized in that, Includes the following steps: Step 1: Hard sampling strategy; The ViT model was used as the feature extractor. Input the sample images into the ViT model to extract features; The extracted features are resampled for use in training the subsequent encoder-decoder module; Step 2: Build an encoder to enable the exchange of relational information; Step 2-1: Perform spatial average pooling on the image features extracted from the feature extractor to compress them into a one-dimensional vector; the pooled query features and database features are represented as F = {f q ,f d0 ,f d1 ,...,f di ,....,f dn }, where f q Indicates query characteristics, f di This represents a database feature, where i is the index of that feature in the database. Database feature f di and query features Where C represents the channel dimension; Step 2-2: Input the pooled feature F into the encoder module. The encoder module consists of 6 Transformer layers, each containing a self-attention layer followed by a feedforward layer. For the input feature F, the self-attention mechanism is expressed as: Projection:Q=FC Q (F),K=FC K (F),V=FC V (F) Aggregation:F a =S`V (1) FC Q FC K FC V All represent fully connected layers, used to embed input features into query Q, key K, and value V vectors respectively; d represents the dimension of the feature vector; S is the calculated affinity measure, reflecting the relationship between the compared features; It is the output feature; Then F a It is fed into the feedforward layer; Step 2-3: Repeat step 2-2 six times to obtain the feature enhanced by the encoder, denoted as F. en ; Step 3: Build a decoder to perform index prediction; Step 3-1: Aggregate image features from the database Through a set of learnable positional embeddings L = {l0, l1, ..., l n ,l End The process of constructing a dynamic codebook can be formally represented as follows: Where φ={φ0,φ1,...,φ n ,φ End } represents the constructed dynamic codebook, n is the number of images in the database, and φ End It is an end marker; Step 3-2: Use the query feature as the starting marker, denoted as... Then it is sent to the decoder module, which consists of two Transformer layers. This process is written as follows: S 0 =Decoder(H 0 ) (3) in It is the output of the decoder module; Step 3-3: Based on S 0 The correlation between the last element in the codebook and each index feature in the dynamic codebook predicts the index position of the matching element. This process can be written as: ψ 0 =softmax(η(S 0 )·φ) (4) Where η is a function S used to select the last element in the set. 0 The index of the maximum value in vector ψ indicates the location of the currently predicted matching image. Steps 3-4: Assuming the predicted position is j, combine the j-th element in the dynamic codebook with H. 0 Connect them to build the input for the next round. After that, H 1 Input the decoder and repeatedly apply equations (3) and (4) to iteratively generate the index positions of all matching images; Step 4: Loss Function; Using the multi-label cross-entropy loss function, it can be written as: Where ψ i It is the i-th element in ψ; pos represents the index of all matching images in the database; By using the loss function described above, the decoder model can be trained to predict the index of any matching image in each round.
2. The end-to-end pedestrian re-identification method based on index prediction according to claim 1, characterized in that, Step 1 specifically involves: A random image feature is selected as the query, and a database is built for it. Specifically, a memory library is used to store the features of the first 10,000 images, and the 100 images most similar to the query but with different identities are selected from them to build the database. Next, the image features with the same identity as the query in the current mini-batch are randomly inserted into different positions in the database. The constructed database, together with the query features, is used as the input to the encoder-decoder model. Finally, the encoder-decoder model is trained to predict the location of the matching image.
3. The end-to-end pedestrian re-identification method based on index prediction according to claim 2, characterized in that, When training the decoder model: to avoid the decoder making repeated predictions, the tags corresponding to the previously predicted indices are discarded from the codebook after each round. Once all the index tags corresponding to the matching elements have been discarded, the decoder is trained to predict the index of the end tag. During the training phase, in the decoder module, all matching image index tags in the dynamic codebook are sorted in ascending order and input into the decoder in parallel. A masked attention mechanism is applied to prevent forward tags from accessing subsequent tags, thereby enabling parallel training of the decoder.
4. A computer program, characterized in that, The computer program causes the computer to perform the method as described in any one of claims 1 to 3.
5. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method as described in any one of claims 1 to 3.
6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 3.
7. A chip, characterized in that, include: A processor for retrieving and running a computer program from memory, causing a device on which the chip is mounted to perform the method as described in any one of claims 1 to 3.
8. A computer program product, characterized in that, The computer program product includes a computer storage medium storing a computer program, the computer program including instructions executable by at least one processor, which, when executed by the at least one processor, implement the method as described in any one of claims 1 to 3.