A lifelong object re-identification method based on component-cue learning

Abstract

The present invention provides a method for lifelong target re-identification based on component cue learning, which relates to the field of target re-identification in computer vision. It does not need to store target image data of old tasks for playback in the subsequent new task learning process, but uses a cue learning method to guide a pre-trained multimodal visual language model to continuously learn multiple re-identification tasks. The present invention divides learnable cues into global cues and multiple component cues, wherein the global cues focus on the overall features of the target image, while the multiple component cues focus on the upper, middle and lower three local areas of the target respectively, and guides the model to focus on the discriminative local areas of the target with fine-grained visual cues, further improving the discriminative expression ability of the model to extract features.

Description

Technical Field

[0001] The present invention relates to the field of target re-identification in computer vision, and specifically proposes a lifelong target re-identification method. Background Art

[0002] Target re-identification refers to the technology of identifying and tracking the same target captured by different cameras at different times and locations in surveillance videos. This technology is one of the hot research directions in the current computer vision field and has been widely used in the fields of intelligent security and intelligent transportation. Existing target re-identification algorithms are generally trained on a pre-collected fixed data set, and the algorithm performance is evaluated on the test set. However, the surveillance video stream in the real world is continuously generated, which requires the target re-identification model to incrementally learn from the continuously generated new data while not forgetting the knowledge that has been learned. To solve this problem, one of the most direct ways is to mix the new and old sample data and train the entire model from scratch. However, this method requires huge storage space, computing resources and time costs. In practical applications, the target re-identification model needs to continuously learn new knowledge from new data in a real-life learning environment to continuously improve the recognition ability of the model while preventing the forgetting of the old knowledge that has been learned previously.

[0003] In recent years, lifelong object re-identification methods have attracted more and more attention from researchers. Existing methods usually save image sample data of old tasks as a memory buffer, then replay the data samples of old tasks in new tasks and use model distillation technology to overcome the catastrophic forgetting problem generated in the incremental learning process. However, previous image data is usually not stored and accessible due to privacy issues. Without reviewing data samples of past old tasks, existing methods will have serious catastrophic forgetting problems, and the incremental learning performance of the model will be significantly reduced. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a lifelong target re-identification method based on component cue learning. There is no need to store additional image samples for data playback in the subsequent new task learning process. Instead, a cue learning method is used to guide the pre-trained multimodal visual language model to continuously learn multiple incremental tasks. In order to guide the pre-trained model to focus on different local areas of the target to extract discriminative fine-grained features, the present invention divides the learnable cues into global cues and multiple component cues, wherein the global cues focus on the overall image features of the target, while the multiple component cues focus on the upper, middle and lower three local areas of the target image respectively, and use fine-grained visual cues to guide the model to focus on the discriminative local areas of the target, thereby improving the model's ability to discriminate and express different target identity features.

[0005] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0006] Step 1: Model and dataset preparation;

[0007] The pre-trained multimodal visual language pre-training model CLIP is used as the backbone network. The model structure of CLIP includes an image encoder and a text encoder. The image encoder converts the input image into visual embedding features, and the text encoder converts the related text description into text embedding features. The original CLIP model uses a contrastive learning strategy to pull the associated image and text embedding features into a shared vector space, thereby achieving cross-modal visual-text semantic understanding and alignment.

[0008] Since the target ID label is a numerical number without semantic meaning, in order to utilize the powerful cross-modal visual language association alignment capability of the pre-trained CLIP model, the present invention adopts a two-stage training strategy, such as Figure 1 As shown in , the first stage uses the CLIP text encoder to learn identity-specific natural language text descriptions for each target identity ID; Figure 2 As shown in the figure, in the second stage, identity-specific text descriptions and identity ID numerical labels are used together to guide the training of the CLIP image encoder. There is no need to fine-tune the entire CLIP image encoder. Instead, a small amount of learnable hint parameters are added through hint learning to achieve efficient parameter fine-tuning.

[0009] Lifelong target re-identification aims to integrate continuous Datasets The data is input into the CLIP model in a sequential data stream and learned in a Task Incremental Learning (TIL) setting, that is, each data set is used as an incremental learning task; Each dataset contains training sets, query sets, and gallery sets. The training sets are used to perform incremental training on the CLIP model, and the query sets and gallery sets are used to test the final trained CLIP model.

[0010] In training When you create a new dataset, you do not need to save the previous The image data of the dataset, the model is continuously learned After the data set is trained, the final test model is obtained.

[0011] Step 2: First phase of training;

[0012] In getting the After the training set of the dataset is obtained, the first stage of training is carried out. The goal of the first stage is to learn the identity-specific natural language text description for each target identity ID, such as Figure 1 As shown, use the text template "A photo of a person." describes a specific target identity, where represents the learnable identity description word embedding vector, , Indicates the number of word embedding vectors; in the first training phase, the parameters of the CLIP text encoder and image encoder are frozen, and only the learnable natural language text description is optimized;

[0013] Step 3: Second stage training;

[0014] like Figure 2 As shown in the figure, the goal of the second stage training is to add a small amount of learnable parameters through prompt learning to efficiently fine-tune the parameters of the CLIP image encoder. The language text description of the target identity and the parameters of the CLIP text encoder remain frozen, and the CLIP image encoder backbone network also remains frozen, and only the additional global prompt and component prompt parameters are trained; the forward pass and parameter optimization process of the CLIP image encoder includes 5 processes, including two-dimensional input image feature serialization, global and component prompt parameter addition, component perception attention calculation, final output feature calculation and loss function calculation;

[0015] Step 4: Figure 3 As shown in In the incremental learning task stage, we first obtain the training set of the data set corresponding to the current task for the first stage of training. In the first stage, we optimize the loss function Learn identity-specific natural language text descriptions for each target identity ID; then enter the second stage of training, which optimizes the loss function Achieve efficient fine-tuning of the parameters of the CLIP image encoder; after completing the second stage of training, save the global prompt and component prompt parameters corresponding to the current task, then learn the next incremental task, and repeat steps 2-3 until all are completed. After learning each task, enter the next step of the testing process;

[0016] Step 5: Figure 3As shown in Figure 2, in the test phase, for each incremental learning task, the prompt parameters of the corresponding task are attached to each layer of the CLIP image encoder, and the query set and gallery set of the corresponding task dataset are sent to the CLIP image encoder to obtain the encoded visual features. Then, the mAP and Rank-1 performance indicators are calculated based on the obtained query and gallery visual features. Finally, The average test performance on the dataset.

[0017] In step 2, the optimization steps are:

[0018] The optimization objective function includes image-to-text contrast loss Contrasting loss with text-to-image , image to text contrast loss The calculation formula is as follows:

[0019]

[0020] in is the batch size during training, Representative target samples, Represents the first target samples, the image encoder extracts the The visual features of the target samples are , the text encoder extracts The natural language text description of the target sample is the text feature , Calculate the similarity between visual features and text features; in the loss function, Text features of target samples Only with Visual features of target samples associated; since multiple target images in a batch correspond to the same target identity ID, the text-to-image comparison loss The calculation method of Slightly different, as shown in the following formula:

[0021]

[0022] in Indicates The identity ID of the target sample, Indicates batch The corresponding index set of all positive sample images, Indicates the number of elements in a collection; represents the first target samples, Indicates The identity ID of the target sample, the total loss function of the first stage training is calculated as follows:

[0023]

[0024] optimization The model learns identity-specific natural language text descriptions for each target identity ID. The target image dataset of identity IDs is obtained after the first stage of training. The natural language text description features are used to supervise the training of the second stage image encoder.

[0025] In step 3, the specific steps are as follows:

[0026] Step 3-1: Convert the two-dimensional input image into sequence features;

[0027] Step 3-2: Add global hint and component hint parameters to the pre-trained model;

[0028] Step 3-3: Calculate component-aware attention;

[0029] Step 3-4: Calculate the final output features;

[0030] Step 3-5: Calculate the second stage training loss function;

[0031] In step 3-1, for the input sample image ,in Represents the number of channels of the image, including three color channels: RGB. Represents the height of the image (Height), Represents the width of the image. First, the input two-dimensional image is divided into pieces of size after passing through the image block embedding layer (Patch Embedding). The image block is obtained Image patch sequence features , Represents the embedding dimension of the vector, each image patch is regarded as a patch token; then, a class token With embedded features After concatenation in the sequence dimension, it is combined with the positional embedding parameters Adding them together gives:

[0032]

[0033] in, is the input sequence feature.

[0034] In step 3-2, the steps of adding global prompts and component prompt parameters to the pre-trained model are:

[0035] Subsequently, the hint learning method is used to add additional hint instructions to the pre-trained model to guide the pre-trained model to adapt to downstream tasks. The hint instructions contain a set of learnable parameters, and the parameters of the pre-trained model itself are frozen. During training, only the learnable hint parameter set is trained. ,in For global prompts, is a component prompt, where Represents the number of layers of the ViT (Vision Transformer) model, and is a pre-set hyperparameter, where represents the sequence length of the global prompt, Represents the sequence length of component hints; global hints add an additional set of learning parameters to each layer of Transformer encoder, and then the corresponding dimension output after each layer of Transformer encoder is discarded, while component hints are only added to the input sequence features before the first layer of Transformer encoder. After concatenation, the corresponding dimensional outputs after each layer of Transformer encoder are not discarded, but directly retained and forwarded to the next layer of Transformer encoder; therefore, the input sequence feature of the first layer of Transformer encoder is :

[0036]

[0037] in Indicates the index No. 0, which is the first-level global prompt parameter, followed by the sequence feature The multi-head self-attention (MHSA) module in the first-layer Transformer encoder is sent to calculate; first, the input sequence features Perform layer normalization to speed up and stabilize model training convergence:

[0038]

[0039] Then, three separate fully connected layers are used Normalize the features Converted into query, key and value features respectively :

[0040]

[0041]

[0042] ;

[0043] Represents query features The global hint features, component hint features, category token features and image patch token features in Respectively represent key features The global hint features, component hint features, category token features and image patch token features in the Represents the value features The global hint features, part hint features, category token features and image patch token features in .

[0044] The steps of calculating the component perception attention in step 3-3 are:

[0045] In the multi-head attention of the Transformer encoder, in order to extract the overall features of the target image, the category token and all image block tokens are self-attentioned as follows:

[0046]

[0047] in ,The category token can focus on the overall features of the target image and is used to determine the overall identity of the target; They respectively represent the query features, key features, and value features composed of the category tokens and all image block tokens involved in the global attention calculation;

[0048] In order to guide the pre-trained model to focus on the local area of ​​the target image to extract discriminative fine-grained features, the present invention introduces component-aware attention calculation in multi-head attention. Winning , the three components prompt to focus on the upper, middle and lower local areas of the target image respectively, The three component cue vectors represent the upper, middle and lower local areas of the target image. The calculation method of component-aware attention is as follows:

[0049]

[0050] in Indicates Part Token, with:

[0051]

[0052]

[0053]

[0054] in, Respectively expressed participation The query features, key features and value features are composed of the component tokens, category tokens and all local area image patch tokens calculated by the component-aware attention. Represents query features The part hint features, category token features, part-aware attention start patch and end patch token features in . Respectively represent key features The part hint features, category token features, part-aware attention start patch and end patch token features in . Represents the value features The component hint features, category token features, component-aware attention start image patch and end image patch token features in Indicates the continuous selection and Component tokens for attention calculation The subscript of the local area image block token, the component-aware attention calculation method can make the component prompt focus on the local area of ​​the target image, and guide the model to focus on the fine-grained features of the target with fine-grained visual cues;

[0055] Global cues are used to fine-tune the pre-trained model. The attention calculation method of global cues is as follows:

[0056]

[0057] That is, the global hint will be calculated with the component hint, category token and all image block tokens. The global hint is used to encode high-level instructions to guide the backbone network to use the feature expression ability of the pre-trained model to complete downstream tasks.

[0058] The steps of calculating the final output features in step 3-4 are:

[0059] Normalized input sequence features After passing through the multi-head attention MHSA module in the Transformer encoder, it will be added to the original input to obtain the output corresponding to the module, as shown in the following formula:

[0060]

[0061] The residual connection method can effectively prevent the gradient vanishing and gradient exploding problems of deep network models during training. Subsequently, the output features of the MHSA module are normalized and sent to the feed forward network (FFN) implemented by two fully connected layers (Multi-Layer Perceptron, MLP), and then added to the original input to obtain the output features of the FFN module. , as shown below:

[0062]

[0063] Since the global hint sets a set of parameters for each layer of Transformer encoder to learn separately, the output of the previous layer of encoder needs to discard the output of the corresponding dimension before passing it to the next layer of encoder, as shown below:

[0064]

[0065] The output of the corresponding dimension of the component prompt is retained, and the obtained features This is the final output of the first layer of Transformer encoder layer. Before being sent to the second layer of Transformer encoder, the global prompt of the second layer needs to be spliced ​​in the sequence dimension and then forwarded layer by layer. After the Transformer encoder layer, the final output features of the entire backbone network are obtained as follows:

[0066]

[0067]

[0068]

[0069]

[0070]

[0071] in Representative Layer Transformer encoder, the final output features , including 3 component tokens, 1 category token and several image patch tokens.

[0072] The steps 3-5 to calculate the second stage training loss function are:

[0073] The goal of the second training phase is to use identity-specific text description features and identity ID numerical labels to guide the training of the CLIP image encoder to improve the identity identification ability of the image encoder. The parameters of the language text description and text encoder remain frozen, and the image encoder backbone network also remains frozen, and only a small number of additional prompt parameters are trained. In order to improve the identity identification ability of the image encoder, the cross entropy loss based on the identity ID is used. and triplet loss To optimize the learning of prompt parameters, the calculation formula is as follows:

[0074]

[0075]

[0076] in Indicates that the model predicts The probability of an identity ID, smooth label Indicates The true probability of an identity ID; if That is to say is the real identity ID, then ,otherwise , is a constant value, which is taken as 0.1 in the present invention, and is used to add noise to the real label to improve the generalization ability of the model. and Respectively represent the distance between the features of the positive sample pair (positive) and the negative sample pair (negative), is a pre-set boundary value; identity ID loss Acting on the category token and the three component tokens in the output features of the last layer Transformer encoder, the triplet loss Acting on the category tokens in the output features of the last layer Transformer encoder; in order to make full use of CLIP's powerful visual language alignment capabilities, the first stage saved The natural language text description features of each identity ID can also provide high-level semantic supervision information, so the image-to-text cross-entropy loss is calculated as follows:

[0077]

[0078] Indicates The identity ID of the target sample, Indicates The identity ID of the target sample, Indicates The natural language text description features corresponding to the target ID, Indicates The natural language text description features corresponding to the target identity ID;

[0079] The total loss function of the second stage training is:

[0080]

[0081] in is the weight coefficient. This training stage further improves the identity identification and generalization capabilities of the image encoder by collaboratively optimizing the identity ID loss, triplet loss, and image-to-text cross entropy loss.

[0082] An electronic device comprises one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described above.

[0083] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.

[0084] The beneficial effects of the present invention are as follows:

[0085] The present invention does not need to additionally store target image data during the incremental learning process for playback during subsequent new task learning, thus avoiding the problems of privacy leakage and inaccessibility of old task data. Task-specific knowledge instructions are encoded in the prompt parameters, which can effectively save storage space expenses.

[0086] The present invention does not need to fine-tune the entire CLIP image encoder model, but instead adds a small amount of learnable hint parameters through hint learning to achieve efficient parameter fine-tuning, thereby greatly reducing the consumption of computing resources, speeding up the training speed and improving the generalization ability of the model. During the model training process, the parameters of the pre-trained CLIP model itself are frozen, and the model's general visual language understanding and alignment capabilities are retained.

[0087] Based on prompt learning, the present invention divides prompts into global prompts and multiple component prompts. The global prompts are responsible for focusing on the overall features of the target image, and the component prompts are responsible for focusing on different local areas of the target to extract discriminative fine-grained features. The two complement each other and synergistically improve the discriminative expression ability of the model's extracted features. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1This is the first stage training network structure diagram of the method of the present invention.

[0089] Figure 2 This is the second stage training network structure diagram of the method of the present invention.

[0090] Figure 3 It is a training and testing flow chart of the method of the present invention. DETAILED DESCRIPTION

[0091] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0092] The present invention provides a lifelong target re-identification method based on component cue learning. It does not need to store the target image data of old tasks for playback in the subsequent new task learning process. Instead, it uses the cue learning method to guide the pre-trained multimodal visual language model to continuously learn multiple re-identification tasks. The present invention divides the learnable cues into global cues and multiple component cues, wherein the global cues focus on the overall features of the target image, while the multiple component cues focus on the upper, middle and lower local areas of the target respectively, and guides the model to focus on the discriminative local areas of the target with fine-grained visual cues, further improving the discriminative expression ability of the model to extract features.

[0093] The technical solution adopted by the present invention comprises the following steps:

[0094] Step 1: Model and dataset preparation;

[0095] The present invention uses the pre-trained multimodal visual language pre-training model CLIP as the backbone network. The model structure of CLIP includes an image encoder and a text encoder. The image encoder is specifically a ViT-B / 16 model, which includes 12 Transformer encoder layers and 12 self-attention heads. " / 16" means that the input image is divided into image blocks of size 16×16, and the embedding dimension of the visual feature is , each encoder layer includes a multi-head self-attention mechanism and a feed-forward neural network. The text encoder is a 12-layer Transformer model with 8 self-attention heads. The embedding dimension of text features is 512. The Transformer model performs Byte Pair Encoding (BPE) on natural language text to represent text features.

[0096] Since the target ID label is a numerical number without semantic meaning, in order to utilize the powerful cross-modal visual language association alignment capability of the pre-trained CLIP model, the present invention adopts a two-stage training strategy. Figure 1 As shown in , the first stage uses the CLIP text encoder to learn identity-specific natural language text descriptions for each target identity ID. Figure 2As shown in FIG, the second stage adds a small amount of learnable hint parameters to the CLIP image encoder through hint learning to achieve efficient parameter fine-tuning.

[0097] Lifelong target re-identification aims to integrate continuous Datasets The sequential data stream is input into the CLIP model for learning in the task incremental learning setting, that is, each data set is used as an incremental learning task. There are three datasets, namely Market-1501, DukeMTMC, CUHK-SYSU and MSMT17 datasets. The Market-1501 dataset was collected from the campus supermarket of Tsinghua University, with a total of 1501 target identity labels; the DukeMTMC dataset was collected from Duke University, with a total of 1404 target identity labels; the CUHK-SYSU dataset was modified from the original dataset customized for target search tasks, with a total of 8432 target identity labels; the MSMT17 dataset is a large dataset close to real scenes proposed at the CVPR 2018 conference, with a total of 4101 target identity labels. In the training When a data set is generated, the present invention does not need to save the previous The image data of the dataset, the model is continuously learned After the data set is trained, the final test model is obtained.

[0098] Step 2: First phase of training;

[0099] In getting the After the training set of the data set is obtained, the first stage of training is performed. Figure 1 As shown, use the text template "A photo of a person." to describe a specific target identity, where represents the learnable identity description word embedding vector, setting word embedding vectors. The optimization objective function of this training phase includes the image-to-text contrast loss and text-to-image contrastive loss , image to text contrast loss The calculation formula is as follows:

[0100]

[0101] in is the batch size during training. In the experiment, the batch size is set to , Representative target samples. Text-to-image contrast loss The calculation formula is as follows:

[0102]

[0103] in Indicates The identity ID of the target sample, Indicates batch The corresponding index set of all positive sample images, Represents the number of elements in the set. The total loss function of the first stage of training is calculated as follows:

[0104]

[0105] The first stage of training is optimized using the Adam optimizer, and the learning rate is initialized to , the learning rate is decayed by the cosine scheduler, and a total of 120 epochs are trained. No data augmentation operation is performed in this stage, and only identity discriminative text descriptions are learned for each target identity ID.

[0106] Step 3: Second stage training;

[0107] like Figure 2 As shown in the figure, the goal of the second stage training is to add a small amount of learnable parameters through prompt learning to efficiently fine-tune the parameters of the CLIP image encoder. The forward pass and parameter optimization process of the CLIP image encoder includes five steps: two-dimensional input image feature serialization, global and component prompt parameter addition, component perception attention calculation, final output feature calculation and loss function calculation. The specific process is as follows:

[0108] Step 3-1: Convert the two-dimensional input image into sequence features;

[0109] Input sample image First, after the image block embedding layer, it is divided into image blocks of size 16×16. Image patch sequence features , each image block is regarded as an image block token, followed by a class token With embedded features After concatenation in the sequence dimension, the position encoding parameter Add to get the input sequence features :

[0110]

[0111] Step 3-2: Add global hint and component hint parameters to the pre-trained model;

[0112] Subsequently, the hint learning method is used to add hint instructions to the pre-trained model to guide the pre-trained model to adapt to downstream tasks. The learnable hint parameters include global hints and , the sequence length of the global prompt , the sequence length of the component prompt , the input sequence feature of the first layer Transformer encoder is :

[0113]

[0114] Then enter the sequence features The multi-head self-attention module in the Transformer encoder is sent for calculation. First, the input sequence features Perform layer normalization to speed up and stabilize model training convergence:

[0115]

[0116] Then, three separate fully connected layers are used Normalize the features Converted into query, key and value features respectively :

[0117]

[0118]

[0119]

[0120] Step 3-3: Calculate component-aware attention;

[0121] In the multi-head attention of the Transformer encoder, in order to extract the overall features of the target image, the category token and all image block tokens are self-attentioned as follows:

[0122]

[0123] in ,The category token can focus on the overall characteristics of the target image and is used to determine the overall identity of the target.

[0124] In order to guide the pre-trained model to focus on the local area of ​​the target to extract discriminative fine-grained features, the present invention introduces component-aware attention calculation in multi-head attention. Winning , which are responsible for paying attention to the upper, middle and lower local areas of the target image respectively. The calculation method of component perception attention is as follows:

[0125]

[0126] in Indicates component tokens, with:

[0127]

[0128]

[0129]

[0130] in Indicates the continuous selection and Component tokens for attention calculation Specifically, the size of ( ) is divided into three overlapping local rectangular areas vertically: , and , the size of each local region is Each component token only performs attention calculation with the image block token of the corresponding local area. The component-aware attention calculation method can make the component prompt pay attention to the local area of ​​the target image and improve the fine-grained feature discrimination ability of the model.

[0131] Global cues are used to fine-tune the pre-trained model. The attention calculation method of global cues is as follows:

[0132]

[0133] That is, the global cue is calculated with the part cue, the category token, and all the patch tokens for attention.

[0134] Step 3-4: Calculate the final output features;

[0135] Normalized input sequence features After passing through the MHSA module, it will be added to the original input to obtain the output corresponding to the module, as shown in the following formula:

[0136]

[0137] Subsequently, the output features of the MHSA module are normalized and sent to the feedforward network implemented by two fully connected layers, and then added to the original input to obtain the output features of the FFN module. , as shown in the following formula:

[0138]

[0139] Since the global hint sets a set of parameters for each layer of Transformer encoder to learn separately, the output of the previous layer of encoder needs to discard the output of the corresponding dimension before passing it to the next layer of encoder, as shown below:

[0140]

[0141] The output of the corresponding dimension of the component prompt is retained, and the obtained features This is the final output of the first-layer Transformer encoder layer. Before being sent to the second-layer Transformer encoder, the global prompt of the second layer needs to be spliced ​​in the sequence dimension, and then forwarded layer by layer according to the above process. After the Transformer encoder layer, the final output features of the entire backbone network are obtained as follows:

[0142]

[0143]

[0144]

[0145]

[0146]

[0147] in , the final output features obtained , including 3 component tokens, 1 category token and several image patch tokens.

[0148] Step 3-5: Calculate the second stage training loss function;

[0149] like Figure 2 As shown, the second training stage uses the cross entropy loss based on the identity ID and triplet loss To optimize the learning of prompt parameters, the calculation formula is as follows:

[0150]

[0151]

[0152] in Indicates that the model predicts The probability of an ID, smooth label Indicates The true probability of an identity ID is set to a constant value is 0.1, and Respectively represent the distance between the features of the positive sample pair and the negative sample pair, and set the boundary value is 0.3. Identity loss Acting on the category token and the three component tokens in the output features of the last layer Transformer encoder, the triplet loss Acting on the category tokens in the output features of the last layer Transformer encoder. In addition, the image-to-text cross entropy loss is calculated as follows:

[0153]

[0154] The total loss function of the second stage training is:

[0155]

[0156] in is the weight coefficient, where , the batch size of this training phase is set to Each batch contains 16 target IDs, each target ID contains 4 images, random horizontal flipping, padding, cropping and erasing data augmentation operations are used, and the Adam optimizer is used for optimization. The initial learning rate is set to After 10 epochs, the learning rate increases linearly to Then the learning rate is decayed to 0.1 times of the original value at the 30th and 50th epochs, and a total of 60 epochs are trained.

[0157] Step 4: Figure 3 As shown in In the incremental learning task stage, we first obtain the training set of the data set corresponding to the current task for the first stage of training. This stage optimizes the loss function For each target identity ID, a natural language text description of the identity is learned. Then the second stage of training begins, which optimizes the loss function After completing the second stage of training, save the global and component cue parameters corresponding to the current task, and then learn the next incremental task, repeating steps 2-3 until all tasks are completed. After completing the learning of each task, proceed to the next step of the testing process.

[0158] Step 5: Figure 3As shown in Figure 2, in the test phase, for each incremental learning task, the prompt parameters of the corresponding task are attached to each layer of the CLIP image encoder, and the query set and gallery set of the corresponding task dataset are fed into the CLIP image encoder to obtain the encoded visual features, and then the mAP and Rank-1 performance indicators are calculated based on the obtained query and gallery visual features. Finally, the calculation The average test performance on the dataset.

[0159] The performance results of the present invention on four incremental object re-identification datasets, namely Market-1501, DukeMTMC, CUHK-SYSU and MSMT17, are shown in Table 1. The comparison algorithms include LwF, GwF, PTKP and KRKC. LwF is a classic and widely used incremental learning method for natural images, which comes from the literature "Li Z, Hoiem D. Learning without forgetting[J]. IEEE transactions on pattern analysis and machine intelligence, 2017, 40(12): 2935-2947.", GwF, PTKP and KRKC are the latest incremental object re-identification methods in recent years, which come from the literature "Wu G, Gong S. Generalising without forgetting for lifelong person re-identification[C] / / Proceedings of the AAAI conference on artificialintelligence. 2021, 35(4): 2889-2897.", and Ge W, Du J, Wu A, et al. Lifelongperson re-identification by pseudo task knowledge preservation[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(1):688-696." and "Yu C, Shi Y, Liu Z, et al. Lifelong person re-identification via knowledge refreshing and consolidation[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(3): 3295-3303."

[0160] Table 1 Comparison of performance results of the method of the present invention and other methods

[0161]

[0162] From the comparison of the experimental results in Table 1, it can be found that the method of the present invention has achieved the best results both in the single task performance of each target re-identification dataset and in the overall average performance of all datasets.

Claims

1. A lifelong object re-identification method based on component-cue learning, characterized in that The steps include: Step 1: Model and dataset preparation; The pre-trained multimodal visual language pre-training model CLIP is used as the backbone network. The model structure of CLIP includes an image encoder and a text encoder. The image encoder converts the input image into visual embedding features, and the text encoder converts the related text description into text embedding features. The original CLIP model uses a contrastive learning strategy to pull the associated image and text embedding features into a shared vector space, thereby achieving cross-modal visual-text semantic understanding and alignment. Lifelong object re-identification aims to integrate T consecutive datasets collected in different data domains. The data is input into the CLIP model in a sequential data stream and learned in a task incremental learning setting, that is, each data set is used as an incremental learning task; Each of the T datasets contains a training set, a query set, and a gallery set. The training set is used to incrementally train the CLIP model, and the query set and gallery set are used to test the final trained CLIP model. When training the tth dataset, there is no need to save the image data of the previous 1st, 2nd, …, t-1th datasets. After the model continuously learns T datasets, it obtains the final trained test model. Step 2: First phase of training; After obtaining the training set of the t-th dataset, the first stage of training is performed. The goal of the first stage is to learn identity-specific natural language text descriptions for each target identity ID, using the text template "A photo of a[X]1[X]2…[X] m …[X] M person." describes a specific target identity, where [X] m represents the learnable identity description word embedding vector, m∈{1,2,…,M}, M represents the number of word embedding vectors; in the first training stage, the parameters of the CLIP text encoder and image encoder are frozen, and only the learnable natural language text description is optimized; In step 2, the optimization steps are: The optimization objective function includes image-to-text contrast loss and text-to-image contrastive loss Image to Text Contrastive Loss The calculation formula is as follows: Where B is the batch size during training, k∈{1,2,…,B} represents the kth target sample, a∈{1,2,…,B} represents the ath target sample in the training batch, p represents the pth target sample in the training batch, and y p Represents the identity ID of the pth target sample, and the image encoder extracts the visual features of the kth target sample as V k , the text encoder extracts the natural language text description of the kth target sample as the text feature T k ,S(V k ,T k )=V k ·T k Calculate the similarity between visual features and text features; in the loss function, the text feature T of the kth target sample k Only the visual feature V of the kth target sample k associated; since multiple target images in a batch correspond to the same target identity ID, the text-to-image comparison loss The calculation method of Slightly different, as shown in the following formula: where y k represents the identity ID of the kth target sample, P(y k )={p∈{1,2,…,B}|y p =y k } indicates that the batch The corresponding index set of all positive sample images, |·| represents the number of elements in the set; the total loss function of the first stage training is calculated as follows: optimization The model is made to learn identity-specific natural language text descriptions for each target identity ID. For a target image dataset containing N identity IDs, N natural language text description features are obtained after the first stage of training to supervise the training of the second stage image encoder. Step 3: Second stage training; The goal of the second stage of training is to add a small amount of additional learnable parameters through prompt learning to efficiently fine-tune the parameters of the CLIP image encoder. The language text description of the target identity and the parameters of the CLIP text encoder remain frozen, and the CLIP image encoder backbone network also remains frozen. Only the additional global prompts and component prompts parameters are trained. In step 3, the specific steps are as follows: Step 3-1: Convert the two-dimensional input image into sequence features; Step 3-2: Add global hint and component hint parameters to the pre-trained model; In step 3-2, the steps of adding global prompts and component prompt parameters to the pre-trained model are: Subsequently, the hint learning method is used to add additional hint instructions to the pre-trained model to guide the pre-trained model to adapt to downstream tasks. The hint instructions contain a set of learnable parameters, and the parameters of the pre-trained model itself are frozen. During training, only the learnable hint parameter set is trained. global ,P part },in For global prompts, is a component hint, where L represents the number of layers in the ViT model, L global and L part is a pre-set hyperparameter, where L global represents the sequence length of the global hint, L part Represents the sequence length of component hints; the global hint adds a set of additional learning parameters to each layer of Transformer encoder, and then the corresponding dimension output after each layer of Transformer encoder is discarded, while the component hint is only concatenated with the input sequence feature F before the first layer of Transformer encoder, and then the corresponding dimension output after each layer of Transformer encoder is not discarded, but directly retained and forwarded to the next layer of Transformer encoder; therefore, the input sequence feature of the first layer of Transformer encoder is F (0) : F (0) =Concat[P global [0];P part ;F] in Indicates the index No. 0, which is the first-layer global prompt parameter, followed by the sequence feature F (0) It is sent to the multi-head self-attention module in the first layer Transformer encoder for calculation; first, the input sequence feature F (0) Perform layer normalization to speed up and stabilize model training convergence: Then, three separate fully connected layers are used Normalize the features Converted into query, key and value features respectively Represent the query features Q (0) The global hint features, component hint features, category token features and image patch token features in Denote the key features K (0) The global hint features, component hint features, category token features and image patch token features in the Represents the value feature V (0) The global hint features, component hint features, category token features and image patch token features in the Step 3-3: Calculate component perceived attention; the steps of calculating component perceived attention in step 3-3 are: In the multi-head attention of the Transformer encoder, in order to extract the overall features of the target image, the category token and all image block tokens are self-attentioned as follows: in The category token can focus on the overall features of the target image and is used to identify the overall identity of the target; They respectively represent the query features, key features, and value features composed of the category tokens and all image block tokens involved in the global attention calculation; In order to guide the pre-trained model to focus on the local area of ​​the target image to extract discriminative fine-grained features, component-aware attention calculation is introduced in the multi-head attention. Take L part =3, the three component prompts focus on the upper, middle and lower local areas of the target image respectively, p1, p2, p3 represent the three component prompt vectors focusing on the upper, middle and lower local areas of the target image respectively, and the calculation method of component perception attention is as follows: Where i∈{1,2,3} represents the i-th part token, and: in, They respectively represent the query feature, key feature, and value feature composed of the component token, category token, and all local area image block tokens involved in the i-th component perception attention calculation; Represents query features The component hint features, category token features, component-aware attention start image patch and end image patch token features in the image; Respectively represent key features The component hint features, category token features, component-aware attention start image patch and end image patch token features in the image; Represents the value features The component hint features, category token features, component-aware attention start image patch and end image patch token features in {s i ,…,s i +n} represents the subscript of n local area image block tokens that are continuously selected for attention calculation with the i-th component token. The component-aware attention calculation method enables the component prompt to pay attention to the local area of ​​the target image, and guides the model to pay attention to the fine-grained features of the target with fine-grained visual cues; Global cues are used to fine-tune the pre-trained model. The attention calculation method of global cues is as follows: That is, the global hint will be used to perform attention calculation with the component hint, category token and all image patch tokens. The global hint is used to encode high-level instructions to guide the backbone network to use the feature expression ability of the pre-trained model to complete downstream tasks; Step 3-4: Calculate the final output features; the steps of calculating the final output features in step 3-4 are: Normalized input sequence features After passing through the multi-head attention MHSA module in the Transformer encoder, it will be added to the original input to obtain the output corresponding to the module, as shown in the following formula: The residual connection method can effectively prevent the gradient vanishing and gradient exploding problems of deep network models during training. Subsequently, the output features of the MHSA module are normalized and sent to the feedforward network implemented by two fully connected layers, and then added to the original input to obtain the output features of the FFN module. As shown below: Since the global hint sets a set of parameters for each layer of Transformer encoder to learn separately, the output of the previous layer of encoder needs to discard the output of the corresponding dimension before passing it to the next layer of encoder, as shown below: The output of the corresponding dimension of the component prompt is retained, and the obtained features This is the final output of the first layer of Transformer encoder layer. Before being sent to the second layer of Transformer encoder, the global prompt of the second layer needs to be spliced ​​in the sequence dimension, and then forward-passed through L=12 layers of Transformer encoder layer by layer to obtain the final output features of the entire backbone network, as shown below: Where l∈{1,2,…,L-1} represents the l-th layer Transformer encoder, and the final output features are obtained Contains 3 component tokens, 1 category token and several image block tokens; Step 3-5: Calculate the second stage training loss function; The goal of the second training phase is to use identity-specific text description features and identity ID numerical labels to guide the training of the CLIP image encoder to improve the identity identification ability of the image encoder. The parameters of the language text description and text encoder remain frozen, and the image encoder backbone network also remains frozen, and only a small number of additional prompt parameters are trained; in order to improve the identity identification ability of the image encoder, the cross entropy loss based on the identity ID is used and triplet loss To optimize the learning of prompt parameters, the calculation formula is as follows: where p j represents the probability of the model predicting the jth identity ID, smooth label q j =(1-∈)δ j,y +∈ / N represents the true probability of the jth identity ID; if j=y, that is, j is the true identity ID, then δ j,y =1, otherwise δ j,y =0,∈ takes 0.1, which is used to add noise to the real label to improve the generalization ability of the model; d p and d n Respectively represent the distance between the features of the positive sample pair and the negative sample pair, α is a pre-set boundary value; identity ID loss Acting on the category token and the three component tokens in the output features of the last layer Transformer encoder, the triplet loss Acting on the category tokens in the output features of the last layer Transformer encoder; in order to make full use of the powerful visual language alignment capability of CLIP, the natural language text description features of the N identity IDs saved in the first stage can also provide high-level semantic supervision information, so the image-to-text cross entropy loss is calculated as follows: y j represents the identity ID of the jth target sample, y a represents the identity ID of the a-th target sample, Represents the natural language text description feature corresponding to the j-th target identity ID, Represents the natural language text description feature corresponding to the a-th target identity ID; The total loss function of the second stage training is: Where β is the weight coefficient. This training phase further improves the identity identification and generalization capabilities of the image encoder by collaboratively optimizing the identity loss, triple loss, and image-to-text cross entropy loss. Step 4: In the tth incremental learning task stage, first obtain the training set of the data set corresponding to the current task for the first stage of training. In the first stage, the loss function is optimized. Learn identity-specific natural language text descriptions for each target identity ID; then enter the second stage of training, the second stage optimizes the loss function Achieve efficient fine-tuning of the parameters of the CLIP image encoder; after completing the second stage of training, save the global prompt and component prompt parameters corresponding to the current task, then learn the next incremental task, repeat steps 2-3 until all T tasks are learned, and enter the next step of the test process; Step 5: In the test phase, for each incremental learning task, the prompt parameters of the corresponding task are attached to each layer of the CLIP image encoder, and the query set and gallery set of the corresponding task dataset are sent to the CLIP image encoder to obtain the encoded visual features. Then, the mAP and Rank-1 performance indicators are calculated based on the obtained query and gallery visual features. Finally, the average test performance of T datasets is calculated.

2. The method for lifelong object re-identification based on component-cue learning according to claim 1, characterized in that: In step 3-1, for the input sample image Where C represents the number of channels of the image, including RGB, a total of three color channels, H represents the height of the image, and W represents the width of the image. First, the input two-dimensional image is divided into image blocks of size P×P after passing through the image block embedding layer, and L is obtained. patch =HW / P 2 Image patch sequence features D represents the embedding dimension of the vector, and each image block is regarded as an image block token; Then, a category token and the embedded feature F patch After concatenation in the sequence dimension, it is combined with the position encoding parameter Adding them together gives: F=Concat[x cls ;F patch ]+E Among them, F is the input sequence feature.

3. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-2.

4. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 2.

Images