Target re-recognition lifelong learning method based on dynamic text prompt

Through dynamic text prompts and multi-module structure, catastrophic forgetting and data storage problems in lifelong learning in target re-identification, achieving efficient lifelong learning ability and anti-forget performance.

CN119888168APending Publication Date: 2025-04-25NORTHWESTERN POLYTECHNICAL UNIV

Patent Information

Application Number
CN202411843018.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-12-13
Publication Date
2025-04-25

AI Technical Summary

Technical Problem

The existing target re-identification lifelong learning method is prone to forget old knowledge when learning new domain data, resulting in a degradation in the performance of the model on the old data set, and data playback technology leads to high memory overhead and is unable to be suitable for end-side deployment.

Method used

Using a lifelong learning method based on dynamic text prompts, the dynamic prompt fusion module, local fine-grained feature enhancement module and learnable knowledge distillation module are used to guide the model to map images of different domains to a unified semantic space, avoid overfitting and maintaining old knowledge.

Benefits of technology

It effectively solves the catastrophic forgetting problem, reduces data storage overhead, realizes the lifelong learning ability of target re-identification deployed on the end side, and improves the anti-forget performance of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119888168A_ABST
    Figure CN119888168A_ABST
Patent Text Reader

Abstract

The invention provides a target re-recognition lifelong learning method based on a dynamic text prompt, and the method comprises the steps: employing the dynamic text prompt as an anchor point from an image space to a semantic space in a dynamic prompt fusion module, guiding a model to achieve the semantics from image features to text prompt, and enabling a local fine-grained feature enhancement module to achieve the recognition of a target in a knowledge extraction process, the guidance model pays attention to knowledge transfer of finer granularity, establishes more accurate mapping between local features of finer granularity to enhance alignment of local text cues and image features, and in a learnable knowledge distillation module, utilizes variable distillation temperature coefficients to control knowledge transfer capabilities of different domains. According to the method, mapping from target images of different domains to a unified semantic space is realized, a local fine-grained feature enhancement module is introduced to realize feature consistency of a local space, and a learnable knowledge distillation module is utilized to adjust a temperature coefficient of a model in a cross-domain knowledge process; and the problem of disastrous forgetting caused by tendency to over-fit the current data set distribution is prevented.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of target re-identification, and in particular to a lifelong learning method for target re-identification based on dynamic text prompts. Background Art

[0002] In recent years, with the continuous development of artificial intelligence, object re-identification technology (Object Re-Identification, ReID) has been widely used and developed, and has important applications in the fields of criminal investigation and automated video retrieval. This task aims to find other corresponding target images in a series of targets in the gallery through a given target image or text description. The existing ReID models deployed in practice need to adapt to the continuous and changing new data in real-world scenarios. Existing methods tend to overfit on the current dataset and forget the knowledge learned on the past dataset. This phenomenon is also called catastrophic forgetting. Therefore, researchers proposed the lifelong learning of object re-identification (Lifelong Object Re-Identification, LReID) task, in which the model forgets old knowledge when learning new data from different domains. LReID assumes that only new datasets are visible at the current moment, and the old datasets previously trained are no longer accessible for use.

[0003] The key challenge of lifelong learning in object re-identification is catastrophic forgetting, that is, the model will forget the old knowledge learned in the old domain when learning data from the new domain, causing the performance of the model on the old dataset to drop significantly. The key challenges of the LReID task can be summarized as follows: 1) The model tends to overfit the domain distribution of the new input data, resulting in forgetting the knowledge acquired from the previous dataset, which is the catastrophic forgetting problem. 2) Existing lifelong learning methods mainly focus on classification tasks and emphasize inter-class differences. However, since the retrieval task itself needs to capture key features to distinguish two similar objects, LReID needs to capture fine-grained inter-class discriminative features. 3) In actual deployment scenarios, each new object is an unseen class.

[0004] Therefore, LReID requires good performance in both seen and unseen datasets. The most classic method is to use data replay technology (Rehearsal), to sample representative samples from the previous old dataset in a targeted manner, and to combine the new data and old data for the training of the current model in the current training phase, so that the model can recall the old knowledge. This data replay method requires a large memory overhead, and after removing the data replay technology, the performance of these methods often deteriorates severely, and with the continuous arrival of new data, the entire data storage overhead will increase linearly. Existing target re-identification methods are often deployed on the end side, and the data storage capacity is limited. Therefore, such methods are not suitable for actual deployment in the real world, and violate the essential requirements of the lifelong learning task of target re-identification, that is, to learn a model that resists forgetting to solve the problem of catastrophic forgetting. Summary of the invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a lifelong learning method for target re-identification based on dynamic text prompts, which uses prompts to guide the evolution of the model structure so that the model can learn to guide the model mapping images of different domains into a unified semantic space, thereby solving the problem of catastrophic forgetting and improving the shortcomings of the end-side model relying on data replay technology and requiring offline retraining. Its structure is as follows Figure 1 As shown:

[0006] The structure of the present invention is divided into three functional modules: a dynamic prompt fusion module, a local fine-grained feature enhancement module, and a learnable knowledge distillation module. Specifically, in the dynamic prompt fusion module, a dynamic text prompt is used as an anchor point from the image space to the semantic space to guide the model to realize the semantics of image features to text prompts, avoiding the tendency to overfit the feature distribution of the current data set. After that, the local fine-grained feature enhancement module is used in the knowledge extraction process to guide the model to focus on finer-grained knowledge transfer and establish a more accurate mapping between fine-grained local features to enhance the alignment of local text prompts and image features. Finally, in the learnable knowledge distillation module, a variable distillation temperature coefficient is used to control the knowledge transfer capabilities of different domains.

[0007] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0008] Step 1: Based on the method of dynamic text prompt guidance, establish the framework of the network. The framework diagram of the network is as follows: Figure 1 As shown; the network includes a dynamic prompt fusion module, a local fine-grained feature enhancement module and a learnable knowledge distillation module;

[0009] After generating the target appearance description, it is output to the encoder through the dynamic prompt fusion module for encoding and then output to the local fine-grained feature enhancement module. At the same time, the visual features extracted by the model are output through the learnable knowledge distillation module.

[0010] Step 2: Model training and network optimization

[0011] The optimization process of the network obtained in step 1 is divided into two independent stages. In the first stage, we focus on how to train the text encoder to adapt the text encoder to the image distribution features extracted by the image encoder to prevent the untrained text encoder from misleading the image encoder. The goal of the optimization is to minimize the distance between the image features of the dynamic prompts and the frozen weights to promote the mapping from text to image. The loss function L in the first stage is stage-I For L global ;

[0012] In the second stage, the image encoder is trained; the optimization goal is to minimize the distance between the image features and the text features of the frozen weights, guiding the model to learn the knowledge of the current domain; the loss function of the second stage is:

[0013] L stage-II =L id +L tri +L global +L partial+ 0.1L LKD (6)

[0014] Where L id represents ID loss, L tri represents the triplet loss, L LKD represents the learnable knowledge distillation loss, a fine-grained text-image mapping of features, L partial Represents the cosine distance between local image features and textual hints;

[0015] The model obtained in step 1 is deployed on two 3090Ti GPUs in a data-parallel manner. The weight decay rate of the first stage is set to 0.0001, the image size is unified to 256*128, and the training is iterated for 60 rounds. The distance between the image features of the dynamic prompt and the frozen weight is obtained by forward calculation, and the loss function L is used to calculate the distance between the dynamic prompt and the frozen weight. atage-I Back propagation optimizes model parameters; the weight decay rate in the second stage is set to 0.0001, the image size is unified to 256*128, and the training is iterated for 60 rounds. stage-II Minimize the distance between image features and text features with frozen weights.

[0016] Step 3: Reasoning stage;

[0017] In the model inference stage, the input image size is uniformly scaled to 256*128, and the corresponding visual features are extracted only through the trained image encoder. No additional text knowledge is introduced for inference. Features are also extracted from images in the gallery (gallery images). The visual features of the image to be retrieved are matched with the visual features in the gallery, and the features of the top five gallery images that are most similar to the image to be retrieved are taken out. If there is a target image that is the same as the image to be retrieved among the five images, the retrieval is considered successful.

[0018] The steps for building the network framework are:

[0019] Step 1-1: Construct the input of the dynamic prompt fusion module;

[0020] Step 1-2: Construct the structure of the dynamic prompt fusion module;

[0021] Step 1-3: Construct a local fine-grained feature enhancement module;

[0022] Step 1-4: Build a learnable knowledge distillation module.

[0023] The step 1-1 of constructing the input of the dynamic prompt fusion module is as follows:

[0024] BLIP (Bootstrapping Language-Image Pre-training) is used to generate a target appearance description as an invariant prompt (IP). Specifically, a series of questions (such as what color is the person's top and what style is the pants) are asked about the appearance features of the target in the BLIP image. That is, a series of target descriptions are obtained by visual question answering and combined to obtain the final target appearance description, for example: this woman has black hair, wears a red top, white pants, and black shoes; learnable parameters are introduced to constitute a target-specific knowledge prompt (Person Knowledge Prompt, PKP), which is a learnable tensor created using the pytorch language features.

[0025] The steps 1-2 of constructing the structure of the dynamic prompt fusion module are:

[0026] A 1*1 convolutional network and ReLU activation function are introduced and stacked in two layers. The 1*1 convolution is used to compress the channel of the feature, and ReLU converts the linear distribution into a nonlinear distribution. The dynamic fusion at the channel level is realized through the collaborative work of the 1*1 convolutional network and the ReLU activation function. The obtained dynamic prompt fusion module dynamically fuses the target invariant prompt and the target specific knowledge prompt in step 1-1 to obtain P DP for:

[0027] P DP =ReLU(w(ReLU(w(P IP ,P PKP )))

[0028] Where P DP Indicates DP (Dynamic Prompt), P IP Indicates the target unchanged prompt, P PKP represents the target-specific knowledge hint, w represents the convolution parameter, ReLU represents the ReLU activation function, and P DP It is fed into the text encoder of CLIP to extract text features; at the same time, the image features related to the target are extracted through the image encoder of CLIP, and the similarity between the image features and the text prompts is minimized using the cosine embedding loss:

[0029]

[0030] Where L global Represents the overall loss of the model in one stage, L sup is the cross entropy loss, and F img represents the image features, and Represents the image F img The result obtained after transposing the matrix is P DP The matrix transpose result of .

[0031] The steps 1-3 of constructing the local fine-grained feature enhancement module are as follows:

[0032] The image features are evenly divided along the dimension of image height. The image is divided into four different regions from top to bottom, representing the head features, upper body features, lower body features and foot features respectively;

[0033] After obtaining the local image features corresponding to the human body parts, the text features are also segmented for alignment in the local fine-grained feature enhancement module. The BLIP target appearance description generated in step 1-1 (for example, this woman has black hair, wears a red top, white pants, and black shoes) is segmented according to the comma as the segmentation point, and four local text semantics blocks are cut out for alignment with the local image blocks. Specifically, the position of the comma is recorded and the P obtained in step 1-2 is used as the segmentation point. DP In the image, the features are segmented according to the recorded positions to ensure that the local features of the human body parts correspond to the local text features. For example, the image feature of the red shirt in the image corresponds to P DP The semantic features of the red shirt in.

[0034] The formula is:

[0035] F img =[f head ; fupperbody ;f lowerbody ;f foot ] (1)

[0036] P DP =[p head ;p upperbody ;p lowerbody ;p foot] (2)

[0037] Among them, F img express Figure 1 The image features generated by the visual encoder in F img According to the method described in steps 1-3, the following four local image features of human body parts are obtained: head Represents the image features of the human head, f upperbody Represents the image features of the upper body, f lowerbody Represents the image features of the lower body, f foot represents the image features of human feet, p head Represents the local text semantic features of the human head, p upperbody Represents the text semantic features of the upper body, p lowerbody The text semantic features representing the lower body of the human body, p foot Text semantic features representing human feet.

[0038] Reduce the cosine distance between the target local image features and the local text hints, and enhance the fine-grained text-image mapping based on local features. The fine-grained text-image mapping of the features is described as:

[0039]

[0040] where p i Represents the text features of a certain part, f i Represents the image feature of a certain part, L partial Represents the cosine distance between local image features and text prompts, N is the number of body parts; it promotes a more fine-grained mapping of local text features and local image features, and promotes fine-grained transfer of knowledge in the spatial dimension, |*|2 represents the L2 norm.

[0041] The steps 1-4 to construct a learnable knowledge distillation module are:

[0042] Taking the feature output of the image encoder as input, in order to achieve an adaptive balance between preserving existing knowledge and learning new knowledge, two learnable parameters δ1 and δ2 are introduced, that is, the learnable values ​​of the two learnable parameters optimized by back propagation, which are used to control the temperature coefficient of the knowledge distillation process, expressed as:

[0043]

[0044] Where L LKD represents the learnable knowledge distillation loss, δ1 represents the learnable temperature coefficient of the student model, and δ2 represents the learnable temperature coefficient of the teacher network.

[0045] An electronic device comprises: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described above.

[0046] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.

[0047] The beneficial effect of the present invention is that in order to solve a series of problems in the existing lifelong learning method for target re-identification, such as the data replay (Rehearsal) method leads to large data memory overhead and cannot be deployed on the terminal side, the present invention proposes a lifelong learning method for target re-identification guided by dynamic text prompts, which can surpass the existing lifelong learning method for target re-identification without data replay. Invariant target text description prompts and learnable prompts are introduced and dynamically fused. During the training process, the structure of the prompt pool is changed and the two prompts are dynamically fused, and the text features of the prompts are extracted as image mapping anchors in the semantic space to achieve the mapping of target images in different domains to the unified semantic space. A local fine-grained feature enhancement module is introduced to achieve feature consistency in the local space. The temperature coefficient of the model in the cross-domain knowledge process is adjusted using the learnable knowledge distillation module to prevent the catastrophic forgetting problem caused by the tendency to overfit the current data set distribution. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a structural diagram of a method based on dynamic text prompt guidance.

[0049] Figure 2 It is the feature map of the local fine-grained feature enhancement module.

[0050] Figure 3 To compare the T-SNE visualization results. DETAILED DESCRIPTION

[0051] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0052] The present invention provides a lifelong learning method for target re-identification based on dynamic text prompts. The method guides target images of different domains to be mapped into a common semantic space through dynamic prompts to prevent the model from overfitting the features of the current domain. In addition, fine-grained information alignment is achieved through a local fine-grained feature enhancement module, which helps the model to perform retrieval at a fine-grained level. A learnable knowledge distillation module is also introduced to dynamically adjust the temperature of knowledge distillation between domains to achieve better knowledge transfer.

[0053] The specific implementation includes the following steps

[0054] Step 1: Data processing and optimizer settings;

[0055] Step 1-1: All target images are converted to a 256×128 image size. Each image is enhanced by random horizontal flipping, padding, cropping, and random erasing. For the optimizer, the learning rate used in this paper is 3.5×10 -4 The Adam optimizer is used, with a pre-start learning rate of 0.01 and a linear warm-up method. The number of training rounds is 60. The weight decay is set to 1e -4 , the batch size of each batch is 128 during the training phase, and the loss weight is set to 1 during the training phase. In addition, the model architecture is built on the CLIP pre-trained visual encoder, including a text and image encoder. The original position encoding of CLIP is adjusted from [197, 768] to [129, 768] to fit the target image by using bilinear interpolation.

[0056] Step 2: Network training

[0057] Step 2-1: Using the existing BLIP model, preprocess the training dataset of the current stage and generate a series of text descriptions of the image. These target appearance descriptions are combined to extract target invariant prompts. Specifically, this section feeds the target image used for training to BLIP and asks BLIP about the target appearance in the current image, such as "What color shirt is this person wearing?". The answers given by BLIP are then merged to generate a basic description of the target appearance, such as "This man is wearing a black hat, pink T-shirt, black pants, and black shoes", so that a brief description of the current dataset can be obtained. Target-specific knowledge prompts (Person Knowledge Prompt, PKP) to enrich learnable knowledge. The target-specific knowledge prompt PKP creates a learnable prompt for each target and parameters it. It learns image supplementary semantic knowledge during training to make up for the missing semantic feature details in the target invariant prompt IP, thereby providing more comprehensive and richer subtle target knowledge.

[0058] Step 2-2: The dynamic prompt module process is as follows: after obtaining the target invariant prompt IP and target specific knowledge prompt mentioned in step 2-1, it is equivalent to establishing an anchor point corresponding to the image feature in the semantic space, guiding the image feature to be mapped to a common semantic space, and narrowing the similarity between the image feature and the text feature, thereby alleviating the catastrophic forgetting problem caused by the domain gap, and obtaining a high-dimensional image feature that is resistant to forgetting.

[0059] Step 2-3: The process of the local fine-grained feature enhancement module is as follows: the dynamic prompt obtained in step 2-2 is segmented according to semantics, and the invariant prompt is used as a template to segment the dynamic prompt features of four parts, representing the text semantics of the four parts respectively. Similarly, the image features extracted by the image encoder are processed and segmented according to the image height, and four features are evenly divided, representing the head, upper body, lower body and feet respectively. The local text features and local images are aligned one-to-one, and the cosine loss function is used to close the cosine similarity between the two, guiding the model to focus on more fine-grained feature extraction.

[0060] Step 2-4: The process of the learnable knowledge distillation module is as follows: save the model of the previous stage and freeze the parameters, send the image of the current domain to the image encoder trained in the previous domain, and extract the image features F of the previous stage T-1 , close to the image features F extracted by the image encoder of the current domain T and F T-1 At this time, the model of the previous stage can be regarded as the teacher model, and the model of the current stage can be regarded as the student model. The student model is guided to learn by using knowledge distillation. In the distillation process, two learnable parameters are introduced and added to the temperature coefficient to obtain the dynamic temperature between domains, which better guides the distillation process of model knowledge.

[0061] Step 3: Model application

[0062] The application process of the present invention is summarized as follows: first, the image to be retrieved is processed and the size is adjusted to [256, 128], and then the three-channel image is converted into a tensor and sent to the image encoder to extract the features of the image to be retrieved. The text description is converted into phrases through text preprocessing to generate corresponding vectors, and then the text description is mapped into a text tensor with a shape of [77, 512] by encoding. In the text preprocessing, the missing words are mapped to 0 by default, and the result tensor of the text and image processing is obtained. The old model of the previous stage is stored, and the parameters of the target specific knowledge prompt and the related parameters of the linear mapping layer are initialized.

[0063] Combining the modules mentioned above, the image to be retrieved is processed and resized to [256,128]. The three-channel image is then converted into a tensor and sent to the image encoder to extract the features of the image to be retrieved. Image features are used for retrieval instead of text features to prevent unfair comparison caused by the introduction of additional knowledge. Specific embodiment:

[0065] Figure 3 The comparative T-SNE visualization results using the method of the present invention are shown. The upper side shows the distribution of the results of the baseline model for different target images in the feature space, and the lower side shows the distribution of the results of the present invention for different target images in the feature space. Points of different color depths represent different domain data distributions. It can be seen that the method of the present invention effectively improves the mapping ability of different domain data distributions to the common semantic space, thereby improving the lifelong learning ability of the model.

Claims

1. A lifelong learning method for object re-identification based on dynamic text prompts, characterized in that The steps include: Step 1: Based on the method guided by dynamic text prompts, the framework of the network is established. The network includes a dynamic prompt fusion module, a local fine-grained feature enhancement module, and a learnable knowledge distillation module; After generating the target appearance description, it is output to the encoder through the dynamic prompt fusion module for encoding and then output to the local fine-grained feature enhancement module. At the same time, the visual features extracted by the model are output through the learnable knowledge distillation module. Step 2: Model training and network optimization The optimization process of the network obtained in step 1 is divided into two independent stages. In the first stage, we focus on how to train the text encoder to adapt the text encoder to the image distribution features extracted by the image encoder to prevent the untrained text encoder from misleading the image encoder. The goal of the optimization is to minimize the distance between the image features of the dynamic prompts and the frozen weights to promote the mapping from text to image. The loss function L in the first stage is stage-I For L global ; In the second stage, the image encoder is trained; The optimization goal is to minimize the distance between the image features and the text features of the frozen weights, guiding the model to learn the knowledge of the current domain; the loss function of the second stage is: L stage-II =L id +L tri +L global +L partial +0.1L LKD (6) Where L id represents ID loss, L tri represents the triplet loss, L LKD Represents the learnable knowledge distillation loss, a fine-grained text-image mapping of features, L partial Represents the cosine distance between local image features and textual hints; The model obtained in step 1 is deployed on two 3090Ti GPUs in a data-parallel manner. The weight decay rate of the first stage is set to 0.0001, the image size is unified to 256*128, and the training is iterated for 60 rounds. The distance between the image features of the dynamic prompt and the frozen weight is obtained by forward calculation, and the loss function L is used to calculate the distance between the dynamic prompt and the frozen weight. atage-I Back propagation optimizes model parameters; the weight decay rate in the second stage is set to 0.0001, the image size is unified to 256*128, and the training is iterated for 60 rounds. stage-II Minimize the distance between image features and text features with frozen weights; Step 3: Reasoning stage; In the model inference stage, the input image size is uniformly scaled to 256*128, and the corresponding visual features are extracted only through the trained image encoder. No additional text knowledge is introduced for reasoning. Features are also extracted from images in the gallery, and the visual features of the image to be retrieved are matched with the visual features in the gallery. The features of the top five gallery images that are most similar to the image to be retrieved are taken out. If there is a target image that is the same as the image to be retrieved among the five images, the retrieval is considered successful.

2. The method for lifelong learning of object re-identification based on dynamic text prompts according to claim 1, characterized in that: The steps for building the network framework are: Step 1-1: Construct the input of the dynamic prompt fusion module; Step 1-2: Construct the structure of the dynamic prompt fusion module; Step 1-3: Construct a local fine-grained feature enhancement module; Step 1-4: Build a learnable knowledge distillation module.

3. The method for lifelong learning of target re-identification based on dynamic text prompts according to claim 2, characterized in that: The step 1-1 of constructing the input of the dynamic prompt fusion module is as follows: BLIP is used to generate a target appearance description as a target invariant prompt. Specifically, a series of questions are asked about the appearance features of the target in the BLIP image. That is, a series of target descriptions are obtained by visual question answering and combined to obtain the final target appearance description. Learnable parameters are introduced to constitute target-specific knowledge prompts. PKP is a learnable tensor created using the pytorch language features.

4. The method for lifelong learning of object re-identification based on dynamic text prompts according to claim 3 is characterized in that: The steps 1-2 of constructing the structure of the dynamic prompt fusion module are: A 1*1 convolutional network and ReLU activation function are introduced and stacked in two layers. The 1*1 convolution is used to compress the channel of the feature, and ReLU converts the linear distribution into a nonlinear distribution. The dynamic fusion at the channel level is realized through the collaborative work of the 1*1 convolutional network and the ReLU activation function. The obtained dynamic prompt fusion module dynamically fuses the target invariant prompt and the target specific knowledge prompt in step 1-1 to obtain P DP for: P DP =ReLU(in(ReLU(in(P IP ,P PKP ))) Where P DP Denotes DP, P IP Indicates the target unchanged prompt, P PKP represents the target-specific knowledge hint, w represents the convolution parameter, ReLU represents the ReLU activation function, and P DP It is fed into the text encoder of CLIP to extract text features; at the same time, the image features related to the target are extracted through the image encoder of CLIP, and the similarity between the image features and the text prompts is minimized using the cosine embedding loss: Where L global Represents the overall loss of the model in one stage, L sup is the cross entropy loss, and F img represents the image features, and Represents the image F img The result obtained after transposing the matrix is P DP The matrix transpose result of .

5. The method for lifelong learning of object re-identification based on dynamic text prompts according to claim 4 is characterized in that: The steps 1-3 of constructing the local fine-grained feature enhancement module are as follows: The image features are evenly divided along the dimension of image height. The image is divided into four different regions from top to bottom, representing the head features, upper body features, lower body features and foot features respectively; After obtaining the local image features corresponding to the human body parts, the text features are also segmented for alignment in the local fine-grained feature enhancement module. The BLIP target appearance description generated in step 1-1 is segmented according to the comma as the segmentation point, and four local text semantics blocks are cut out to align with the local image blocks. Specifically, the position of the comma is recorded and the P obtained in step 1-2 is used as the segmentation point. DP The features are segmented according to the recorded positions to ensure that the local features of the human body parts correspond to the local text features; The formula is: F img =[f head ;f upperbody ;f lowerbody ;f foot ] (1) P DP =[p head ;p upperbody ;p lowerbody ;p foot ] (2) Among them, F img Represents the image features generated by the visual encoder, and F img According to the method described in steps 1-3, the following four local image features of human body parts are obtained: head Represents the image features of the human head, f upperbody Represents the image features of the upper body, f lowerbody Represents the image features of the lower body, f foot represents the image features of human feet, p head Represents the local text semantic features of the human head, p upperbody Represents the text semantic features of the upper body, p lowerbody The text semantic features representing the lower body of the human body, p foot Text semantic features representing human feet. Reduce the cosine distance between the target local image features and the local text hints, and enhance the fine-grained text-image mapping based on local features. The fine-grained text-image mapping of the features is described as: where p i Represents the text features of a certain part, f i Represents the image feature of a certain part, L partial Represents the cosine distance between local image features and text prompts, N is the number of body parts; it promotes a more fine-grained mapping of local text features and local image features, and promotes fine-grained transfer of knowledge in the spatial dimension, |*|2 represents the L2 norm.

6. The method for lifelong learning of object re-identification based on dynamic text prompts according to claim 5, characterized in that: The steps 1-4 to construct a learnable knowledge distillation module are: Taking the feature output of the image encoder as input, in order to achieve an adaptive balance between preserving existing knowledge and learning new knowledge, two learnable parameters δ1 and δ2 are introduced, that is, the learnable values ​​of the two learnable parameters optimized by back propagation, which are used to control the temperature coefficient of the knowledge distillation process, expressed as: Where L LKD represents the learnable knowledge distillation loss, δ1 represents the learnable temperature coefficient of the student model, and δ2 represents the learnable temperature coefficient of the teacher network.

7. An electronic device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Unsupervised domain adaptive pedestrian re-identification method based on dictionary learning intermediate domain

    CN117765464A

  • Model training method and device, pedestrian re-identification method and device and computer storage medium

    CN118397649A

  • Generalized pedestrian re-identification method based on visual language multi-granularity distillation

    CN118506267A

  • Partial prompt learning-based lifelong target re-identification method

    CN118864825A

Cited By

  • Federal incremental learning method and system based on prompt

    CN120725098A