Pedestrian re-identification data enhancement method based on image-text modal partition interpolation
Through the data enhancement method of graphic and text modal partition interpolation, the problem of overfitting in pedestrian re-identification technology is solved, the generalization ability and recognition accuracy of the model are improved, the diversity of training samples is enhanced, and the better Cross-domain adaptability and recognition effects.
Patent Information
- Application Number
- CN202411009580.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-22
AI Technical Summary
Existing pedestrian re-identification technology can easily lead to overfitting and insufficient model generalization ability during training, especially due to excessive dependence on specific data patterns, resulting in a decrease in recognition accuracy on unseen data.
Using a data enhancement method based on graphic and text modal partition interpolation, the image parts and attribute words are mapped through semantic dictionary, and local linear interpolation and vocabulary replacement are performed on this basis to generate enhanced image-text pairs and combine external data enhancement. technology to enhance the diversity of training samples and generalization capabilities of models.
Improve the generalization ability and cross-domain adaptability of the model, enhance the accuracy of pedestrian recognition in text retrieval, alleviate the problem of unadaptation of multimodal data in graphics and text, avoid the cost of manually labeling data sets, and increase the The robustness and feature representation of the model.
Smart Images

Figure CN119007242A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of retrieval of text person re-identification, and more specifically, to a method for enhancing person re-identification data based on image-text modality partition interpolation. Background Art
[0002] With the development of computer vision and artificial intelligence technology, person re-identification from text images has become one of the important research topics in the field of computer vision. The goal of the person re-identification task from text images is to accurately identify specific pedestrians by matching natural language descriptions (such as descriptions of pedestrians' clothing and posture) with pedestrian images captured by surveillance cameras. This task has important application value in multiple scenarios, such as video surveillance, public safety, and intelligent transportation. Through text descriptions, we can more flexibly query and identify pedestrians in images, especially when the facial features of pedestrians are blurred, blocked, or the camera angle is not good. Text descriptions can serve as powerful supplementary information to improve the accuracy and efficiency of pedestrian recognition. Due to factors such as lighting, occlusion, background changes, and changing pedestrian postures, the representation of the same person in different images is significantly different, which brings great difficulties to matching and recognition. In addition, the text in the training samples of the person re-identification model from text images needs to be manually annotated, which results in limited acquisition of training samples. Therefore, how to obtain more abundant and diverse training samples to improve the robustness of the model is an important goal to promote the advancement of person re-identification technology.
[0003] In terms of images, simple data enhancement techniques used in the early days include transformation-based methods such as cropping, rotation, and color jittering. These methods can generate diverse samples to increase the size of the data set. Subsequently, generative data enhancement technology based on GAN was introduced, which can generate new high-quality samples after learning the distribution of the data set. However, it is a challenge to use existing data enhancement methods for multimodal learning because these methods need to maintain the correlation between the generated image-text pairs used as enhanced data. In CN202311052722.7, a pedestrian re-identification method based on multimodal images and texts is used to extract features from images and texts. In addition, a multimodal feature fusion network is constructed to align and fuse features of different modalities, thereby solving the problem that features of different modalities are difficult to fuse. Finally, the fused feature vector and the feature vector of the pedestrian to be detected are used as the loss function to achieve multimodal pedestrian re-identification of images and texts. Although this method can solve the problem caused by the variability of text descriptions, from the perspective of the dataset, this strategy of excessive pursuit of precise alignment may sometimes cause the model to rely too much on specific data patterns during training, leading to overfitting problems and limiting the model's generalization ability on unseen data. Summary of the invention
[0004] The main technical problem to be solved by the present invention is to provide a pedestrian re-identification data enhancement method based on image-text modality partition interpolation in view of the deficiency of the prior art that relies on specific parameters during the training process leading to overfitting.
[0005] The purpose of the present invention is achieved through the following technical solutions:
[0006] A method for enhancing person re-identification data based on image-text modality partition interpolation, comprising the following steps:
[0007] S1. Obtain the image-text pairs in the dataset and use the partition difference method to generate enhanced data sample pairs P au {I au ,T au}:
[0008] P au =PaSeMix(P tg ,P sr )
[0009] Where PaSeMix represents the image and text data enhancement algorithm;
[0010] S2. Extract visual and text features from image data and text data;
[0011] S3. Perform feature alignment at the global and local levels respectively, and then train the model;
[0012] S4. The enhanced data sample pairs and the original data are used as training data to input into the person re-identification model for training.
[0013] Furthermore, the steps of the image and text data enhancement algorithm include:
[0014] S11. Given an image text {I sr ,T sr} is defined as the target data, {I tg ,T tg} is defined as source data;
[0015] S12. According to T tg Get the chunked vocabulary word from the word m , and query the semantic dictionary to get the corresponding area part k ;
[0016] S13. I sr part k Partial linear interpolation to I sr The corresponding part k Part, get the enhanced image sample I au :
[0017]
[0018] S14. k Mapping T in the semantic dictionary sr Chunk words at the same position n , and use word n Replace T tg word m , get the enhanced text sample T au :
[0019] T au =Replace(T tg [word m ],T sr [word n ])
[0020] Get the enhanced data sample pair P au {I au ,T au}:
[0021] P au =PaSeMix(P tg ,P sr )
[0022] Among them, I and T represent the matched image and text respectively. au and T au Together they form the enhanced sample P au , operator Indicates that the block-enhanced samples are inserted into the original target samples, λ m is the coefficient of linear interpolation, and the Replace(*) operation indicates T tg Chinese word n Partially replaced by T sr Chinese word m .
[0023] Furthermore, in visual and text feature extraction, source data P sr 、Target data P tg and enhanced data P au as data input.
[0024] Furthermore, the step of extracting visual features includes:
[0025] The image data is extracted using a visual encoder to obtain the global visual feature F, and then the global visual feature F is evenly divided into K non-overlapping parts F k ∈R C×H′×W (1≤k≤K), from F k Extract local visual features.
[0026] Furthermore, the text feature extraction step includes:
[0027] For a given language where q t Represents the tth word in the sentence, and each word q t Mapped to its corresponding word embedding, and then each corresponding word and its position are input into the language encoder to obtain text features
[0028] Furthermore, the global feature alignment step includes:
[0029] Perform global maximum pooling on the global visual feature F and the text feature E, and obtain the following through the convolution layer:
[0030] v g =W g GMP(F)
[0031] t g =W g RMP(E)
[0032] Where W g ∈R M×C , v g ,t g ∈R M , and represents global visual and textual features.
[0033] Furthermore, the similarity of global features is expressed as:
[0034]
[0035] Furthermore, the local feature alignment step includes:
[0036] S321. Predict the probability that the i-th word belongs to the k-th part:
[0037]
[0038] in represents probability, σ represents Sigmoid function;
[0039] S322. Modify the text feature E to represent the text description of the kth part, as follows:
[0040]
[0041] S323. F k The visual features of the kth part are obtained by global maximum pooling and convolution. k Max pooling is performed to generate the kth part-level text features and fed into the samek The same convolutional layer obtains the visual local features and text local features, which are expressed as:
[0042]
[0043] Furthermore, the similarity of local features is expressed as:
[0044]
[0045] Furthermore, the total loss of the person re-identification model is expressed as:
[0046]
[0047] represents the label loss (ID loss), represents the composite ranking loss (CR Loss), Used to control the impact parameter of the loss term.
[0048] Compared with the prior art, the beneficial effects are:
[0049] The present invention uses a semantic dictionary to map the image part and the attribute word, and then performs local linear interpolation and local vocabulary replacement on the image part and the local vocabulary respectively under this relationship to generate an enhanced image-text pair, and then the enhanced sample and the original sample participate in the training of the model together. The method of the present invention generates local enhanced samples according to the partition semantics to enhance the generalization of the model and adapt to the learning of different features. Specifically, based on the characteristics of text pedestrian re-identification data, we no longer rely solely on the alignment mechanism inside the model, but combine external data enhancement technology, use the image and text modality to establish a connection to partition interpolation to generate diversified training samples. These samples not only retain the core features of the original data, but also incorporate new change elements, which helps the model learn more robust and generalized feature representations, enrich the number of training samples, expand the data distribution, enhance the generalization ability of the model, and fuse complementary information to the model, which is conducive to the improvement of the accuracy of text retrieval pedestrian re-identification tasks, and has competitiveness in cross-domain generalization ability. Through the input-level data enhancement method of partitioning and interpolating the two modules of image and text, the present invention can achieve effective fusion of features without changing these basic components. The method of the present invention improves the retrieval accuracy, base model method capability and cross-domain generalization capability of the text retrieval person re-identification model, alleviates the incompatibility problem of image and text multimodal data enhancement, and also enhances the diversity of samples, avoiding the cost of manually annotating data sets. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 It is a flow chart of the method of the present invention.
[0051] Figure 2 It is a data enhancement method for image and text partition interpolation.
[0052] Figure 3 It is a visual display to alleviate the overfitting problem.
[0053] Figure 4 It is the performance of different mixing parameters. DETAILED DESCRIPTION
[0054] The present invention will be further explained and illustrated below in conjunction with the embodiments, but the specific embodiments do not limit the present invention in any form.
[0055] Example 1
[0056] This embodiment provides a method for enhancing pedestrian re-identification data based on image-text modality partition interpolation, the steps comprising:
[0057] S1. Generate image and text input level enhanced data;
[0058] S11. Given an image-text pair and Where I and T represent the matching image and text respectively. For any i,j∈{1,...,N} and i≠j, N represents the total amount of data in the dataset. sr ,T sr} is defined as the target data, {I tg ,T tg} represents the source data.
[0059] S12. According to T tg Randomly get a block vocabulary word from the word m , query the semantic dictionary to get the corresponding area part k The semantic dictionary is pre-trained using the PEDE-CUHK dataset. Based on named entity recognition (NER), it can identify common attributes and clothing, and construct a semantic dictionary with word-area-part mapping. The structure is:
[0060] word area part glasses head [1] t-shirt upper body [2,3] jeans lower body [4,5] sneakers foot [6] … … …
[0061] S13. I sr part k Partial linear interpolation to I sr The corresponding part k Part, get the enhanced image sample I au .
[0062] S14. Obtained part k Mapping T in the semantic dictionary sr Chunk words at the same positionn , and use word n Replace T tg word m , get the enhanced text sample T au ;
[0063] S15. Integrate the source data into the target data to synthesize enhanced data with target labels, that is, enhance the image text pair P au {I au ,T au}, expressed as:
[0064] P au =PaSeMix(P tg ,P sr )
[0065]
[0066] T au =Replace(T tg [word m ],T sr [word n ])
[0067] PaSeMix represents the image and text data enhancement algorithm, operator Indicates that the block-enhanced samples are inserted into the original target samples, and the coefficient of linear interpolation is λ m , the Replace(*) operation represents T tg Chinese word n Partially replaced by T sr Chinese word m .
[0068] S2. Extract visual and text features;
[0069] S21. The source data P sr 、Target data P tg and enhanced data P au As data input, visual and text feature extraction is performed.
[0070] S22. Pedestrian image I∈R W×H×3 The image is resized to 3×384×128, and then the visual encoder of the ResNet-50 model is used to extract the global visual features F.
[0071] S23. First, divide the global visual feature F evenly into K non-overlapping parts F k ∈R C×H′×W (1≤k≤K), then from F k Extract local visual features.
[0072] S24. Given language where q t Represents the tth word in the sentence, and each word q t Mapped to its corresponding word embedding, and then each q t The position of the machine index is input into the language encoder to obtain text features
[0073] S25. Map the text features to the same dimension as the image d by normalization.
[0074] S3. Global and local feature alignment;
[0075] S31. Global feature alignment;
[0076] S311. Perform global maximum pooling (GMP) on the image feature F, perform maximum pooling (RMP) on the text feature E, and then pass through a shared 1×1 convolutional layer W g ,get:
[0077] v g =W g GMP(F)
[0078] t g =W g RMP(E)
[0079] Where W g ∈R M×C , v g ,t g ∈R M , and represents global visual and textual features.
[0080] S312. The obtained v g ,t g The features are projected into a common feature space, where the parameters are shared by the 1×1 convolutional layer W g Encourage F and E to be closer together in terms of semantics. The similarity between the global features of the resulting image-text pair is expressed as follows:
[0081]
[0082] S32. Local feature alignment;
[0083] S321. Predict the probability that the i-th word belongs to the k-th part:
[0084]
[0085] in represents probability, and σ represents the Sigmoid function.
[0086] S322. Construct a matrix by embedding the words of all unique words in the training set 1×C represents the word embedding dimension and the number of unique words.
[0087] S323. Modify E to indicate that the text of Part K is expressed as follows:
[0088]
[0089] S324.F k Input to a GMP layer and a 1×1Conv layer to obtain the visual features of the kth part. k RMP is performed to generate the k-th part-level text features and fed into the same k The same 1×1 Conv layer is represented as:
[0090]
[0091] in and represents the parameters of the shared 1×1 convolutional layer. and represent the visual and textual features of the kth part, respectively.
[0092] The similarity between the partial-level features of a picture-text pair is expressed as:
[0093]
[0094] where v l and t l ∈R KM .
[0095] S4. The enhanced image-text pairs and the original data are input as training data into the classic dual-tower structure pedestrian re-identification model SSAN (Suo, W. et al. 2021) and SRCF (Ding, Z. et al. 2022). The label loss (IDloss) and composite ranking loss (CR Loss) are used as training loss functions to train the model to the best accuracy.
[0096] The total loss of the entire model is expressed as:
[0097]
[0098] in represents the label loss (ID loss), represents the composite ranking loss (CR Loss), Used to control the impact of the loss term.
[0099] Example 2
[0100] This embodiment provides a test experiment. This implementation uses the large-scale text retrieval pedestrian re-identification datasets CUHK-PEDES and ICFG-PEDES, and the evaluation index uses the Rank-k accuracy (Acc top-k ) and mean average precision (mAP). The calculation method is:
[0101]
[0102] Among them, k represents the first k search results, N k Represents the number of correct matches in the first k search results.
[0103]
[0104] Where N represents the total number of samples, TP represents the number of correctly predicted samples, FP represents the number of false positive samples, and FN represents the number of false negative samples.
[0105] As can be seen from Table 1, based on the PEDE-CUHK dataset and the PEDE-ICFG dataset, the use of this patented method has improved on a variety of classic text person re-identification models. In addition, the mAP results based on the PEDE-CUHK dataset in SSAN and SRCF are 41.5% and 43.6% respectively; the mAP results based on the PEDE-ICFG dataset in SSAN and SRCF are 37.73% and 40.13% respectively. Based on the two commonly used datasets, the text retrieval person re-identification task has improved Rank-1, Rank-5, Rank-10 and mAP.
[0106] Table 1
[0107]
[0108] *MG stands for MixGen method; PS stands for PaSeMix method proposed in this patent
[0109] Example 3
[0110] This embodiment provides a test experiment for cross-domain tasks. A summer dataset D is constructed from PEDE-CUHK and PEDE-ICFG according to the seasonality of clothing. s and winter D w As shown in Table 2, in cross-domain retrieval D w →D s The Rank-1, Rank-5, and Rank-10 are 30.98%, 51.64%, and 59.32% respectively, which are significantly improved compared to the model without data enhancement, and have certain improvements compared to the non-partitioned data enhancement method MixGen.
[0111] Table 2
[0112] Train Data Test Data Top-1 Top-5 Top-10 Dw(a) Dw 56.98 74.64 81.24 Dw(a)+Ds(n) Dw 55.24 73.69 80.4 Dw(a) Ds 23.03 41.62 51.32 Dw(a)+Ds(n) Ds 22.36 37.59 47.57 Dw(a)+MixGen(Ds,Dw)(n) Ds 26.49 46.72 57.42 Dw(a)+PaSeMix(Ds,Dw)(n) Ds 30.98 51.64 59.32 Dw(a)+MixGen(Ds,Dw)(n) Dw 57.24 77.63 88.3 Dw(a)+PaSeMix(Ds,Dw)(n) Dw 58.36 77.98 88.13
[0113] *n and a represent the sample size (n = 1 / 4a)
[0114] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.
Claims
1. A pedestrian re-identification data enhancement method based on image-text modality partition interpolation, characterized in that the steps include: S1. Obtain the image-text pairs in the dataset and use the partition difference method to generate enhanced data sample pairs P au {I au ,T au }: P.S au (PaSeMix(P tg ,P sr ) Among them, PaSeMix represents the image and text data enhancement algorithm; S2. Extract visual and text features from image data and text data; S3. Perform feature alignment at the global and local levels respectively, and then train the model; S4. The enhanced data sample pairs and the original data are used as training data to input into the person re-identification model for training.
2. According to claim 1, the method for enhancing pedestrian re-identification data based on image-text modality partition interpolation is characterized in that: The steps of the image and text data enhancement algorithm include: S11. Given an image text {I sr ,T sr } is defined as the target data, {I tg ,T tg } is defined as source data; S12. According to T tg Get the chunked vocabulary word from the word m , and query the semantic dictionary to get the corresponding area part k ; S13. I sr part k Partial linear interpolation to I sr The corresponding part k Part, get the enhanced image sample I au : S14. k Mapping T in the semantic dictionary sr Chunk words at the same position n , and use word n Replace T tg word m , get the enhanced text sample T au : T au =Replace(T tg [word m ],T sr [word n ]) Get the enhanced data sample pair P au {I au ,T au }: P.S au (PaSeMix(P tg ,P sr ) Among them, I and T represent the matched image and text respectively. au and T au Together they form the enhanced sample P au , operator Indicates that the block-enhanced samples are inserted into the original target samples, λ m is the coefficient of linear interpolation, and the Replace(*) operation indicates T tg Middle work n Partially replaced by T sr Middle work m .
3. According to claim 1, the method for enhancing pedestrian re-identification data based on image-text modality partition interpolation is characterized in that: Source data P in visual and text feature extraction sr 、Target data P tg and enhanced data P au as data input.
4. According to claim 1, the method for enhancing pedestrian re-identification data based on image-text modality partition interpolation is characterized in that: The steps of visual feature extraction include: The image data is extracted using a visual encoder to obtain the global visual feature F, and then the global visual feature F is evenly divided into K non-overlapping parts F k ∈R C×H′×w (1≤k≤K), from F k Extract local visual features.
5. According to claim 1, the method for enhancing pedestrian re-identification data based on image-text modality partition interpolation is characterized in that: The steps of extracting text features include: For a given language where q t Represents the tth word in the sentence, and each word q t Mapped to its corresponding word embedding, and then each corresponding word and its position are input into the language encoder to obtain text features 6. According to claim 1, the method for enhancing pedestrian re-identification data based on image-text modality partition interpolation is characterized in that: The global feature alignment steps include: Perform global maximum pooling on the global visual feature F and the text feature E, and obtain the following through the convolution layer: v g =W g GMP(F) t g =W g RMP(E) Where W g ∈R M×C , v g ,t g ∈R M , and represents global visual and textual features.
7. The method for enhancing person re-identification data based on image-text modality partition interpolation according to claim 6, characterized in that: The similarity of global features is expressed as:
8. The method for enhancing person re-identification data based on image-text modality partition interpolation according to claim 1, characterized in that: The local feature alignment steps include: S321. Predict the probability that the i-th word belongs to the k-th part: in represents probability, σ represents Sigmoid function; S322. Modify the text feature E to represent the text description of the kth part, as follows: S323. F k The visual features of the kth part are obtained by global maximum pooling and convolution. k Max pooling is performed to generate the kth part-level text features and fed into the same k The same convolutional layer obtains the visual local features and text local features, which are expressed as:
9. The method for enhancing pedestrian re-identification data based on image-text modality partition interpolation according to claim 8, characterized in that: The similarity of local features is expressed as:
10. The method for enhancing person re-identification data based on image-text modality partition interpolation according to claim 4, characterized in that: The total loss of the pedestrian re-identification model is expressed as: represents the label loss (ID loss), represents the composite ranking loss (CR Loss), Used to control the impact parameter of the loss term.
Citation Information
Cited By
Pedestrian attribute recognition method based on attribute semantic similarity matching
CN119964204A
