Personalized appearance modeling-based pedestrian re-identification method
By introducing double constraints and multi-view multi-template generalized expressions of posture nearest neighbors and textures in pedestrian re-identification technology, the problem of appearance differences caused by changes in pedestrian poses and perspectives is solved, and the recognition accuracy and robustness are improved.
Patent Information
- Application Number
- CN202510209175.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-25
- Publication Date
- 2025-06-27
AI Technical Summary
Existing pedestrian re-identification technology is difficult to effectively deal with appearance differences caused by changes in pedestrian posture and perspective in an unconstrained environment, resulting in insufficient recognition accuracy and robustness.
Through double constraints on the pose nearest neighbors and textures, interfering images with large pose gaps but similar visual appearance are eliminated, and through the generalized expression of multi-view angles and multi-templates, the single image retrieval task is converted into multi-view angle images to participate in the retrieval, improving the robustness of query operations.
It significantly improves the performance of pedestrian re-identification in complex scenarios, reduces pedestrian identity confusion with similar appearances, and enhances the ability to adapt to perspective changes.
Smart Images
Figure CN120220179A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical fields of deep learning and computer vision, and particularly relates to a person re-identification method based on personalized appearance modeling. Background Art
[0002] Person re-identification is a technology that uses computer vision and machine learning techniques to retrieve pedestrian images and videos captured by different monitoring devices, mainly solving the problems of pedestrian recognition and retrieval across cameras and scenes. The technology aims to identify pedestrian identities based on human characteristics such as the clothing, posture, and hairstyle of pedestrians. After decades of research, the efficiency and accuracy of person re-identification have been significantly improved. However, in an unconstrained environment, the performance of person re-identification is still easily affected by many factors. For example, changes in equipment, lighting, scale, occlusion, posture, etc. during the data collection process will affect the image quality to varying degrees. There are large intra-class differences in pedestrian images, resulting in unstable person retrieval and sensitivity to outliers. Considering the movement of pedestrians and the changes in the camera shooting angle, pose / viewpoint change is one of the common phenomena in an unconstrained environment, which significantly changes a person's appearance and reduces the robustness of visual representation. Due to the large intra-class differences, it is challenging to learn discriminative features to identify people from a large-scale gallery.
[0003] Regarding the problem that pedestrian images taken from different viewpoints may have huge visual differences, existing technologies include pose-based feature alignment methods, multi-view modeling methods, and local feature matching, etc.
[0004] The literature
Zhihui Z, Jiang X, Zheng F, et al. Viewpoint-aware loss with angular regularization for person re-identification. AAAI 34, 13114–13121 (2020) [EB / OL]
[0005] The literature [Bhuiyan A, Liu Y, Siva P, et al. Pose guided gated fusion for person re-identification[C] / / Proceedings of the IEEE / CVF winter conference on applications of computer vision. 2020:2675-2684.] proposed a pose-guided gated fusion framework to address the feature alignment problem caused by view and pose changes in person re-identification. The pose-guided gated fusion framework dynamically selects and adjusts appearance features using the pose features of pedestrians. The pose network generates the pose confidence map and part association field of the human body. The gated network dynamically selects the feature filters of the backbone network based on these pose features to enhance the local regions of the input image, making the model pay more attention to the pose-related regions. However, in real-world scenarios, the view change of pedestrians is often continuous (such as gradually changing from the front to the side). This method mainly performs feature fusion on single-frame images and does not consider the sequence information of view changes, which may lead to a decline in the performance of the model in cross-view dynamic matching.
[0006] Most of the existing studies on re-ranking methods rely on texture similarity constraints and use a single image as the retrieval benchmark. Under the condition of variable poses, these methods cannot effectively distinguish pedestrians with highly similar appearances.
[0007] The literature [Zhong Z, Zheng L, Cao D, et al. Re-ranking person re-identification with k-reciprocal encoding[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017:1318-1327] proposed an unsupervised re-ranking method based on k-Reciprocal nearest neighbor encoding. By encoding the k-Reciprocal nearest neighbor set as a vector and calculating the similarity using the Jaccard distance, the initial ranking is improved. The final distance is a weighted combination of the original distance and the Jaccard distance, further enhancing the robustness of the ranking. This method mainly relies on appearance features and neighbor relationships and does not introduce means of view modeling (such as geometric transformation or view-based feature alignment methods), which limits its performance in the case of large view changes.
[0008] The literature
Zhang Y, Qian Q, Wang H, et al. Graph convolution based efficient re-ranking for visual retrieval[J]. IEEE Transactions on Multimedia, 2023, 26: 1089-1101
[0009] The object of the present invention is to provide a pedestrian re-identification method based on personalized appearance modeling to solve the above technical problems.
[0010] To solve the technical problem that it is difficult to identify pedestrians with large perspective differences and highly similar appearances in existing pedestrian re-identification, the present invention excludes interfering images with large pose gaps but similar visual appearances through double constraints of pose neighbor and texture similarity, and overcomes the aliasing difficulties brought by simple texture similarity constraints; through multi-view multi-template generalized representation, the single-image retrieval task is transformed into a retrieval jointly participated by multi-view images, thereby greatly increasing the robustness of the query operation. The specific technical solution of the pedestrian re-identification method based on personalized appearance modeling of the present invention is as follows:
[0011] A pedestrian re-identification method based on personalized appearance modeling, comprising the following steps:
[0012] Step 1: Collect outdoor activity images of pedestrian objects, use a feature extractor to obtain the feature vectors of pedestrian images, and construct a similarity matrix by calculating the distances between the feature vectors of pedestrian images;
[0013] Step 2: Train a pose estimation model to estimate the camera perspective parameters of pedestrians in the images, with the input of the model being the pedestrian images and the output being the camera perspectives of pedestrians;
[0014] Step 3: According to the perspective information of pedestrian images, divide the images into multiple perspective categories, and then establish a personalized multi-view appearance feature model of pedestrians through the texture information of pedestrian images and the relationship of perspective neighbors;
[0015] Step 4: The secondary search is performed based on the pedestrian personalized appearance expression obtained in Step 3. For the remaining images to be retrieved in the image gallery, image sequence re-ranking is carried out.
[0016] Further, the said Step 1 includes the following specific steps:
[0017] Images of pedestrians during outdoor activities are collected through a camera array, and pedestrian IDs, camera numbers, and image sequence number tags are marked to form a data set. The data set is divided into a training set and a test set. A feature extractor f(·) in the source domain is established using the pre-trained ResNet50 network. The pre-processed images are input, and 2048-dimensional feature vectors are output; let the query image feature matrix be Q ∈ R m×d , where m is the number of query images and d is the dimension of the feature vector. Let the image gallery image feature matrix be G ∈ R n ×d , where n is the number of images in the image gallery. Each feature vector v is normalized:
[0018]
[0019] where v is the original feature vector and v ′ is the normalized feature vector. Calculate the similarity matrix S Q between the query image and the images in the image gallery, and calculate the similarity matrix S G of the images in the image gallery with themselves.
[0020] Further, the said Step 2 includes the following specific steps:
[0021] Use the pre-trained key point detection model to obtain the 2D coordinates of the human skeleton as the prior information for body direction estimation. The direction estimation module combines the human key point information and the regional features of the human body, and uses the regression model E(·) to estimate the 3D direction angle.
[0022] Further, the said Step 3 includes the following specific steps:
[0023] Step 3.1: Design a dual-constraint method based on view proximity and texture similarity for retrieving neighboring images; given a query image I q , the goal is to construct a set C = {c0, c1, c2,..., c M} of pedestrian P's personalized appearance expressions, where c0 = I q represents the query image, and c1, c2,..., c M are images with different views from the query image. The process of constructing the personalized appearance expression follows the following two conditions: 1) Similarity constraint: For any image c m in the set C, if the image gallery image I g is in the image cm ranked top - 1 in the similarity ranking, and I g has a similarity with the query image I q greater than the threshold η d , then the similarity constraint is satisfied; 2) Viewpoint constraint: If the viewpoint difference between the gallery image I g and the image c m is less than the threshold η θ , then the viewpoint constraint is satisfied. The following example will illustrate the construction process of the personalized appearance expression. At the t - th retrieval, given the personalized appearance expression set C t-1 , the similarity matrix S and the similarity distance D are calculated by the following formulas:
[0024]
[0025] The candidate set C cand can be calculated by the following formula:
[0026]
[0027] To determine whether the images in the candidate set C cand should be added to the set C, the following rules need to be checked:
[0028] C t ←I g
[0029] s.t.E(c m ) - E(I g ) ≤ η θ ∧D(f(c0), f(IG)) ≥ η d
[0030] where c m ∈C t-1 , I g ∈C cand . The first condition ensures that the selected image has the most similar features to any image in the set C, and the selected image has a high similarity with the reference image c0 in the set C. The threshold η d is used to constrain the similarity. The second condition limits the viewpoint difference of the candidate image to maintain a certain similarity in viewpoint with the image c m in the set C. The threshold η θ is used to control the viewpoint gap.
[0031] Step 3.2: Construction of multi-view appearance representation Starting from each query image, repeat the operation of Step 3.1 in an iterative manner. In each round of iteration, at most one eligible image is added to set C. When the number of images in set C exceeds a preset threshold, or the number of iterations reaches the limit, the first image detection process ends, and the personalized appearance representation of the pedestrian is obtained. During this process, images from different perspectives jointly constitute the generalized feature representation of the pedestrian's appearance.
[0032] Further, in Step 4, two methods, similarity nearest neighbor and credible distance, are used for image sequence re-ranking;
[0033] Method 1: According to the similarity matrix S G Retrieve the images in the gallery that rank high in similarity to the images in set C, and add these images to the ranking list of query image I q ;
[0034] Method 2: Calculate the credible distance from the gallery image I g to set C, and sort the gallery images according to the credible distance U.
[0035] Further, the calculation formula for the credible distance of Method 2 is as follows:
[0036]
[0037] The specific steps are as follows:
[0038] Step 4.1: Calculate the cosine similarity distance D between the candidate image I g and the image c m in set C;
[0039] Step 4.2: Calculate the Euclidean distance of the image pair [I g , c m , and perform normalization processing to obtain the weight W;
[0040] Step 4.3: Use the weight W to perform weighted summation on the distance D to obtain the credible distance U from image I g to set C.
[0041] A pedestrian re-identification method based on personalized appearance modeling of the present invention has the following advantages:
[0042] In view of the problem of appearance differences caused by the changes in pedestrian postures in natural monitoring scenarios, the present invention first verifies that there is a consistency relationship between perspective neighbors and texture similarity. Within a certain perspective range, there is an inverse relationship between the similarity distance and the perspective difference, that is, the larger the perspective difference, the smaller the similarity. Then, based on the collaborative constraints of pedestrian posture neighbors and texture similarity, a multi-perspective appearance feature expression of pedestrian objects is established. Finally, sorting optimization is performed according to the similarity nearest neighbor and the credible distance, improving the performance of pedestrian re-identification in complex scenarios.
[0043] Compared with the re-ranking techniques in existing pedestrian re-identification, the advantages of the present invention include:
[0044] In response to the challenges brought by individual posture differences in the pedestrian re-identification task, the present invention innovatively proposes a method for personalized appearance modeling of pedestrians.
[0045] The present invention realizes the collaborative constraints of two elements, namely pedestrian perspective neighbors and texture similarity, excludes interference images with large perspective gaps but similar visual appearances, and reduces the identity confusion of pedestrians with similar appearances.
[0046] The present invention makes full use of the posture information in the image to establish a multi-perspective and multi-template generalized expression of pedestrian objects, effectively combines multi-perspective images to jointly participate in the retrieval of neighboring images, optimizes the ranking of neighboring images, and significantly improves the performance of the re-ranking method. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a schematic diagram of the overall process flow of the method of the present invention;
[0048] Figure 2 It is a schematic diagram of the specific process flow of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION
[0049] In order to better understand the purpose, structure and function of the present invention, the pedestrian re-identification method based on personalized appearance modeling of the present invention will be further described in detail below with reference to the accompanying drawings.
[0050] Please refer to Figure 1 、 Figure 2 The pedestrian re-identification method based on personalized appearance modeling provided by the present invention includes the following steps:
[0051] Step 1: Collect outdoor activity images of pedestrian objects, use a feature extractor to obtain the feature vectors of pedestrian images, and construct a similarity matrix by calculating the distances (such as cosine similarity, Euclidean distance, etc.) between the feature vectors of pedestrian images.
[0052] Collect images of pedestrians during outdoor activities through a camera array, label tags such as pedestrian ID, camera number, image serial number, etc. to form a dataset, and divide the dataset into a training set and a test set. Use the pre-trained ResNet50 network to establish a feature extractor f(·) in the source domain. Input the pre-processed images and output 2048-dimensional feature vectors. Let the query image feature matrix be Q ∈ R m×d , where m is the number of query images and d is the dimension of the feature vector. Let the gallery image feature matrix be G ∈ R n×d , where n is the number of gallery images. Normalize each feature vector v:
[0053]
[0054] where v is the original feature vector and v ‘ is the normalized feature vector. Calculate the similarity matrix S Q of the query image set and the gallery image set, and calculate the similarity matrix S G of the gallery image set with itself.
[0055] Step 2: Train a pose estimation model to estimate the camera view parameters of pedestrians in the image. The input of the model is the pedestrian image, and the output is the camera view of the pedestrian.
[0056] Use the pre-trained key point detection model to obtain the 2D coordinates of the human skeleton as prior information for body direction estimation. The direction estimation module combines the human key point information and the regional features of the human body, and uses the regression model E(·) to estimate the 3D direction angle.
[0057] Step 3: According to the view information of the pedestrian image, divide the image into multiple view categories. Then, establish a personalized pedestrian multi-view appearance feature model through the texture information of the pedestrian image and the relationship of view neighbors.
[0058] Step 3.1: The present invention designs a method based on dual constraints of view neighbor and texture similarity for retrieving neighbor images. In this embodiment, given a query image I q , the goal is to construct a personalized appearance expression set C = {c0, c1, c2,..., c M} of pedestrian P, where c0 = I q represents the query image, and c1, c2,..., c M are images with different views from the query image. The process of constructing the personalized appearance expression follows the following two conditions: 1) Similarity constraint: For any image c m in the set C, if the gallery image I g ranks top-1 in the similarity ranking of the image c m , and I ghas a similarity greater than the threshold η with the query image I q , then the similarity constraint is satisfied; 2) Viewpoint constraint: If the viewpoint difference between the gallery image I d and the image c g is less than the threshold η m , then the viewpoint constraint is satisfied. The following example will illustrate the construction process of the personalized appearance expression. At the t-th retrieval, given the personalized appearance expression set C θ , the similarity matrix S and the similarity distance D are calculated by the following formulas: t-1
[0059]
[0060] The candidate set C cand can be calculated by the following formula:
[0061]
[0062] To determine whether the images in the candidate set C cand should be added to the set C, the following rules need to be checked:
[0063] C t ← I g
[0064] s.t. E(c m ) - E(I g ) ≤ η θ ∧ D(f(c0), f(I g )) ≥ η d
[0065] where c m ∈ C t-1 , I g ∈ C cand . The first condition ensures that the selected image has the most similar features to any image in the set C, and the selected image has a high similarity with the reference image c0 in the set C. The threshold η d is used to constrain the similarity. The second condition limits the viewpoint difference of the candidate images, making them have a certain similarity in viewpoint with the image c m in the set C. The threshold η θ is used to control the viewpoint gap.
[0066] Step 3.2: Construction of multi-view appearance expression Starting from each query image, the operation of step 3.1 is repeated in an iterative manner. In each iteration, at most one qualified image is added to set C. When the number of images in set C exceeds the preset threshold, or the number of iterations reaches the limit, the first image detection process ends and the personalized appearance expression of the pedestrian is obtained. In this process, images from different perspectives together constitute the generalized feature representation of the pedestrian's appearance.
[0067] Step 4: A secondary search is performed based on the personalized appearance expression of the pedestrian obtained in step 3. For the remaining images to be retrieved in the gallery, the present invention uses two methods, namely, the nearest neighbor similarity method and the credible distance method, to re-rank the image sequence.
[0068] Method 1: Based on the similarity matrix S G Retrieve the top image pairs in the gallery that are similar to the images in set C and add these images to the query image I q 's ranked list.
[0069] Method 2: Calculate the gallery image I g The trusted distance to the set C is used to sort the images in the gallery according to the trusted distance U. The calculation formula of the trusted distance is as follows:
[0070]
[0071] The specific steps are as follows:
[0072] Step 4.1: Calculate candidate image I g and image c in set C m The cosine similarity distance D between them;
[0073] Step 4.2: Calculate the image pair [I g ,c m ] and normalize it to get the weight W;
[0074] Step 4.3: Use weight W to perform weighted summation on distance D to obtain image I g The credible distance U to the set C.
[0075] The present invention uses Rank-1 and mAP indicators for performance evaluation. Rank-n indicates the probability that the top n images in the search results are correct results. mAP indicates the average accuracy of all categories, which is obtained by comprehensive weighted average of the average accuracy AP of all categories.
[0076] The present invention proves that, without considering influencing factors such as background and light transformation, there is a near-neighbor consistency relationship between the viewing angle and similarity. Through the collaborative constraint of the two elements of viewing angle neighborhood and texture similarity, interfering images with large pose gaps but similar visual appearances are excluded, and the aliasing difficulty caused by pure texture similarity constraints is overcome. A personalized appearance model of pedestrians is established, and the retrieval task of a single image is transformed into the retrieval jointly participated by images from multiple viewing angles, greatly increasing the robustness of image retrieval.
[0077] The present invention proposes a pedestrian re-identification method based on personalized appearance modeling. By utilizing the pose differences of different individuals, a personalized appearance feature expression of pedestrian objects is established, and the image ranking is optimized through the collaborative retrieval of multiple viewing angles, realizing more robust pedestrian identity identification.
[0078] Parts not described in detail in this specification belong to the prior art.
[0079] It can be understood that the present invention is described through some embodiments. Those skilled in the art know that, without departing from the spirit and scope of the present invention, various changes or equivalent replacements can be made to these features and embodiments. Additionally, under the teaching of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application belong to the scope protected by the present invention.
Claims
1. A pedestrian re-identification method based on personalized appearance modeling, characterized in that: The following steps are involved: Step 1: Collect outdoor activity images of pedestrians and use the feature extractor to obtain the feature vector of the pedestrian image. By calculating the distance between the feature vectors of pedestrian images, a similarity matrix is constructed; Step 2: Train a pose estimation model to estimate the camera view parameters of pedestrians in the image. The model input is the pedestrian image and outputs the camera view of the pedestrian. Step 3: According to the perspective information of the pedestrian image, the image is divided into multiple perspective categories, and then a personalized pedestrian multi-perspective appearance feature model is established through the texture information of the pedestrian image and the relationship between perspective neighbors; Step 4: A secondary search is performed based on the personalized appearance expression of pedestrians obtained in step 3. For the remaining images to be retrieved in the gallery, the image sequence is re-ranked.
2. The method for person re-identification based on personalized appearance modeling according to claim 1, characterized in that: The step 1 comprises the following specific steps: The camera array is used to collect images of pedestrians in outdoor activities, and the pedestrian ID, camera number, and image sequence number labels are annotated to form a data set. The data set is divided into a training set and a test set. The pre-trained ResNet50 network is used to establish a feature extractor f(·) in the source domain. The pre-processed image is input and a 2048-dimensional feature vector is output. Let the query image feature matrix be Q∈R m×d , where m is the number of query images, d is the dimension of the feature vector, and the feature matrix of the gallery image is G∈R n×d , where n is the number of gallery images, each feature vector v is normalized: Where v is the original feature vector and v′ is the normalized feature vector. Calculate the similarity matrix S between the query image and the gallery image Q , calculate the similarity matrix S between the gallery image and itself G .
3. The pedestrian re-identification method based on personalized appearance modeling according to claim 1, characterized in that: The step 2 comprises the following specific steps: The pre-trained key point detection model is used to obtain the 2D coordinates of the human skeleton as the prior information for body direction estimation. The direction estimation module combines the key point information and the regional features of the human body and uses the regression model E(·) to estimate the 3D direction angle.
4. The method for person re-identification based on personalized appearance modeling according to claim 1, characterized in that: The step 3 comprises the following specific steps: Step 3.1: Design a dual constraint method based on view neighbors and texture similarity to retrieve neighbor images. Given a query image I q The goal is to construct a personalized appearance expression set C = {c0,c1,c2,...,c M }, where c0 = I q represents the query image, and c1,c2,...,c M is an image with a different perspective from the query image. The process of constructing a personalized appearance expression follows the following two conditions:
1. Similarity constraint: For any image c in the set C m , if the gallery image I g In image c m Ranked top-1 in the similarity ranking, and I g With query image I q The similarity is greater than the threshold η d , then the similarity constraint is satisfied; Second, the perspective constraint: If the gallery image I g With image c m The viewing angle difference between them is less than the threshold η θ , then the view constraint is satisfied; at the tth retrieval, the known personalized appearance expression set C t-1 , the similarity matrix S and similarity distance D are calculated by the following formula: Candidate set C cand It can be calculated by the following formula: In order to determine the candidate set C cand Whether the image in should be added to set C, the following rules need to be checked: C t ←I g s.t.E(c m )-E(I g )≤η θ ∧D(f(c0),f(I g ))≥η d Among them, c m ∈C t-1 ,I g ∈C cand The first condition ensures that the selected image has the most similar features to any image in the set C, and the selected image has a high similarity to the reference image c0 in the set C. The threshold η d Used to constrain similarity; the second condition limits the perspective difference of the candidate image to make it similar to the image c in the set C m To maintain a certain similarity in perspective, the threshold η θ Used to control the perspective gap; Step 3.2: Construction of multi-view appearance expression Starting from each query image, the operation of step 3.1 is repeated iteratively. In each round of iteration, at most one qualified image is added to set C; when the number of images in set C exceeds the preset threshold, or the number of iterations reaches the limit, the first image detection process ends and the personalized appearance expression of the pedestrian is obtained. In this process, images from different perspectives together constitute the generalized feature representation of the pedestrian's appearance.
5. The method for person re-identification based on personalized appearance modeling according to claim 1, characterized in that: Step 4 uses two methods, similarity nearest neighbor and credible distance, to re-rank the image sequence; Method 1: Based on the similarity matrix S G Retrieve the top image pairs in the gallery that are similar to the images in set C and add these images to the query image I q Ranked list of; Method 2: Calculate the gallery image I g The credible distance to the set C, the gallery images are sorted according to the credible distance U.
6. The method for person re-identification based on personalized appearance modeling according to claim 5, characterized in that: The calculation formula of the credible distance of the second method is as follows: The specific steps are as follows: Step 4.1: Calculate candidate image I g and image c in set C m The cosine similarity distance D between them; Step 4.2: Calculate the image pair [I g ,c m ] and normalize it to get the weight W; Step 4.3: Use weight W to perform weighted summation on distance D to obtain image I g The credible distance U to the set C.