An unsupervised video pedestrian re-identification method and system based on ensemble clustering

Abstract

The present invention discloses an unsupervised video pedestrian re-identification method and system based on integrated clustering, the method comprising: S1, obtaining a pedestrian data set and extracting pedestrian data; S2, extracting features from pedestrian data based on a feature extractor to obtain data features; S3, clustering data features based on an integrated clustering method; S4, constructing pseudo labels according to clustering results and updating feature extractor parameters; S5, returning to step S2 until the feature extractor converges; S6, extracting features based on the converged feature extractor, and completing pedestrian re-identification through distance measurement and sorting. The system comprises: an acquisition module, an extraction module, a clustering module, an update module, a loop module and a recognition module. By using the present invention, the clustering results can be made more robust and stable, which helps to improve the recognition accuracy. As an unsupervised video pedestrian re-identification method and system based on integrated clustering, the present invention can be widely used in the field of pedestrian recognition.

Description

Technical Field

[0001] The present invention relates to the field of pedestrian recognition, and in particular to an unsupervised video pedestrian re-recognition method and system based on integrated clustering. Background Art

[0002] Visual person re-identification based on video images refers to the use of computer vision technology to associate the same pedestrians in a network of non-overlapping domain cameras. It has important applications in video security and commercial passenger flow analysis. The solutions for person re-identification can generally be divided into two types: fully supervised and unsupervised. At present, the limitation of the fully supervised method is that it can only use data sets with manual annotations, which is undoubtedly affected by the high cost of manual annotation and the small size of the annotated data set; the pseudo-labels generated by the completely unsupervised video pedestrian recognition method are all based on the clustering results of a certain stage of training. Re-optimizing the feature extractor with noisy pseudo-labels will cause the network to degenerate and the performance to decrease. Summary of the invention

[0003] In order to solve the above technical problems, the purpose of the present invention is to provide an unsupervised video pedestrian re-identification method and system based on ensemble clustering, which can obtain more reliable pseudo-labels, so that the clustering results are more robust and stable.

[0004] The first technical solution adopted by the present invention is: an unsupervised video pedestrian re-identification method based on ensemble clustering, comprising the following steps:

[0005] S1. Obtain a pedestrian data set and extract pedestrian data;

[0006] S2, extracting features from pedestrian data based on a feature extractor to obtain data features;

[0007] S3, clustering the data features based on the integrated clustering method to obtain the clustering results;

[0008] S4, construct pseudo labels based on clustering results and update feature extractor parameters;

[0009] S5, returning to step S2 until the feature extractor converges, and obtaining a converged feature extractor;

[0010] S6. Extract features based on the converged feature extractor, measure and sort them by distance, and complete pedestrian re-identification.

[0011] Furthermore, the step of obtaining a pedestrian data set and extracting pedestrian data specifically includes:

[0012] S11, acquiring monitoring data based on monitoring cameras to obtain a pedestrian data set;

[0013] S12, performing pedestrian detection on the pedestrian data set according to the pedestrian detector to obtain a pedestrian detection frame;

[0014] S13, cropping is performed according to the pedestrian detection frame to obtain pedestrian data.

[0015] Furthermore, the step of clustering the data features based on the integrated clustering method to obtain the clustering results specifically includes:

[0016] S31, separating data features to obtain global features and sub-features;

[0017] S32, clustering the global features and sub-features based on the DBSCAN clustering method, and obtaining multiple preliminary results;

[0018] S33. Integrate multiple preliminary results based on the hierarchical clustering method to obtain clustering results.

[0019] Furthermore, the step of separating the data features to obtain global features and sub-features specifically includes:

[0020] S311, taking the feature of each picture in the data feature as a sub-feature;

[0021] S312. Construct a unified representation of the features of all images in the data features by average pooling and maximum pooling to obtain global features.

[0022] Furthermore, the step of integrating the multiple preliminary results based on the hierarchical clustering method to obtain the clustering result specifically includes:

[0023] S331, calculating a co-occurrence matrix according to a plurality of preliminary results;

[0024] S332, construct a stepwise phylogenetic tree diagram based on the co-occurrence matrix and hierarchical clustering method;

[0025] S333, cutting the stepwise phylogenetic tree diagram based on a preset cutting threshold and generating an integrated clustering result.

[0026] Further, the formula of the co-occurrence matrix is ​​expressed as follows:

[0027]

[0028]

[0029] In the above formula, P k (x i ) represents the sample x i In the k-th clustering result, P k (x j ) represents the sample x jIn the k-th clustering result, n represents the number of samples.

[0030] The second technical solution adopted by the present invention is: an unsupervised video pedestrian re-identification system based on integrated clustering, comprising:

[0031] An acquisition module is used to acquire a pedestrian data set and extract pedestrian data;

[0032] The extraction module extracts features from pedestrian data based on the feature extractor to obtain data features;

[0033] Clustering module, clusters data features based on integrated clustering method to obtain clustering results;

[0034] The update module builds pseudo labels based on the clustering results and updates the feature extractor parameters;

[0035] A loop module returns to the feature extraction step until the feature extractor converges to obtain a converged feature extractor;

[0036] The recognition module extracts features based on the converged feature extractor, and completes pedestrian re-identification through distance measurement and sorting.

[0037] The beneficial effects of the method and system of the present invention are as follows: the present invention calculates the co-occurrence matrix, obtains hierarchical clustering based on multiple feature clustering results and fuses them, obtains more accurate pseudo-labels to optimize the network, and makes the final feature clustering results more robust and stable, thereby making the recognition results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a flowchart of the steps of an unsupervised video pedestrian re-identification method based on integrated clustering of the present invention;

[0039] Figure 2 It is a structural block diagram of an unsupervised video pedestrian re-identification system based on integrated clustering of the present invention. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only provided for the convenience of explanation and description, and the order between the steps is not limited in any way. The execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0041] like Figure 1 As shown, the present invention provides an unsupervised video pedestrian re-identification method based on ensemble clustering, the method comprising the following steps:

[0042] S1. Obtain a pedestrian data set and extract pedestrian data;

[0043] S11, acquiring monitoring data based on monitoring cameras to obtain a pedestrian data set;

[0044] S12, performing pedestrian detection on the pedestrian data set according to the pedestrian detector to obtain a pedestrian detection frame;

[0045] S13, cropping is performed according to the pedestrian detection frame to obtain pedestrian data.

[0046] Specifically, the invention does not require manual annotation of the association between pedestrians across cameras. It only needs to obtain monitoring data from multiple cameras and use a pedestrian detector to detect pedestrians, and then crop the pedestrian detection frame to obtain data only of pedestrians.

[0047] S2, extracting features from pedestrian data based on a feature extractor to obtain data features;

[0048] Specifically, the feature extractor can be any convolutional network model. The present invention adopts the Resnet50 network as the feature extractor, which extracts features from two-dimensional image data or three-dimensional video data and maps them into a low-dimensional embedding space. The present invention uses a 2048-dimensional feature to represent a picture or video of a pedestrian.

[0049] The present invention adopts Resnet50 as a feature extractor, and replaces the last fully connected classification layer of the original Resnet50 with a Gem pooling operation that generates global features and sub-features. The residual structure of Resnet50 helps to solve the problems of network degradation and gradient diffusion. The Gem pooling operation is generalized mean pooling, which is a pooling operation between maximum pooling and average pooling. It contains parameters to be learned to facilitate the network to automatically select a pooling method that better matches the current task.

[0050] The output size of the Conv5.x module of Resnet50 is 4×2048×8×4, which represent the number of frames, the number of channels, the length and the width respectively. Here, the global feature Gempool Gem pools the entire output feature to obtain a global feature of size , and the 4 sub-feature Gempools perform Gem pooling operations on the features of a certain frame to obtain 4 sub-features of size . Since each Gem pooling has its own pooling parameters to be learned, different Gem layers are set for the extraction of each sub-feature and global feature. Different from the previous method of separating sub-features spatially in pedestrian re-identification, the present invention separates sub-features in the video frame dimension, rather than separating the pedestrian's body parts into several sub-features.

[0051] S3, clustering the data features based on the integrated clustering method to obtain the clustering results;

[0052] S31, separating data features to obtain global features and sub-features;

[0053] Specifically, for a video sample containing several pictures (for example, four or eight pictures per video sample), several picture-level features can be obtained after passing through the feature extractor. The features of the picture are used as sub-features of the video sample, and the features of all pictures are formed into a unified representation called global features by average pooling, maximum pooling, etc. Different from the previous method of separating sub-features spatially in pedestrian re-identification, the present invention separates sub-features in the video frame dimension, rather than separating the pedestrian's body parts into several sub-features.

[0054] S311, taking the feature of each picture in the data feature as a sub-feature;

[0055] S312. Construct a unified representation of the features of all images in the data features by average pooling and maximum pooling to obtain global features.

[0056] S32, clustering the global features and sub-features based on the DBSCAN clustering method, and obtaining multiple preliminary results;

[0057] Specifically, the specific steps of the DBSCAN clustering method are as follows:

[0058] ε-neighborhood: for f i ∈F, whose ε-neighborhood contains the sample feature set F that is related to f j The distance between the samples is not greater than ε, that is, N ε (f j )={f i ∈D|d(f i ,f j )≤ε}, the number of this sub-sample feature set is recorded as |N ε (f j )|.

[0059] Core object: For any sample feature f j ∈F, if its ε-neighborhood corresponds to N ε (f j ) contains at least MinPts sample features, that is, if |N ε (f j )|≥MinPts, then f j Is the core object.

[0060] Input: video sequence sample features F = {f 1 .f 2 ...f n}, neighborhood parameters (ε, MinPts), distance metric d.

[0061] Output: DBSCAN cluster class C.

[0062] 1) Initialize the core object collection Initialize the number of clusters k = 0, initialize the unvisited sample set Γ = F, clustering results

[0063] 2) For j = 1, 2, ... n, find all core objects by following the steps below:

[0064] A) Find sample f using distance metric d j The ε-neighborhood subsample set N ε (f j )

[0065] B) If the number of samples in the subsample set satisfies N ε (f j )≥MinPts, the sample x j Add core object sample set: Ω=Ω∪{f j}

[0066] 3) If the core object collection The algorithm ends, otherwise it goes to step 4.

[0067] 4) Randomly select a core object o from the core object set Ω and initialize the current cluster core object queue Ω cur = {o}, initialize the category number k = k + 1, initialize the current cluster sample set C k ={o}, update the unvisited sample set Γ = Γ-{o}.

[0068] 5) If the current cluster core object queue The current cluster C k Generation is complete, update cluster partition C = {C 1 ,C 2 ,...,C k}, Update the core object set Ω = Ω - C k , go to step 5. Otherwise update the core object set Ω = Ω - C k .

[0069] 6) In the current cluster core object queue Ω cur Take out a core object o′ and find all ε-neighborhood subsample sets N by using the neighborhood distance threshold ε ε (o′), let Δ=N ε (o′)∩Γ, update the current cluster sample set C k =C k ∪Δ, update the unvisited sample set Γ=Γ-Δ, update Ω cur =Ω cur∪(Δ∩Ω)-o′, go to step 5.

[0070] S33. Integrate multiple preliminary results based on the hierarchical clustering method to obtain clustering results.

[0071] Specifically, because each clustering of global features and sub-features is performed separately, in order to integrate multiple clustering results, the label correspondence problem must be solved. The method used here is to use the co-occurrence matrix as the distance matrix of hierarchical clustering and combine multiple clustering results using the hierarchical clustering method.

[0072] S331, calculating a co-occurrence matrix according to a plurality of preliminary results;

[0073] Specifically, the co-occurrence matrix is ​​a matrix that can map the partition results of different clusters to an intermediate representation. Assuming that the number of samples in the sample set X is n, the size of the co-occurrence matrix M is n×n. The formula of the co-occurrence matrix is ​​as follows:

[0074]

[0075]

[0076] In the above formula, P k (x i ) represents the sample x i In the k-th clustering result, P k (x j ) represents the sample x j In the k-th clustering result, n represents the number of samples. The value at position (i, j) represents x i and x j The average number of times that two samples are classified into the same category in all clustering results. The larger the value, the more times two samples are classified into the same category, and the higher the similarity between them. Therefore, the co-occurrence matrix can be regarded as a similarity measure of sample set X. After obtaining the co-occurrence matrix, the present invention uses a hierarchical clustering algorithm for consistency partitioning.

[0077] S332, construct a stepwise phylogenetic tree diagram based on the co-occurrence matrix and hierarchical clustering method;

[0078] S333, cutting the stepwise phylogenetic tree diagram based on a preset cutting threshold and generating an integrated clustering result.

[0079] Specifically, the input of the algorithm is the video sequence sample feature set F = {f 1 .f 2 ...f n}, distance update function Func d , cutting threshold t, the output is the integrated clustering result S. The distance update function Func dIt is a function that updates the distance between clusters in the process of hierarchical clustering. It can be the update method of the minimum distance between classes, the maximum distance between classes and the average distance between classes. They represent the minimum distance, the maximum distance and the average distance between samples of clusters as dissimilarity, that is, the updated distance. Finally, according to the experimental results, the present invention adopts the average distance update function, that is, the distance between one cluster and another cluster is considered to be equal to the average distance from any member of one cluster to any member of another cluster. The algorithm first generates a global feature clustering result S based on the feature F. g And the sub-feature clustering result S 1 ,S 2 ...S n , and use the formula to calculate the co-occurrence matrix M. The co-occurrence matrix M can be used as a measure of dissimilarity between samples. After that, the co-occurrence matrix M and the hierarchical clustering algorithm are used to gradually build the system tree map. First, it is necessary to initialize the step-by-step system tree map L, the cluster pairwise difference dictionary D, and the label set S label The stepwise dendrogram L can store hierarchical clustering information, which changes according to the combined samples. The cluster pairwise difference dictionary D initially stores the distance between samples, and as the samples are combined, it stores the distance between clusters. The cluster label set S label Then store the combined cluster set. For each cluster merge, find the updated minimum distance between clusters and update L and S label , while in S label Create a new merged cluster class and then label All cluster pairwise differences in the are updated. The whole process is iterated until all samples are classified into one cluster. Finally, the stepwise phylogenetic tree L is cut using the cutting threshold t to generate the integrated clustering result S.

[0080] S4, construct pseudo labels based on clustering results and update feature extractor parameters;

[0081] S5, returning to step S2 until the feature extractor converges, and obtaining a converged feature extractor;

[0082] The memory loss function has been applied to supervised and unsupervised person re-identification tasks in previous work and can improve performance. The memory loss function is a contrastive learning loss. The general practice is to use a current iteration lookup table V to record the features belonging to each cluster center and compare the samples with the center features of each cluster during the forward propagation. However, in previous unsupervised work, only the features of the samples with the same cluster pseudo-label in the current iteration were considered to initialize the current iteration lookup table V, without combining the clustering results of the previous iteration. This paper sets up a center feature memory lookup table C, which is responsible for recording and dynamically updating the center features of the cluster to which each sample belongs. In the initialization stage of each round of iteration, the features in the center feature memory lookup table C can be used to initialize the center features of each cluster in the lookup table, so that the features of the cluster center not only depend on the features of all samples classified into the cluster, but also come from the surrounding samples.

[0083] Since the network optimization and clustering are iterative in the unsupervised person re-identification task, the number and index of pseudo labels obtained in each iteration are disrupted, so the integrated memory module needs to be initialized in each iteration. A clustering result is obtained after the features extracted by the feature extractor are clustered separately and integrated, and the cluster center of the lookup table is initialized using the average center feature vector of each sample in each cluster. For the i-th cluster in the current iteration lookup table, its initialized center feature is:

[0084]

[0085] N i represents the number of the i-th cluster, C j represents the central feature of the jth sample belonging to the i-th cluster. Its original intention is to use the average value of these central features to initialize the cluster central feature of the i-th cluster.

[0086] Given the features of a sample, the contrast loss function of the inherited memory module is defined as

[0087]

[0088] Where V is the lookup table that stores the current iteration, and is the temperature parameter used to control the softness of the probability distribution of different categories. It is set to 0.05 in this paper. The main idea of ​​contrast loss is to narrow the metric distance of the same cluster and push the metric distance of different clusters away. In each round of iteration, the features in the lookup table and the features in the center feature memory are passively updated. Where α 1 and α 2 They are used to control the momentum of feature updates in the current iteration lookup table V and the central feature memory lookup table C respectively.

[0089] Vi ←α 1 V i +(1-α 1 )f i

[0090] C j ←α 2 C j +(1-α 2 )V i

[0091] The update of the feature extractor uses the conventional stochastic gradient descent method, which passes through the feature extractor network and calculates the loss function to obtain the gradient backpropagation, thereby updating the parameters of the feature extractor network.

[0092] S6. Extract features based on the converged feature extractor, measure and sort them by distance, and complete pedestrian re-identification.

[0093] The present invention makes full use of the rich data of the video pedestrian re-identification data set and mitigates the negative effects of noisy pseudo-labels on network optimization. Compared with the previous unsupervised pedestrian re-identification method based on a single frame image, the present invention integrates the clustering results of sample sub-features and global features in the process of generating pseudo-labels. The previous method relied too much on the clustering results in the current iteration process, and the generated noisy pseudo-labels damaged the optimization of the feature extractor, thereby reducing the performance. The pseudo-labels of integrated clustering can further reduce the noise influence of edge samples and outlier samples, provide more reliable and stable pseudo-labels for the optimization process of the feature extractor, and optimize the network in the right direction. In the previous methods based on contrast loss, the memory initialization part of each round of iteration comes from the sample feature average value of each cluster in the clustering result. The previous methods directly split the results of each iteration, which is not conducive to the smooth iterative optimization of the feature extractor. The present invention sets a central feature memory module, which is responsible for recording and dynamically updating the central feature of the cluster to which each sample belongs. It is used to initialize the central feature of each cluster in the initialization stage of each round of iteration, so that the features of the cluster center not only depend on the features of all samples divided into the cluster, but also come from the surrounding samples. The present invention transmits clustering results generated by different iterations using a memory module, making the training process more stable.

[0094] like Figure 2 As shown, an unsupervised video pedestrian re-identification system based on integrated clustering includes:

[0095] An acquisition module is used to acquire a pedestrian data set and extract pedestrian data;

[0096] The extraction module extracts features from pedestrian data based on the feature extractor to obtain data features;

[0097] Clustering module, clusters data features based on integrated clustering method to obtain clustering results;

[0098] The update module builds pseudo labels based on the clustering results and updates the feature extractor parameters;

[0099] A loop module returns to the feature extraction step until the feature extractor converges to obtain a converged feature extractor;

[0100] The recognition module extracts features based on the converged feature extractor, and completes pedestrian re-identification through distance measurement and sorting.

[0101] An unsupervised video person re-identification device based on ensemble clustering:

[0102] at least one processor;

[0103] at least one memory for storing at least one program;

[0104] When the at least one program is executed by the at least one processor, the at least one processor implements the unsupervised video pedestrian re-identification method based on integrated clustering as described above.

[0105] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0106] A storage medium storing processor-executable instructions, characterized in that: the processor-executable instructions are used to implement the above-mentioned unsupervised video pedestrian re-identification method based on ensemble clustering when executed by the processor.

[0107] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0108] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An unsupervised video person re-identification method based on ensemble clustering, characterized in that: The following steps are involved: S1. Obtain a pedestrian data set and extract pedestrian data; S2, extracting features from pedestrian data based on a feature extractor to obtain data features; S3, clustering the data features based on the integrated clustering method to obtain the clustering results; S4, construct pseudo labels based on clustering results and update feature extractor parameters; S5, returning to step S2 until the feature extractor converges, and obtaining a converged feature extractor; S6, extract features based on the converged feature extractor, measure and sort them by distance, and complete pedestrian re-identification; The step of clustering the data features based on the integrated clustering method to obtain the clustering results specifically includes: S31, separating data features to obtain global features and sub-features; S32, clustering the global features and sub-features based on the DBSCAN clustering method, and obtaining several preliminary results; S33, integrating multiple preliminary results based on a hierarchical clustering method to obtain a clustering result; The step of integrating multiple preliminary results based on the hierarchical clustering method to obtain a clustering result specifically includes: S331, calculating a co-occurrence matrix according to a plurality of preliminary results; S332, construct a stepwise phylogenetic tree diagram based on the co-occurrence matrix and hierarchical clustering method; S333, cutting the stepwise system tree diagram based on a preset cutting threshold and generating an integrated clustering result; The formula of the co-occurrence matrix is ​​as follows: In the above formula, P k (x i ) represents the sample x i In the k-th clustering result, P k (x j ) represents the sample x j In the k-th clustering result, n represents the number of samples.

2. According to claim 1, the unsupervised video pedestrian re-identification method based on ensemble clustering is characterized in that: The step of obtaining a pedestrian data set and extracting pedestrian data specifically includes: S11, acquiring monitoring data based on monitoring cameras to obtain a pedestrian data set; S12, performing pedestrian detection on the pedestrian data set according to the pedestrian detector to obtain a pedestrian detection frame; S13, cropping is performed according to the pedestrian detection frame to obtain pedestrian data.

3. The unsupervised video pedestrian re-identification method based on ensemble clustering according to claim 2 is characterized in that: The step of separating the data features to obtain global features and sub-features specifically includes: S311, taking the feature of each picture in the data feature as a sub-feature; S312. Construct a unified representation of the features of all images in the data features by average pooling and maximum pooling to obtain global features.

4. An unsupervised video person re-identification system based on ensemble clustering, characterized in that: The method for performing unsupervised video pedestrian re-identification based on integrated clustering as claimed in claim 1 comprises: An acquisition module is used to acquire a pedestrian data set and extract pedestrian data; The extraction module extracts features from pedestrian data based on the feature extractor to obtain data features; Clustering module, clusters data features based on integrated clustering method to obtain clustering results; The update module builds pseudo labels based on the clustering results and updates the feature extractor parameters; A loop module returns to the feature extraction step until the feature extractor converges to obtain a converged feature extractor; The recognition module extracts features based on the converged feature extractor, and completes pedestrian re-identification through distance measurement and sorting.

Images