A semi-supervised learning based cross-modal pedestrian re-identification method and system

Abstract

The application provides a cross-modal pedestrian re-identification method and system based on semi-supervised learning. The basic idea of the application includes two aspects: (1) under the guidance of a small amount of labeled samples, more reliable pseudo-labeled cross-modal data are mined from a large amount of unlabeled data; and (2) discriminative information contained in a small amount of labeled cross-modal data and the pseudo-labeled cross-modal data is fully utilized, the influence of the difference between modes is reduced, and the discriminability of the training model is improved.

Description

Technical Field

[0001] This invention relates to the field of pedestrian recognition technology, and in particular to a cross-modal pedestrian re-identification method and system based on semi-supervised learning. Background Technology

[0002] With the increasing demand for public safety, more and more infrared cameras are being integrated into surveillance systems to enhance the ability to accurately identify specific individuals day and night. During pedestrian retrieval, the combination of identity-related infrared information and visible information creates a new pedestrian re-identification problem—cross-modal pedestrian re-identification.

[0003] Compared to visible light cameras, infrared cameras can capture sufficient information from scenes in dark environments, compensating for the performance limitations of existing unimodal person re-identification models in low-light conditions, which is helpful for pedestrian retrieval. However, compared to unimodal person re-identification, cross-modal person re-identification faces challenges in two main aspects: it requires not only a large amount of labeled data but also significant intra-class differences in pedestrian images and inter-modal differences between visible light and infrared images. These differences make cross-modal person re-identification tasks more challenging.

[0004] Existing cross-modal person re-identification methods are mainly divided into two categories: supervised cross-modal person re-identification and unsupervised cross-modal person re-identification. Supervised cross-modal person re-identification improves cross-modal matching performance by learning a feature extraction model for discrimination from a large amount of labeled cross-modal data. However, supervised methods usually require labeling a large number of cross-modal samples for the cross-modal person re-identification dataset, which inevitably consumes a lot of human and material resources, thus limiting the applicability of supervised cross-modal person re-identification methods in real-world scenarios. To reduce labeling costs and alleviate excessive reliance on labeled data, some unsupervised methods have been proposed. According to existing unsupervised work, the performance of current unsupervised methods lags significantly behind that of supervised methods. The main reason for this large performance difference may be that the unlabeled cross-modal data itself has significant inter-modal differences, and the lack of guidance from real labels in the process of generating pseudo-labeled cross-modal data from unlabeled data makes it impossible to guarantee the reliability of the generated pseudo-labeled data, thereby affecting the performance of the algorithm. Summary of the Invention

[0005] To reduce reliance on labeled data while mitigating model performance degradation, this invention provides a cross-modal pedestrian re-identification method and system based on semi-supervised learning. The aim is to learn a discriminative model from a small amount of labeled data and a large amount of unlabeled data, thereby reducing the need for labeled data and using a small amount of labeled data to guide the use of unlabeled data, thus improving the reliability of the model.

[0006] On the one hand, this invention provides a cross-modal person re-identification method based on semi-supervised learning, comprising:

[0007] Step 1: Acquire labeled pedestrian images and unlabeled pedestrian images; the pedestrian images include two modalities: visible light pedestrian images and infrared pedestrian images;

[0008] Step 2: Construct a feature extraction network, and use the feature extraction network to extract features from labeled and unlabeled pedestrian images respectively, to obtain labeled data features and unlabeled data features;

[0009] Step 3: Cluster the labeled pedestrian images according to the features of the labeled data to obtain the first clustering result and the pseudo-labels corresponding to each cluster. Calculate the image pair selection rate using the contour coefficients of the real and pseudo-labels of the labeled pedestrian images.

[0010] Step 4: Cluster the unlabeled pedestrian images according to the features of the unlabeled data to obtain the second clustering results and the pseudo-labels corresponding to each cluster, and save the class center features of each cluster to the class center in the class center memory.

[0011] Step 5: Calculate the similarity between any unlabeled visible light pedestrian image and infrared pedestrian image, and perform cross-modal image pair matching on two pedestrian images with a similarity value not less than the first similarity threshold to obtain the set of unlabeled image pairs R;

[0012] Step 6: Select unlabeled image pairs with high similarity values ​​from the set of unlabeled image pairs R according to the image pair selection rate, thereby updating the mixed samples in the class center memory;

[0013] Step 7: Sample the second clustering results to select the most similar P-1 class hard samples as the training set;

[0014] Step 8: Design a supervised loss for labeled pedestrian images and a class center loss for unlabeled pedestrian images. Utilize the supervised loss and class center loss together to optimize the feature extraction network.

[0015] Step 9: Based on the given query samples, use the trained feature extraction network to search the target image database.

[0016] Further, in step 3, the image pair selection rate is calculated using the contour coefficients of the real and pseudo markers in the labeled pedestrian images, specifically including:

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] Where lengthratio represents the image pair selection ratio, Let represent the t-th labeled visible light pedestrian image. Let t represent the t-th unmarked visible light pedestrian image. Let represent the t-th marked infrared pedestrian image. This represents the t-th unmarked infrared pedestrian image. c(*) represents the average distance between image * and other images within the same cluster, d(*) represents the average minimum distance between image * and all images in different clusters, Score(·) represents the mean of the silhouette coefficients of all sample images in the input sample, and avg(·) represents calculating the mean.

[0023] Furthermore, it also includes:

[0024] Calculate the similarity between any labeled visible light pedestrian image and infrared pedestrian image, and perform cross-modal image pair matching on two pedestrian images with a similarity value not less than a second similarity threshold to obtain a set of labeled image pairs;

[0025] The mean similarity of all image pairs in the labeled image pair set is calculated and used as the first similarity threshold.

[0026] Furthermore, step 6 specifically includes:

[0027]

[0028]

[0029] in, and This indicates that based on the sample pairs in R In the The features of the mixed samples updated in the next iteration. q i and q v Let represent the features corresponding to the infrared pedestrian image query sample and the visible light pedestrian image query sample extracted by the feature extraction network, respectively, and β be the momentum update factor. Indicates the number of iterations.

[0030] Furthermore, step 7 specifically includes:

[0031] At the beginning of each epoch, an image is randomly selected from each cluster in the second clustering result to construct a subset dataset;

[0032] Feature extraction is performed on the subset of data using the current network model to obtain the feature set X∈R. C×d , where C and d represent the number of cluster categories and feature dimension corresponding to the second clustering result, respectively;

[0033] For each cluster c i From its P-1 closest classes, randomly select one image to form a set. And construct a graph G = (V, E), where V = {c | c = 1, 2, ..., C} represents the vertices of the graph. Denotes the edges in the graph; where c i ∈C, P≤C, where P represents the number of pseudo-labeled classes to be sampled in each mini-batch;

[0034] Each cluster c in C i As an anchor point, retrieve c i In G, find all the connection classes, then the anchor class c, and the connection class closest to it. Combined to form a set And thus obtain

[0035] For all classes in A, randomly select K instances of each class to generate a mini-batch of data of B = P × K samples, which will serve as the training set for each round in the current epoch.

[0036] Furthermore, the feature extraction network adopts the ResNet50 network.

[0037] Furthermore, in step 8, a supervised loss is designed for the labeled pedestrian images, specifically including:

[0038] Calculate the cross-entropy loss between the predicted value and the true label according to formula (1);

[0039]

[0040] Where n and m represent the number of visible light pedestrian images and infrared pedestrian images in the current batch, respectively. and Let represent the feature vector corresponding to the i-th visible light pedestrian image and the j-th infrared pedestrian image, respectively. and They represent and The corresponding real marker, and This indicates that under the same classifier C(·) and Identity prediction.

[0041] Furthermore, in step 8, a class center loss is designed for unlabeled pedestrian images, specifically including:

[0042] The class center loss of infrared pedestrian images is designed according to formulas (2) to (4);

[0043]

[0044]

[0045]

[0046] in, This represents the aggregated class center features corresponding to infrared pedestrian images in the class center memory and the infrared pedestrian image query sample q. i Class center loss between This represents the mixture of class center features of the infrared pedestrian images in the class center memory and the infrared pedestrian image query sample q. i Class center loss between This indicates that the query sample q is in the class central memory. i The feature vector of the corresponding positive sample of the class, This indicates that the query sample q is in the class central memory. i The updated feature vector of the corresponding class, where K represents the number of classes, λ represents the weight parameter, and τ represents the hyperparameter;

[0047] The class center loss for visible light pedestrian images is designed according to formulas (5) to (7);

[0048]

[0049]

[0050]

[0051] in, This represents the aggregated class center features corresponding to visible light pedestrian images in the class center memory and the visible light pedestrian image query sample q. v Class center loss between This represents the class center features of the mixed samples corresponding to visible light pedestrian images in the class center memory and the visible light pedestrian image query sample q. v Class center loss between This indicates that the query sample q is in the class central memory. vThe feature vector of the corresponding positive sample of the class, This indicates that the query sample q is in the class central memory. v The feature vector of the updated sample corresponding to the class;

[0052] According to L i and L v The design yielded a total class center loss.

[0053] Furthermore, in step 8, the network is optimized by comprehensively utilizing the supervision loss and class center loss, specifically including:

[0054] L ReID =L uns +L sid (16).

[0055] On the other hand, the present invention provides a cross-modal pedestrian re-identification system based on semi-supervised learning, comprising: a pseudo-label cross-modal data generation module and a pseudo-label cross-modal data refinement module; a class center memory is provided in the pseudo-label cross-modal data refinement module;

[0056] The pseudo-labeled cross-modal data generation module is used to extract features from labeled pedestrian images using a feature extraction network to obtain labeled data features; cluster the labeled pedestrian images according to the labeled data features to obtain a first clustering result and pseudo-labels corresponding to each cluster; calculate the image pair selection rate using the contour coefficients of the real and pseudo-labels of the labeled pedestrian images; cluster the unlabeled pedestrian images according to the unlabeled data features to obtain a second clustering result and pseudo-labels corresponding to each cluster; and save the class center features of each cluster to the class centers in the class center memory; calculate the similarity between any unlabeled visible light pedestrian image and infrared pedestrian image; perform cross-modal image pair matching on two pedestrian images with a similarity value not less than a first similarity threshold to obtain an unlabeled image pair set R; and select unlabeled image pairs with higher similarity values ​​from the unlabeled image pair set R according to the image pair selection rate, thereby updating the mixed samples in the class center memory.

[0057] The pseudo-labeled cross-modal data refinement module is used to extract features from unlabeled pedestrian images using a feature extraction network to obtain unlabeled data features; sample the second clustering results to select the most similar P-1 class hard samples as the training set; design a supervised loss for labeled pedestrian images and a class center loss for unlabeled pedestrian images, and comprehensively utilize the supervised loss and class center loss to optimize the feature extraction network; input the pedestrian image to be tested into the trained feature extraction network to determine whether there is a target pedestrian in it.

[0058] The beneficial effects of this invention are:

[0059] 1. By introducing semi-supervised methods into cross-modal person re-identification, the dependence of supervised cross-modal person re-identification on labeled data is alleviated, thereby making better use of pseudo-labeled data to improve the performance and scalability of the person re-identification model.

[0060] 2. Design a pseudo-label cross-modal data generation module. This module is based on the cosine similarity and silhouette coefficient of a small amount of labeled visible light and infrared image data. It adaptively selects pseudo-labeled visible light and infrared image pairs with high cosine similarity scores according to the silhouette coefficient ratio to update the class center memory features corresponding to the pseudo-labels.

[0061] 3. Design a pseudo-labeled cross-modal data refinement module. In the sampling stage, this module calculates the similarity between samples in each class based on the features of unlabeled data and the corresponding pseudo-labels and constructs a graph. Then, samples with high similarity between classes in the graph are selected for training. Secondly, the model is continuously optimized by dynamically updating the class center features of the memory and calculating the class center loss, so that the feature quality of the pseudo-labeled cross-modal data is continuously improved. Attached Figure Description

[0062] Figure 1 A flowchart illustrating a cross-modal person re-identification method based on semi-supervised learning provided in an embodiment of the present invention;

[0063] Figure 2 This invention provides a research framework for a cross-modal person re-identification method based on semi-supervised learning, as presented in the embodiments of the present invention.

[0064] Figure 3 The retrieval results of different pedestrian images in the RegDB dataset provided in this embodiment of the invention;

[0065] Figure 4 A t-SNE visualization of 20 randomly selected identities provided for an embodiment of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] like Figure 1 As shown, this embodiment of the invention provides a cross-modal person re-identification method based on semi-supervised learning, including the following steps:

[0068] S101: Acquire labeled pedestrian images and unlabeled pedestrian images; wherein, pedestrian images include two modalities: visible light pedestrian images and infrared pedestrian images;

[0069] S102: Construct a feature extraction network (in this embodiment, a ResNet50 network is used), and use the feature extraction network to extract features from labeled pedestrian images and unlabeled pedestrian images respectively to obtain labeled data features and unlabeled data features;

[0070] S103: Cluster the labeled pedestrian images according to the features of the labeled data to obtain the first clustering result and the pseudo-labels corresponding to each cluster. Calculate the image pair selection rate using the contour coefficients of the real and pseudo-labels of the labeled pedestrian images.

[0071] Specifically, the corresponding values ​​of the real markers in the visible light pedestrian images are calculated separately. Corresponding to pseudo-tags Infrared pedestrian image real marker corresponding Corresponding to pseudo-tags Then according to and To calculate the image pair selection ratio (lengthratio), the corresponding formula is as follows:

[0072]

[0073]

[0074]

[0075]

[0076]

[0077] Wherein, lengthratio represents the average ratio of the contour coefficients of visible light pedestrian images and infrared pedestrian images using pseudo-labels and real labels, respectively, which is also the image pair selection rate; Let represent the t-th labeled visible light pedestrian image. Let t represent the t-th unmarked visible light pedestrian image. Let represent the t-th marked infrared pedestrian image. This represents the t-th unmarked infrared pedestrian image. c(*) represents the average distance between image * and other images within the same cluster, d(*) represents the average minimum distance between image * and all images in different clusters, Score(·) represents the mean silhouette coefficient of all sample images in the input sample, and avg(·) represents calculating the mean. During calculation, The data uses the corresponding real labels. and The data uses pseudo-labels generated by a clustering algorithm. In this embodiment, the DBSCAN clustering algorithm is used for clustering.

[0078] S104: Cluster the unlabeled pedestrian images according to the features of the unlabeled data to obtain the second clustering results and the pseudo-labels corresponding to each cluster, and save the class center features of each cluster to the class center in the class center memory.

[0079] S105: Calculate the similarity between any unlabeled visible light pedestrian image and infrared pedestrian image, and perform cross-modal image pair matching on two pedestrian images with a similarity value not less than the first similarity threshold to obtain the set of unlabeled image pairs R;

[0080] Specifically, in this embodiment, cosine similarity is used as the similarity index between images. The first similarity threshold is set using a dynamic update method, which specifically includes: calculating the similarity between any labeled visible light pedestrian image and infrared pedestrian image; performing cross-modal image pair matching on two pedestrian images with a similarity value not less than a second similarity threshold (in order to reduce the impact of incorrectly clustered image pairs, the second similarity threshold is set to 0.5 in this embodiment) to obtain a set of labeled image pairs; calculating the mean similarity of all image pairs in the set of labeled image pairs and using it as the first similarity threshold. This calculation process can be represented by the following formula (1).

[0081]

[0082] Where avg(·) represents calculating the mean. Image of a marked visible light pedestrian And marked infrared pedestrian images Similarity between them and These represent labeled pedestrian images. and The true label is Δ, which represents the confidence score of the labeled data similarity constraint, i.e., the first similarity threshold. In this embodiment, the superscript or subscript in each parameter represents visible light, and i represents infrared light.

[0083] S106: Select unlabeled image pairs with high similarity values ​​from the set of unlabeled image pairs R according to the image pair selection rate, thereby updating the mixed samples in the class center memory and reducing the impact of erroneous samples on the feature vectors in the memory.

[0084] Specifically, unlabeled image pairs are sorted in descending order based on the number of samples in the same category. Image pairs selected from the sequence using an image pair selection rate are then used to update the memory database, thereby reducing the occurrence of a small number of erroneous image pairs and improving the reliability of the image pairs in the memory database. The update process is represented by the following formula:

[0085]

[0086]

[0087] in, and This indicates that based on the sample pairs in R In the The features of the mixed samples updated in the next iteration. q i and q v Let represent the features corresponding to the infrared pedestrian image query sample and the visible light pedestrian image query sample extracted by the feature extraction network, respectively, and β be the momentum update factor. This represents the number of iterations. In this embodiment, β is set to 0.1.

[0088] S107: Sample the second clustering results to select the most similar P-1 class hard samples as the training set;

[0089] Specifically, this embodiment designs a sampler to shift the mining of hard samples with pseudo-labels to the data sampling stage. The basic idea of ​​this sampler is: at the beginning of each epoch, the latest feature extraction network is used to evaluate the distance between categories, and then a graph containing all categories is constructed. In this way, training samples can be collected by utilizing the similarity relationships between categories. The specific process is as follows:

[0090] First, an image is randomly selected from each cluster in the second clustering results to construct a small subset. Features are then extracted from this subset using the current feature extraction network, resulting in a feature set denoted as X∈R. C ×d Where C and d represent the number of cluster categories (also known as the number of pseudo-labeled data categories) and feature dimension corresponding to the second clustering result, respectively.

[0091] Secondly, the pairwise distances between all selected samples are calculated to obtain the distance matrix dist∈R for all classes.C×C Based on this distance matrix, for each pseudo-labeled class c i For each ci∈C, randomly select an image from its P-1 (P≤C) closest classes to form a set. Where P is the number of pseudo-labeled classes to be sampled in each mini-batch. Each class is treated as a vertex in the graph, thus the constructed graph can be represented as G = (V, E), where V = {c | c = 1, 2, ..., c} represents the vertices of the graph. This represents the edges in the graph.

[0092] Finally, during the sampling phase, each class c in the pseudo-labeled class C is... i As an anchor point, retrieve all its connection classes in G, and then retrieve the anchor class c. i Its closest connection class Combined to form a set Then you can get For all classes in A, randomly select K instances of each class to generate a mini-batch of data of B = P × K samples, which will be used for iterative training in each round of this epoch.

[0093] During the next epoch of training, the latest feature extraction network is used to extract the feature vector of the data, X is updated, and then the graph G is updated following X. The selected mini-batch samples for iteration are also updated.

[0094] S108: Design a supervised loss for labeled pedestrian images and a class center loss for unlabeled pedestrian images. Utilize the supervised loss and class center loss together to optimize the feature extraction network.

[0095] Specifically, the cross-entropy loss between the predicted value and the true label of the labeled pedestrian image is calculated according to formula (1) as the supervision loss;

[0096]

[0097] Where n and m represent the number of visible light pedestrian images and infrared pedestrian images in the current batch, respectively. and Let represent the feature vector corresponding to the i-th visible light pedestrian image and the j-th infrared pedestrian image, respectively. and They represent and The corresponding real marker, and This indicates that under the same classifier C(·) and Identity prediction.

[0098] The class center loss of infrared pedestrian images is designed according to formulas (2) to (4);

[0099]

[0100]

[0101]

[0102] in, This represents the aggregated class center features corresponding to infrared pedestrian images in the class center memory and the infrared pedestrian image query sample q. i Class center loss between This represents the mixture of class center features of the infrared pedestrian images in the class center memory and the infrared pedestrian image query sample q. i Class center loss between This indicates that the query sample q is in the class central memory. i The feature vector of the corresponding positive sample of the class, This indicates that the query sample q is in the class central memory. i The updated feature vector of the corresponding class, where K represents the number of classes, λ represents the weight parameter (set to 0.5 here), and τ represents the hyperparameter; in this embodiment, τ is set to 0.1.

[0103] The class center loss for visible light pedestrian images is designed according to formulas (5) to (7);

[0104]

[0105]

[0106]

[0107] in, This represents the aggregated class center features corresponding to visible light pedestrian images in the class center memory and the visible light pedestrian image query sample q. v Class center loss between This represents the class center features of the mixed samples corresponding to visible light pedestrian images in the class center memory and the visible light pedestrian image query sample q. v Class center loss between This indicates that the query sample q is in the class central memory. v The feature vector of the corresponding positive sample of the class, This indicates that the query sample q is in the class central memory. v The updated feature vector of the corresponding class;

[0108] According to L i and Lv The design yielded a total class center loss.

[0109] Finally, the total loss function for pedestrian re-identification is shown in Equation (9).

[0110] L ReID =L uns +L sid (16).

[0111] S109: Input the pedestrian image to be tested into the trained feature extraction network to determine whether there is a target pedestrian in it.

[0112] The present invention provides a cross-modal person re-identification method based on semi-supervised learning, the basic idea of ​​which includes two aspects: (1) under the guidance of a small number of labeled samples, more reliable pseudo-labeled cross-modal data is mined from a large number of unlabeled data. (2) the discriminative information contained in the small number of labeled cross-modal data and the mined pseudo-labeled cross-modal data is fully utilized to reduce the influence of inter-modal differences and improve the discriminative power of the trained model.

[0113] Example 2

[0114] To implement the above method, embodiments of the present invention provide a cross-modal person re-identification system based on semi-supervised learning, such as... Figure 2 As shown, the system includes a pseudo-labeled cross-modal data generation module and a pseudo-labeled cross-modal data refinement module. The pseudo-labeled cross-modal data refinement module includes a sampler and a class center memory. First, labeled and unlabeled data are used by the pseudo-labeled cross-modal data generation module to generate pseudo-labeled cross-modal image pairs, which are then used to update the class center memory in the pseudo-labeled cross-modal data refinement module. Then, labeled and unlabeled data are used by the pseudo-labeled cross-modal data refinement module for model learning and updating. The processing procedures of the two modules are described below:

[0115] The pseudo-labeled cross-modal data generation module is used to extract features from labeled pedestrian images using a feature extraction network to obtain labeled data features; cluster the labeled pedestrian images according to the labeled data features to obtain a first clustering result and pseudo-labels corresponding to each cluster; calculate the image pair selection rate using the contour coefficients of the real and pseudo-labels of the labeled pedestrian images; cluster the unlabeled pedestrian images according to the unlabeled data features to obtain a second clustering result and pseudo-labels corresponding to each cluster; and save the class center features of each cluster to the class centers in the class center memory; calculate the similarity between any unlabeled visible light pedestrian image and infrared pedestrian image; perform cross-modal image pair matching on two pedestrian images with a similarity value not less than a first similarity threshold to obtain an unlabeled image pair set R; and select unlabeled image pairs with higher similarity values ​​from the unlabeled image pair set R according to the image pair selection rate, thereby updating the mixed samples in the class center memory.

[0116] The pseudo-labeled cross-modal data refinement module is used to extract features from unlabeled pedestrian images using a feature extraction network to obtain unlabeled data features; sample the second clustering results to select the most similar P-1 class hard samples as the training set; design a supervised loss for labeled pedestrian images and a class center loss for unlabeled pedestrian images, and comprehensively utilize the supervised loss and class center loss to optimize the feature extraction network; input the pedestrian image to be tested into the trained feature extraction network to determine whether there is a target pedestrian in it.

[0117] In this embodiment of the invention, a class center memory is used to retain a corresponding class feature for each category, rather than retaining features for each instance. The corresponding class centers are updated regardless of the cluster size. Specifically, a two-layer class center memory consisting of class centers and mixed samples is designed here. The class centers are composed of sample class center features, and the mixed samples are the updated sample class center features of cross-modal label pairs selected by the selection rate in the pseudo-labeled cross-modal data generation module. The two feature sets are used to calculate the class center loss with the query features to improve the model's internal tightness of samples of the same category in the pseudo-labels and the separability between samples of different categories. This helps to enhance the model's performance and improve its robustness and generalization ability.

[0118] It should be noted that the pedestrian re-identification system provided in this embodiment is for the purpose of implementing the above method embodiment. Its specific functions can be referred to the above method embodiment, and will not be repeated here.

[0119] To verify the effectiveness of this invention, Table 1 provides experimental results for all supervised and unsupervised methods, as well as the semi-supervised method proposed in this invention, on the RegDB dataset. SVI-ReID is a supervised cross-modal person re-identification method, while USVI-ReID is an unsupervised cross-modal person re-identification method. Semi-supervised training uses 30% of the training data as labeled data, with the remainder as unlabeled data. Experimental results show that on the RegDB dataset, compared to supervised and unsupervised methods, the method of this invention achieves good performance and outperforms other works that rely on the same level of unsupervised training.

[0120] Table 1. Performance comparison of the method of the present invention with other methods on the RegDB dataset.

[0121]

[0122]

[0123] Figure 3 The images in the table are rank-10 retrieval results obtained by the method of this invention on the cross-modal dataset RegDB based on a given query sample. For each retrieval case, green boxes represent correct matches corresponding to the given query, while red boxes represent incorrect matches.

[0124] To better demonstrate the effectiveness of this invention, we plotted t-SNE (t-Distributed Stochastic Neighbor Embedding) maps for 20 randomly selected identities from the RegDB dataset to evaluate the features acquired by the network, such as... Figure 4 As shown, from initialization to our semi-supervised cross-modal framework, the positive sample points in visible light and infrared gradually converge, indicating that the method of this invention can reduce the differences between pedestrian images of different modalities.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal person re-identification method based on semi-supervised learning, characterized in that, include: Step 1: Acquire labeled pedestrian images and unlabeled pedestrian images; the pedestrian images include two modalities: visible light pedestrian images and infrared pedestrian images; Step 2: Construct a feature extraction network, and use the feature extraction network to extract features from labeled and unlabeled pedestrian images respectively, to obtain labeled data features and unlabeled data features; Step 3: Cluster the labeled pedestrian images according to the features of the labeled data to obtain the first clustering result and the pseudo-labels corresponding to each cluster. Calculate the image pair selection rate using the contour coefficients of the real and pseudo-labels of the labeled pedestrian images. Step 4: Cluster the unlabeled pedestrian images according to the features of the unlabeled data to obtain the second clustering results and the pseudo-labels corresponding to each cluster, and save the class center features of each cluster to the class center in the class center memory. Step 5: Calculate the similarity between any unlabeled visible light pedestrian image and infrared pedestrian image, and perform cross-modal image pair matching on two pedestrian images with a similarity value not less than the first similarity threshold to obtain the set of unlabeled image pairs R; Step 6: Select unlabeled image pairs with high similarity values ​​from the set of unlabeled image pairs R according to the image pair selection rate, thereby updating the mixed samples in the class center memory; Step 7: Sample the second clustering results to select the most similar clusters. Difficult samples are used as the training set; Step 8: Design a supervised loss for labeled pedestrian images and a class center loss for unlabeled pedestrian images. Utilize the supervised loss and class center loss together to optimize the feature extraction network. Step 9: Based on the given query samples, use the trained feature extraction network to search the target image database.

2. The cross-modal person re-identification method based on semi-supervised learning according to claim 1, characterized in that, In step 3, the image pair selection rate is calculated using the contour coefficients of the real and pseudo markers in the labeled pedestrian images. Specifically, this includes: (1) (2) (3) (4) (5) in, Indicates the selection rate of image pairs. Let represent the t-th labeled visible light pedestrian image. ; Let t represent the t-th unmarked visible light pedestrian image. ; Let represent the t-th marked infrared pedestrian image. ; This represents the t-th unmarked infrared pedestrian image. ; Representing an image The average distance to other images within the same cluster. Representing an image The average minimum distance to all images in different clusters, This represents the mean contour coefficient of all sample images in the input sample. This indicates that the mean value is being calculated.

3. The cross-modal person re-identification method based on semi-supervised learning according to claim 1, characterized in that, Also includes: Calculate the similarity between any labeled visible light pedestrian image and infrared pedestrian image, and perform cross-modal image pair matching on two pedestrian images with a similarity value not less than a second similarity threshold to obtain a set of labeled image pairs; The mean similarity of all image pairs in the labeled image pair set is calculated and used as the first similarity threshold.

4. The cross-modal person re-identification method based on semi-supervised learning according to claim 2, characterized in that, Step 6 specifically includes: (7) (8) in, and This indicates that based on the sample pairs in R In the The features of the mixed samples updated in the next iteration. , , and These represent the features corresponding to the infrared pedestrian image query samples and the features corresponding to the visible light pedestrian image query samples extracted by the feature extraction network, respectively. It is the momentum renewal factor. Indicates the number of iterations.

5. The cross-modal person re-identification method based on semi-supervised learning according to claim 1, characterized in that, Step 7 specifically includes: At the beginning of each epoch, an image is randomly selected from each cluster in the second clustering result to construct a subset dataset; Feature extraction is performed on the subset of data using the current network model to obtain a feature set. ,in and These represent the number of cluster categories and the feature dimension corresponding to the second clustering result, respectively. For each cluster From its front A set is formed by randomly selecting an image from the closest similar classes. and construct a graph , Represents the vertices of the graph. Denotes the edges in the graph; where, , , This indicates the number of pseudo-labeled classes to be sampled in each mini-batch; Will Each cluster in As an anchor point, retrieval exist All the connection classes, then the anchor class Its closest connection class Combined to form a set And thus obtain ; against From all classes, randomly select K instances of each class to generate a... A small batch of data from each sample is used as the training set for each round in the current epoch.

6. The cross-modal person re-identification method based on semi-supervised learning according to claim 4, characterized in that, The feature extraction network used is the ResNet50 network.

7. A cross-modal person re-identification method based on semi-supervised learning according to claim 6, characterized in that, In step 8, a supervised loss is designed for the labeled pedestrian images, specifically including: Calculate the cross-entropy loss between the predicted value and the true label according to formula (9); (9) Where n and m represent the number of visible light pedestrian images and infrared pedestrian images in the current batch, respectively. and Let i and j represent the feature vectors corresponding to the i-th visible light pedestrian image and the i-th visible light pedestrian image, respectively. Feature vectors corresponding to infrared pedestrian images and They represent and The corresponding real marker, and Indicates that they are in the same classifier Down and Identity prediction.

8. A cross-modal person re-identification method based on semi-supervised learning according to claim 7, characterized in that, In step 8, a class center loss is designed for unlabeled pedestrian images, specifically including: The class center loss of infrared pedestrian images is designed according to formulas (10) to (12); (10) (11) (12) in, This represents the aggregated class center features corresponding to infrared pedestrian images in the class center memory and the infrared pedestrian image query samples. Class center loss between This represents the mixture of class center features of the infrared pedestrian images in the class center memory and the infrared pedestrian image query samples. Class center loss between This indicates the relationship between the query sample and the class central memory. The feature vector of the corresponding positive sample of the class, This indicates the relationship between the query sample and the class central memory. The updated feature vector of the corresponding class, where K represents the number of classes. Represents the weight parameters. Indicates hyperparameters; The class center loss for visible light pedestrian images is designed according to formulas (13) to (15); (13) (14) (15) in, This represents the aggregated class center features corresponding to visible light pedestrian images in the class center memory and the visible light pedestrian image query samples. Class center loss between This represents the class center features of the mixed samples corresponding to visible light pedestrian images in the class center memory and the visible light pedestrian image query samples. Class center loss between This indicates the relationship between the query sample and the class central memory. The feature vector of the corresponding positive sample of the class, This indicates the relationship between the query sample and the class central memory. The feature vector of the updated sample corresponding to the class; according to and The design yielded a total class center loss. .

9. A cross-modal person re-identification method based on semi-supervised learning according to claim 8, characterized in that, In step 8, the network is optimized by comprehensively utilizing the supervision loss and class center loss, specifically including: (16)。 10. A cross-modal person re-identification system based on semi-supervised learning, characterized in that, include: The pseudo-labeled cross-modal data generation module and the pseudo-labeled cross-modal data refinement module; A class-centered memory is provided in the pseudo-marker cross-modal data refinement module; The pseudo-labeled cross-modal data generation module is used to extract features from labeled pedestrian images using a feature extraction network to obtain labeled data features; to cluster the labeled pedestrian images according to the labeled data features to obtain the first clustering result and pseudo-labels corresponding to each cluster; and to calculate the image pair selection rate using the contour coefficients of the real labels and pseudo-labels of the labeled pedestrian images. Based on the characteristics of unlabeled data, unlabeled pedestrian images are clustered to obtain the second clustering result and the pseudo-labels corresponding to each cluster. The class center features of each cluster are saved to the class centers in the class center memory. Calculate the similarity between any unlabeled visible light pedestrian image and infrared pedestrian image. Perform cross-modal image pair matching on two pedestrian images with a similarity value not less than a first similarity threshold to obtain an unlabeled image pair set R. Select unlabeled image pairs with higher similarity values ​​from the unlabeled image pair set R according to the image pair selection rate, thereby updating the mixed samples in the class center memory. The pseudo-labeled cross-modal data refinement module is used to extract features from unlabeled pedestrian images using a feature extraction network to obtain unlabeled data features; The second clustering results are sampled to select the most similar cluster. Difficult samples are used as the training set; a supervised loss is designed for labeled pedestrian images, and a class center loss is designed for unlabeled pedestrian images. The supervised loss and the class center loss are used together to optimize the feature extraction network. The pedestrian image to be tested is input into a trained feature extraction network to determine whether the target pedestrian exists in it.

Images