Unsupervised pedestrian re-identification method based on cross-camera channel information interaction and space-time semantic perception

Abstract

The invention discloses an unsupervised pedestrian re-identification method based on cross-camera channel information interaction and space-time semantic perception. The method mainly comprises the following steps: inputting an unsupervised pedestrian re-identification data set into a cross-camera joint learning framework; in the cross-camera joint learning framework, local features in cameras and cross-camera global features are extracted, the local features and the cross-camera global features are combined to capture a dependency relationship of cross-camera feature distribution, and a pseudo tag is generated; a relation between channels is modeled through a multi-scale channel interactive attention module, the distinction degree of identity related characteristics is improved, and background interference is suppressed; a space-time label punishment mechanism is used, label distribution is refined in combination with time information, and robustness of pseudo label uncertainty is improved; constructing a loss function to iteratively adjust network parameters so as to optimize the model; in the training stage, similarity comparison is carried out on the inquired pedestrian image and the image library, and the image of the same pedestrian is searched. According to the method, the problem of label-free cross-camera difference of pedestrian images is effectively solved, and the robustness of pedestrian re-identification is enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of pedestrian re-identification, and in particular to an unsupervised pedestrian re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception. Background Art

[0002] As an indispensable part of the social public security management system, the intelligent video surveillance system plays an indispensable role in maintaining social security and stability. As the amount of data generated by surveillance videos is growing exponentially, the traditional method of relying on manual monitoring cannot meet the real-time processing needs of massive video data. Therefore, intelligent analysis technology has emerged and is widely used in public security, traffic management, retail analysis and other fields. With the continuous growth of data volume, how to efficiently store this huge amount of data and quickly retrieve the required information has become a problem that needs to be solved.

[0003] The goal of person re-identification is to find images of people with a given identity in a large-scale image library and determine whether the pedestrians in different perspectives, different cameras, and different video clips are the same person. The acquisition of labeled data will result in a lot of labor and computational costs. Therefore, supervised person re-identification cannot be extended to large-scale datasets. In order to solve the scalability problem of supervised person re-identification, researchers have begun to focus on unsupervised person re-identification in recent years. Unsupervised person re-identification aims to achieve accurate re-identification by utilizing the potential structure and patterns between data without the need for labeled data.

[0004] Existing unsupervised person re-identification methods directly perform clustering algorithms on target features and assign pseudo labels to images. They then roughly divide images into classes based on the clustering measure of cross-camera feature similarity. However, they ignore the feature distribution differences caused by the gap in camera domains, which leads to inevitable performance degradation.

[0005] In person re-identification, images contain not only the features of the pedestrians, but also the background, lighting and noise information. The use of attention mechanism can enhance the extraction of identity-related features in unsupervised person re-identification while suppressing background interference. Luo et al. designed a novel cross-channel attention module that focuses on the maximum pooling features and average pooling features and generates channel weights through convolutional layers. The feature enhancement module in PGAL improves the feature representation of non-keypoint areas of the human body through multi-head self-attention. However, these methods only focus on the channel features or spatial features of a single scale of pedestrian images, without paying attention to these features at the same time.

[0006] Confidence-based sample filtering methods use the similarity between samples and their pseudo-label centers as confidence scores, and remove samples with low confidence or assign them lower weights. Miao et al. use confidence to guide pseudo-labels, which allows samples to be close not only to the originally assigned centroids, but also to other centroids that may embed their identity information. Wang et al. proposed self-consistency constraints to soften label noise in a single model, reducing the amount of computation and network parameter costs. Modeling the relationship between samples can better capture the similarities between samples of the same type and the distinguishability between heterogeneous samples, thereby optimizing the pseudo-label generation and feature learning process. These methods have alleviated the uncertainty problem of pseudo-labels to a certain extent, but they often ignore the potential constraints of temporal information on the quality of pseudo-labels.

[0007] Therefore, in the field of person re-identification, although research based on deep learning has made significant progress in improving model accuracy, current research often ignores the camera differences caused by changes in perspective, lighting, and background in practical applications, which increases the difficulty of person re-identification, and the accuracy of existing cross-camera unsupervised person re-identification models is relatively poor. Therefore, a method focusing on cross-camera unsupervised person re-identification is urgently needed to solve the above problems. Summary of the invention

[0008] The object of the present invention is to provide an unsupervised pedestrian re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception, comprising the following steps:

[0009] S1: Obtain an unsupervised person re-identification dataset and input it into the cross-camera joint learning framework;

[0010] S2: In the cross-camera joint learning framework, local features within the camera and global features across cameras are extracted simultaneously to generate consistent feature representations inside and outside the camera. The local and global features are combined to capture the dependency of feature distribution across cameras and generate pseudo labels.

[0011] S3: In the multi-scale channel interaction attention module, the feature expression is optimized by modeling the relationship between channels, focusing on the distinguishability of identity-related features and suppressing background interference;

[0012] The multi-scale channel interaction attention module includes channel grouping, channel shuffling, multi-scale channel interaction branch and multi-scale space branch; the multi-scale channel interaction branch uses multi-scale convolution to interact channel information between multi-scale branches, and each channel combines information from other channels; the multi-scale space branch uses multi-scale convolution combined with spatial attention mechanism to effectively integrate spatial information of different receptive fields and enhance the model's perception of local key areas and global feature distribution of pedestrians; finally, channel shuffling is used to model and interact with all inter-channel relationships;

[0013] S4: Use the spatiotemporal label penalty mechanism to combine time information to give samples a more detailed label distribution;

[0014] The spatiotemporal label penalty mechanism includes intra-camera time constraints and inter-camera time and number constraints, and the final matching confidence is generated by combining the two constraints; using the intra-camera time constraint, for pedestrians belonging to the same camera, the matching confidence is adjusted according to the proximity of the pseudo timestamps; using the inter-camera time and number constraints, for pedestrians from different cameras, the matching confidence is adjusted according to the time interval and camera number information; the intra-camera and inter-camera constraints are combined to generate the final matching confidence, and the matching confidence is integrated into the pseudo label generation process to optimize the label distribution;

[0015] S5: Iteratively adjust network parameters by designing loss functions to optimize model performance;

[0016] S6: In each training stage, the queried pedestrian image is compared with the pedestrian images in the image library for similarity to find images of the same pedestrian.

[0017] Furthermore, the overall framework includes a cross-camera joint learning framework; a backbone network and multi-scale channel interaction attention module; and a spatiotemporal label penalty mechanism.

[0018] Furthermore, the cross-camera joint learning framework includes an intra-camera local feature extraction branch and a cross-camera global feature extraction branch; the cross-camera global feature extraction branch performs feature extraction on the input images of all cameras and the segmented images to generate global features; the intra-camera local feature extraction branch performs feature extraction on the pedestrian images in each camera; and the samples are clustered according to the extracted features to generate pseudo labels.

[0019] Furthermore, the multi-scale channel interaction branch encodes channel features through multi-scale convolution operations of 1x1 convolution and 3x3 convolution and a channel interaction module to obtain the relationship between channels.

[0020] Furthermore, the channel interaction module includes segmentation, average pooling, GN, Sigmoid, matrix multiplication, concatenation and element-level addition.

[0021] Furthermore, the multi-scale spatial branch enhances the model's perception of local key areas and global feature distribution of pedestrians through multi-scale convolution operations of 1x1 convolution and 3x3 convolution and spatial attention modules, including GN, ReLU, Sigmoid and element-level addition.

[0022] Furthermore, the loss function is calculated by using cross entropy loss and triplet loss, and the obtained results are used to train pseudo label generation.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] The present invention proposes a novel cross-camera joint learning framework, which can simultaneously combine the local features within the camera and the global features between cameras to capture the global dependencies of the cross-camera feature distribution; the present invention proposes a novel multi-scale channel interaction attention module, which can model the relationship between channels, optimize feature expression, and improve the robustness of the model in complex scenes; the present invention proposes a novel spatiotemporal label penalty mechanism, which improves the stability and accuracy of the training process by refining the label distribution and semantic relationship modeling; the present invention verifies the effectiveness and superiority of the method of the present invention on two unsupervised pedestrian re-identification datasets Market1501 and MSMT17. Comprehensive evaluation indicators are used to evaluate the model accuracy: including mAP, Rank-1, Rank-5 and Rank-10. The experimental results under four indicators fully demonstrate the effectiveness of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings:

[0026] Figure 1 A flowchart of the steps of an unsupervised pedestrian re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception provided by the present invention;

[0027] Figure 2 The overall framework diagram of an unsupervised pedestrian re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception provided by the present invention;

[0028] Figure 3 A schematic diagram of the structure of a cross-camera joint learning framework according to a preferred embodiment of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a multi-scale channel interaction attention module of a preferred embodiment of the invention;

[0030] Figure 5 A flowchart of a spatiotemporal label penalty mechanism of a preferred embodiment provided for the invention. DETAILED DESCRIPTION

[0031] In order to make the technical personnel understand the present invention more clearly, the following is a related description in conjunction with specific examples. The specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0032] like Figure 1As shown, it is a flowchart of the steps of an unsupervised pedestrian re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception provided by the present invention, including:

[0033] S1: Obtain an unsupervised person re-identification dataset and input it into the cross-camera joint learning framework;

[0034] S2: In the cross-camera joint learning framework, local features within the camera and global features across cameras are extracted simultaneously to generate consistent feature representations inside and outside the camera. The local and global features are combined to capture the dependency of feature distribution across cameras and generate pseudo labels.

[0035] S3: In the multi-scale channel interaction attention module, the feature expression is optimized by modeling the relationship between channels, focusing on the distinguishability of identity-related features and suppressing background interference;

[0036] The multi-scale channel interaction attention module includes channel grouping, channel shuffling, multi-scale channel interaction branch and multi-scale space branch; the multi-scale channel interaction branch uses multi-scale convolution to interact channel information between multi-scale branches, and each channel combines information from other channels; the multi-scale space branch uses multi-scale convolution combined with spatial attention mechanism to effectively integrate spatial information of different receptive fields and enhance the model's perception of local key areas and global feature distribution of pedestrians; finally, channel shuffling is used to model and interact with all inter-channel relationships;

[0037] S4: Use the spatiotemporal label penalty mechanism to combine time information to give samples a more detailed label distribution;

[0038] The spatiotemporal label penalty mechanism includes intra-camera time constraints and inter-camera time and number constraints, and the final matching confidence is generated by combining the two constraints; using the intra-camera time constraint, for pedestrians belonging to the same camera, the matching confidence is adjusted according to the proximity of the pseudo timestamps; using the inter-camera time and number constraints, for pedestrians from different cameras, the matching confidence is adjusted according to the time interval and camera number information; the intra-camera and inter-camera constraints are combined to generate the final matching confidence, and the matching confidence is integrated into the pseudo label generation process to optimize the label distribution;

[0039] S5: Iteratively adjust network parameters by designing loss functions to optimize model performance;

[0040] S6: In each training stage, the queried pedestrian image is compared with the pedestrian images in the image library for similarity to find images of the same pedestrian.

[0041] like Figure 2As shown in the figure, it is the overall framework diagram of an unsupervised pedestrian re-identification method based on cross-camera channel information interaction and spatio-temporal semantic perception provided by the present invention. It mainly includes three modules, namely, a cross-camera joint learning framework, which includes an in-camera local feature extraction branch and a cross-camera global feature extraction branch, and a multi-scale channel interaction attention module, which includes a backbone network, a multi-scale channel interaction attention module, and a spatio-temporal label penalty mechanism.

[0042] The present invention provides a preferred embodiment to execute S2, simultaneously extracting the local features within the camera and the global features across cameras, generating a feature representation that is consistent inside and outside the camera, combining the local and global features, capturing the dependency relationship of the cross-camera feature distribution, and generating pseudo-labels. As Figure 3 shown, it includes the following steps:

[0043] S2-1: Based on the feature map, it is divided into multiple local regions and vertically divided into n blocks. Applying multi-scale channel interaction attention to each region separately can be expressed as:

[0044]

[0045] where F gi represents the i-th region after segmentation.

[0046] Generate optimized local features, and this process can be expressed as:

[0047]

[0048] Pool and splice the local features to generate an overall local feature representation:

[0049]

[0050] S2-2: Pass all input images X through the backbone network ResNet to extract the feature map F g , which is expressed as follows:

[0051] F g = ResNet(X),

[0052] where C is the number of channels, and H, W are the spatial dimensions.

[0053] Apply multi-scale channel interaction attention to the feature map to model the inter-channel and spatial relationships and generate the optimized global feature F' g , and the formula is as follows:

[0054] F' g = MSCI(F g ),

[0055] Among them, MSCI stands for multi-scale channel interaction attention.

[0056] S2-3: For each pedestrian image X in the camera i After the backbone network ResNet, extract the feature map F ci , which is expressed as follows:

[0057]

[0058] in, C is the number of channels, H and W are the spatial dimensions, and i is the camera number.

[0059] Apply multi-scale channel interaction attention on the feature map to model the relationship between channels and space, and generate optimized in-camera local features F' ci , the formula is as follows:

[0060]

[0061] Among them, MSCI stands for multi-scale channel interaction attention.

[0062] S2-4: In the in-camera local feature extraction branch, pseudo labels are generated by clustering the samples of each camera in each camera; in the cross-camera global feature extraction branch, all samples of all cameras are clustered to generate pseudo labels.

[0063] The present invention provides a preferred embodiment to perform S3, optimize feature expression by modeling the relationship between channels, improve the discrimination of identity-related features and suppress background interference. Figure 4 As shown, the following steps are included:

[0064] S3-1: First, each channel of the feature map is grouped; secondly, two branches are used to perform multi-scale channel interaction on the sub-features.

[0065] S3-2: In the multi-scale channel interaction branch, 1x1 convolution and 3x3 convolution are used for multi-scale operations. In the 1x1 convolution, one-dimensional global average pooling is used to encode the channels along the width and height directions respectively, and the spatial distribution characteristics of each channel are obtained for channel interaction. At the same time, combined with 3x3 convolution, through the interaction between multi-scale branches, each channel can combine information from other channels. The formula is as follows:

[0066] f 1 =S(GN(Conv 1×1 (Concat(HAP(F),WAP(F))))),

[0067] f 3 =Conv 3×3 (F),

[0068] F 1 =mat(AP(f 1 ),Softmax(f 3 )),

[0069] F 2 =mat(AP(f 3 ),Softmax(f 1 )),

[0070] F' c =F 1 +F 2 ,

[0071] F C =F·F' c .

[0072] S3-3: In the multi-scale spatial branch, multi-scale convolution is combined with the spatial attention mechanism to effectively integrate information from different receptive fields and enhance the model's ability to perceive the local key areas and global feature distribution of pedestrians. The formula is as follows:

[0073] f s3 =SA(Conv 3×3 (F)),

[0074] f s =SA(F),

[0075] F S =F·(f s3 +f s ),

[0076] S3-4: Finally, the obtained feature map is subjected to channel shuffling operation as follows:

[0077] F'=Shuffle(Concat(F C +F S )).

[0078] The present invention provides an embodiment to perform S4, combining time information to give samples a more detailed label distribution. Figure 5 As shown, the following steps are included:

[0079] S4-1: Intra-camera time constraint: Assume that two images i and j belong to the same camera, camID i =camID j , then according to the pseudo timestamp t i and t j The degree of closeness adjusts the matching confidence:

[0080]

[0081] Among them, γ>0 controls the speed of time decay. The image pairs close in time, i.e. |t i -t j The smaller the |, the higher the matching confidence will be.

[0082] S4-2: Time and number constraints between cameras: Assume that two images i and j come from different cameras, i.e., camID i ≠camID j , whose matching confidence is affected by the time interval and camera number differences:

[0083]

[0084] Where, δ>0: cross-camera time interval attenuation coefficient. κ: allowed camera ID difference threshold. When the camera ID difference is large, that is, camID i -camID j >κ, directly set the matching confidence to zero to suppress possible false matches.

[0085] S4-3: Integrate intra-camera and inter-camera constraints to generate the final matching confidence:

[0086]

[0087] Incorporate the matching confidence S(i,j) into the pseudo-label generation process to optimize the label distribution. Combined with the initial pseudo-label And confidence S(i,j), update the pseudo label, and decide whether to accept the pseudo label based on the confidence. It can be expressed as:

[0088]

[0089] where ∈ R is the neighborhood set of image i and ΙΙ(·) indicates the function.

[0090] The present invention provides an embodiment to perform S5. In this embodiment, a loss function including a cross entropy loss and a triplet loss is used for calculation, and the result obtained is used for training of pseudo label generation, including the following steps:

[0091] S5-1: Construct the in-camera loss function. This loss function is used to optimize the local features in the camera, and is implemented using cross entropy loss. It can be expressed as:

[0092]

[0093] Among them, p i is the pedestrian prediction score of the local feature, y i is the pseudo label value of the pedestrian, and n represents the number of samples.

[0094] S5-2: Construct the inter-camera loss function. This loss function is used to optimize the global features across cameras and is implemented using triplet loss. It can be expressed as:

[0095]

[0096] Where d is the distance function between features and ɑ is the margin parameter of the triplet loss.

[0097] S5-3: Construct a penalty loss function. This loss function is used to reduce the uncertainty of pseudo labels and is implemented using cross entropy loss. It can be expressed as:

[0098]

[0099] Among them, p i is the pedestrian prediction score of the global feature, is the pseudo label value of the pedestrian, and n represents the number of samples.

[0100] S5-4: Construct a classification loss function. This loss function is used to optimize the final classification result and is implemented using cross entropy loss. It can be expressed as:

[0101]

[0102] Among them, p i is the final pedestrian prediction score, y i is the pseudo label value of the pedestrian, and n represents the number of samples.

[0103] S5-5: Construct a joint loss function. This loss function is the sum of the above loss functions. It can be expressed as:

[0104] L Joint =L Intra +L Inter +L penalty +L Classifier .

[0105] In this embodiment, the proposed method is compared with other advanced methods on two unsupervised person re-identification datasets Market1501 and MSMT17, and the commonly used evaluation indicators are selected: mAP, Rank-1, Rank-5 and Rank-10. The experimental results are shown in Table 1.

[0106] Table 1 Comparison of the proposed methods on Market1501 and MSMT17 datasets

[0107]

[0108] As shown in Table 1, this embodiment is ahead of the existing methods in multiple indicators on the two data sets, which proves the effectiveness of the method of this embodiment.

[0109] The above is a detailed description of a specific embodiment of the present invention, but the present invention is not limited to the above specific implementation. Those skilled in the art can make various deformations or modifications to the above implementation without affecting the essential content of the present invention, which should all be within the protection scope of the present invention.

Claims

1. An unsupervised person re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception, characterized in that The following steps are involved: S1: Obtain the unsupervised person re-identification dataset and input it into the built model; S2: In the cross-camera joint learning framework, local features within the camera and global features across cameras are extracted simultaneously to generate consistent feature representations inside and outside the camera. The local and global features are combined to capture the dependency of feature distribution across cameras and generate pseudo labels. S3: In the multi-scale channel interaction attention module, the feature expression is optimized by modeling the relationship between channels, focusing on the distinguishability of identity-related features and suppressing background interference; The multi-scale channel interaction attention module includes channel grouping, channel shuffling, multi-scale channel interaction branch and multi-scale space branch; The multi-scale channel interaction branch uses multi-scale convolution to interact channel information between multi-scale branches, and each channel combines information from other channels; The multi-scale spatial branch combines multi-scale convolution with the spatial attention mechanism to effectively integrate the spatial information of different receptive fields and enhance the model's perception of the local key areas and global feature distribution of pedestrians. Finally, channel shuffling is used to model and interact with the relationships between all channels. S4: Use the spatiotemporal label penalty mechanism to combine time information to give samples a more detailed label distribution; The spatiotemporal label penalty mechanism includes intra-camera time constraints and inter-camera time and number constraints, and the final matching confidence is generated by combining the two constraints; using the intra-camera time constraint, for pedestrians belonging to the same camera, the matching confidence is adjusted according to the proximity of the pseudo timestamps; using the inter-camera time and number constraints, for pedestrians from different cameras, the matching confidence is adjusted according to the time interval and camera number information; the intra-camera and inter-camera constraints are combined to generate the final matching confidence, and the matching confidence is integrated into the pseudo label generation process to optimize the label distribution; S5: Iteratively adjust network parameters by designing loss functions to optimize model performance; S6: In each training stage, the queried pedestrian image is compared with the pedestrian images in the image library for similarity to find images of the same pedestrian.

2. The unsupervised person re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception according to claim 1 is characterized in that The overall structure includes a cross-camera joint learning framework; a backbone network and a multi-scale channel interaction attention module; and a spatiotemporal label penalty mechanism.

3. The unsupervised person re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception according to claim 1 is characterized in that: The cross-camera joint learning framework includes an intra-camera local feature extraction branch and a cross-camera global feature extraction branch; The cross-camera global feature extraction branch extracts features from the input images of all cameras and the segmented images to generate global features; the in-camera local feature extraction branch extracts features from the pedestrian images in each camera; and the samples are clustered according to the extracted features to generate pseudo labels.

4. The unsupervised person re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception according to claim 1 is characterized in that: The multi-scale channel interaction branch encodes channel features through multi-scale convolution operations of 1x1 convolution and 3x3 convolution and a channel interaction module to obtain the relationship between channels.

5. The unsupervised person re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception according to claim 4 is characterized in that: The channel interaction module includes segmentation, average pooling, GN, Sigmoid, matrix multiplication, concatenation and element-level addition.

6. The unsupervised person re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception according to claim 1 is characterized in that: The multi-scale spatial branch enhances the model's perception of local key areas and global feature distribution of pedestrians through multi-scale convolution operations of 1x1 convolution and 3x3 convolution and spatial attention modules, including GN, ReLU, Sigmoid and element-level addition.

7. The unsupervised person re-identification method based on cross-camera channel information interaction and spatiotemporal semantic perception according to claim 1 is characterized in that: The loss function is calculated by using cross entropy loss and triplet loss, and the obtained results are used for training of pseudo label generation.

Images