A cross-resolution pedestrian re-identification method based on double-flow input feature reconstruction

Abstract

The application discloses a cross-resolution pedestrian re-identification method based on double-flow input feature reconstruction, and relates to the fields of computer vision and artificial intelligence. First, a double-flow input and feature reconstruction module is constructed to adaptively reconstruct the features of low-resolution images, and a feature degradation loss and a reconstruction loss are combined with a light residual decoder to learn the feature reconstruction capability; second, a weight-shared twin network structure is constructed in combination with a feature enhancement module to obtain deep invariant features of pedestrians; finally, a maximum average fusion pooling operation is used to pool the deep invariant features to obtain vectorized representation, and a cross-resolution triplet loss, a cross-resolution center loss and an identity loss are used to constrain the distribution of pedestrian features and improve the cross-resolution matching precision. The application is mainly applied to a video monitoring intelligent analysis system and has broad application prospects in the fields of intelligent security and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to a cross-resolution person re-identification method based on double-flow input feature reconstruction, and a cross-resolution person re-identification problem in the field of intelligent video monitoring, and belongs to the field of computer vision and security monitoring. BACKGROUND

[0002] Person Re-Identification (ReID) is a task of great practical significance and challenge, which aims to identify the same target person under different monitoring camera perspectives. In recent years, due to its practicability in the fields of person tracking and monitoring, ReID has become a research hotspot in the field of computer vision. Although the person re-identification method based on deep learning has made significant progress in human posture changes, background confusion and partial occlusion, and has shown performance close to or even exceeding human level in many public benchmark tests, most of these methods assume that the resolution of the query image and the image library is comparable. However, in practical applications, image resolution varies due to the differences in camera performance and the changes in the distance between the camera and the target person, resulting in images of the same person at different resolution levels, which makes it difficult for current research to effectively deal with the resolution mismatch problem existing in the person images in real application scenarios. To solve this problem, cross-resolution person re-identification (CRReID) has emerged as a new research branch.

[0003] The challenges of CRReID mainly lie in two aspects: first, the blurred appearance of low-resolution pedestrian images often leads to the loss of key discriminative information. Second, due to the different resolutions of the collected images, pedestrians may present different fine-grained and detailed textures, so that the features extracted from the same pedestrian image at different resolutions may have obvious mismatches. To solve this problem, in recent years, many studies have proposed different cross-resolution pedestrian re-identification algorithms, which can be generally divided into two strategies: a) image resolution reconstruction and b) resolution-invariant feature extraction. Strategy a) uses super-resolution reconstruction technology to reconstruct low-resolution (LR) images to solve the resolution mismatch problem in cross-resolution pedestrian re-identification, and uses the super-resolution image as a bridge to match the real high-resolution (HR) image. However, the general super-resolution method aims to improve the visual fidelity of the image, and direct combination with the ReID network may have compatibility problems. Strategy b) notes the compatibility problem of super-resolution technology with pedestrian re-identification, and tries to learn a pedestrian feature representation that is invariant or robust to resolution to achieve cross-resolution pedestrian comparison. Although this framework pays attention to information recovery, this method of forcibly using the same ReID network to extract invariant representations at different resolutions can hardly guarantee effective information recovery and still has a high risk of losing detailed information. In summary, existing research either has compatibility problems or cannot effectively recover lost information, so it is necessary to develop a method that effectively recovers image information while solving the compatibility problem of super-resolution technology and organically combines super-resolution technology into the field of CRReID. SUMMARY

[0004] To solve the limitations of the prior art, the present application proposes a cross-resolution pedestrian re-identification method based on double-flow input feature reconstruction, which aims to complete feature reconstruction at the shallow level of the framework and obtain depth-invariant features at the deep level of the framework. A double-flow input structure is designed for feature reconstruction of low-resolution images, and an image feature enhancement module is used to obtain depth-invariant features for cross-resolution pedestrian matching, so that image information is effectively recovered, and the distribution of pedestrian features at different resolutions is closer, improving the accuracy of cross-resolution pedestrian re-identification.

[0005] The application adopts the following technical scheme: a cross-resolution pedestrian re-identification method based on double-flow input feature reconstruction, which comprises the following steps:

[0006] (1) Construct a cross-resolution pedestrian image pair. First, obtain a batch of HR images from the standard data set according to the pedestrian identity label, then randomly downsample the HR images with a scale factor of {2, 3, 4}, and use the obtained LR images and HR images as image pairs for training;

[0007] (2) input the pedestrian image (pair) into the double-flow reconstruction network, different input branches are selected according to the resolution of the image in the process, the low-resolution branch will adaptively reconstruct the features of the low-resolution image, two branches get and In the training process, the feature degradation loss and the reconstruction loss are used to learn the feature reconstruction capability in combination with the lightweight residual decoder (RLDecoder);

[0008] (3) input and into the weight-shared twin network structure, first pass through the fourth five residual blocks of the backbone to obtain the preliminary embedded resolution-invariant features Then input it into the feature enhancement module (MGLIF) to obtain pedestrian depth-invariant features of different scales and different granularities and

[0009] (4) use the max average pooling operation to pool and to obtain the vectorized pedestrian representation corresponding to different features and And splice them as the final feature vector used for pedestrian matching, and in the training process, the cross-resolution triplet loss, cross-resolution center loss and identity loss are used to constrain the distribution of pedestrian features, and the distance of different resolution image features of the same pedestrian is reduced.

[0010] Compared with the prior art, the beneficial effects of the present application are:

[0011] 1. The present application introduces a low-resolution feature reconstruction strategy, which uses the feature information and resolution information of high-resolution images to constrain the reconstruction process at the shallow layer of the backbone, so that the information loss in the finally obtained depth-invariant features is smaller;

[0012] 2. The present application adds a pedestrian feature enhancement module to model and fuse the features of pedestrians at multiple levels and different scales, and obtain more discriminative pedestrian representations;

[0013] 3. The present application introduces a new pooling operation, which can make more full use of the information in the feature map. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 It is the network structure diagram of the present application based on double-flow input feature reconstruction cross-resolution pedestrian re-identification;

[0015] Figure 2 It is the lightweight residual decoder in the feature reconstruction of the present application;

[0016] Figure 3 Feature enhancement module in invariant feature learning of the application;

[0017] Figure 4 Maximum average fusion pooling module in feature vectorization of the application; DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the application more clear and understandable, the application will be further described in detail below in combination with the drawings. It should be understood that the specific embodiments described herein are only used to explain the application and not to limit the application.

[0019] As shown in Figure 1 A cross-resolution pedestrian re-identification method based on double-flow input feature reconstruction, comprising the following steps:

[0020] (1) Constructing a cross-resolution pedestrian image pair, first obtaining a batch of HR images from the standard data set according to the pedestrian identity label, then randomly down-sampling the HR images with a scale factor of {2, 3, 4}, and taking the obtained LR images and HR images as the image pair used for training;

[0021] (2) Input the pedestrian image (pair) into the double-flow reconstruction network, in which different input branches will be selected according to the resolution of the image, the low-resolution branch will adaptively reconstruct the features of the low-resolution image, and the two branches will obtain and In the training process, feature degradation loss and reconstruction loss are used to learn the feature reconstruction capability in combination with a light residual decoder (RLDecoder);

[0022] (3) Input and into a weight-shared twin network structure, first pass through the fourth and fifth residual blocks of the backbone to obtain the preliminary embedded resolution-invariant features Then input it into the feature enhancement module (MGLIF) to obtain pedestrian deep-invariant features of different scales and different granularities and

[0023] (4) Use the maximum average fusion pooling operation to pool and to obtain the vectorized pedestrian representation corresponding to different features and These features are then concatenated to form the final feature vector for pedestrian matching. During the training process, cross-resolution triplet loss, cross-resolution center loss, and identity loss are used to constrain the distribution of pedestrian features and reduce the distance between the features of the same pedestrian in images of different resolutions.

[0024] The detailed steps are as follows:

[0025] Step (1): Obtain cross-resolution image pairs. From a standard pedestrian re-identification dataset, select P pedestrian identities, K images for each pedestrian, to obtain PK HR images. For each image, randomly select a ratio from the {2,3,4} scaling factors for downsampling to obtain PK LR images. After this operation, each HR image corresponds to one HR image, which facilitates subsequent feature reconstruction and identity representation learning.

[0026] Step (2): Input the image pair into the dual-stream input and feature reconstruction network.

[0027] For network structure, such as Figure 1 As shown in the "Dual-Stream Input and Feature Reconstruction" section, the backbone network is divided into 5 residual blocks {R1, R2, R3, R4, R5}. The output of the last layer of each residual block is defined as {f1, f2, f3, f4, f5}, where... And d is the number of channels in the corresponding feature map. Under this setting, the feature extraction and reconstruction function of the dual-stream input branch of this invention can be expressed as:

[0028]

[0029]

[0030] in The CBAM module enhances the feature extraction capability of the LR branch, making feature reconstruction more stable and reliable. Furthermore, the weights of the residual blocks in the HR and LR branches are independent and not shared. Given an HR image x... h And a corresponding LR image x l The outputs of the HR and LR input branches can be expressed as and To ensure that the LR branch has the ability to reconstruct and recover features, two losses are used to constrain the output features, namely the feature degradation loss. and reconstruction losses in Directly to f hr and f lr To impose constraints, and Indirect constraints through RL Decoder It can be represented as:

[0031]

[0032] where N denotes the number of HR and LR sample pairs in the current batch, i denotes the i-th pair of samples, and stopgrad denotes the "gradient stop" operation. The output f lr is calculated indirectly by the RL Decoder is denoted as follows:

[0033]

[0034] where is the RL Decoder module. Through the above operations, the output f lr of the dual-stream input branch can be made as close as possible to f hr , thereby achieving the restoration of the LR image information at the feature level.

[0035] As for the RL Decoder, a simple and lightweight decoder is more suitable for information restoration in the cross-resolution pedestrian re-identification task. The present application constructs a lightweight residual decoder (RL Decoder) to constrain the reconstruction of the LR image features, and the overall structure is as shown in Figure 2 The input feature map f lr is input into the lightweight convolution block and the f lr that has undergone dimension reduction is added, then bilinear interpolation up-sampling is performed, followed by nonlinear activation, and finally the feature map is up-sampled again and the image channels are restored through pixel shuffling to obtain the reconstructed image.

[0036] Step (3): Obtain the depth-invariant features using the weight-shared twin network.

[0037] For the network structure, as shown in the "Invariant Feature Learning and Embedding" section of Figure 1 This section is the depth-invariant feature extraction strategy proposed by the present application, which uses a weight-shared twin network to extract pedestrian features at different resolutions, so that the distribution of the final pedestrian features at different resolutions should be as similar as possible, with better matching performance. First, the 4th and 5th residual blocks of the backbone are used to obtain invariant features with similar distributions Then, the depth-invariant features are obtained using the feature enhancement module.

[0038] As for the feature enhancement module, as shown in Figure 3 , where the Bottleneck (denoted as ) is the basic convolution block of the backbone, and the Reduction (denoted as ) compress the feature map in the channel dimension. The bottleneck processes the final output feature of the backbone as high-level features, and the corresponding unprocessed features as low-level features. The high-level features are cut into pieces in the horizontal direction as local scale features, and the entire feature map is taken as the global scale feature. The low-level features are compressed in the channel dimension to obtain low-level global features:

[0039]

[0040] wherein is the input feature, i.e. the invariant feature is the obtained low-level global feature, C in and C glo are the channel number of the input feature and the compressed feature, respectively. The high-level features are processed to obtain high-level global features and local features. First, the high-level global feature f hl is obtained by compressing the channel dimension: ll

[0041]

[0042] As for the local relationship feature f loc , first cut the feature map along the horizontal direction, then splice along the channel dimension, and use a 1x1 convolution layer to establish the correlation between different parts of the pedestrian features, as follows:

[0043]

[0044] wherein represents the "cutting and splicing" operation, In order to establish the relationship between the global feature and the local feature, the present application uses a "stacking" operation to make the size consistent with the global feature, and then uses a 1x1 convolution layer to fuse f ll , f hl and f loc to establish the relationship between the global feature and the local feature:

[0045]

[0046] wherein Cat[·] represents splicing along the channel dimension, represents the stacking operation along the height direction of the feature map, is the result of the interaction and fusion of different levels and different scales of features.

[0047] Step (4): The obtained deep invariant feature is vectorized for pedestrian matching.

[0048] Regarding the max average pooling, as Figure 4 ​As shown, the adaptive fusion of the results of average pooling and maximum pooling is performed to compress and refine the feature map information. The adaptive average pooling (AAP) and adaptive maximum pooling (AMP) operations are used at the same time to ensure information retention and significant feature extraction, respectively. Finally, a 1x1 convolution layer is used for adaptive fusion to obtain the final feature vector Seq:

[0049]

[0050] where Seq t , t∈{ll, fus, loc, hl} respectively represent f ll , f fus , f loc , f hl corresponding feature vectors, n controls the output size of the adaptive pooling layer, C out controls the dimension of the final feature vector. The feature vector used for pedestrian matching by the model is obtained by fusing the feature vectors obtained from the corresponding resolution images, that is:

[0051]

[0052] Regarding the cross-resolution triplet loss The existing triplet loss only mines difficult triplets at a single resolution level, but cross-resolution pedestrian matching needs to pay special attention to the influence of the differences between the same pedestrians at different resolutions in addition to considering the differences between pedestrians within the resolution. In each training batch, P pedestrian categories are randomly selected, K HR and LR images are selected for each pedestrian category, and a total of 2PK images are selected. The formula is as follows:

[0053]

[0054] where D(·) is the Euclidean distance, [z] + is equivalent to max(z, 0), t represents different feature vectors, the anchor is selected from the set of HR and LR features, is a positive sample of the same category as the anchor, is a negative sample different from the anchor category, and the selection of all difficult positive and negative samples considers the feature space of HR and LR, rather than only selecting samples in the feature space of a single resolution.

[0055] Regarding the cross-resolution center loss The feature center of the existing center loss is established at a single resolution level. In cross-resolution pedestrian matching, the feature center should be established between the HR and LR feature spaces, so as to better constrain the distribution of pedestrian features, so that the pedestrian features at different resolution levels can be punished and the distribution of the pedestrian features is more compact, so as to better perform cross-resolution pedestrian matching, which is expressed as:

[0056]

[0057] wherein and are pedestrian features finally obtained by the model, is a processing function of the entire network model on the input y i is a label of an i-th image pair in a current batch, represents a center of a y i -th pedestrian image feature established between the HR and LR feature spaces.

[0058] Regarding the identity loss a cross-entropy label smoothing loss is adopted and is expressed as follows:

[0059]

[0060] wherein M represents a number of pedestrian identities in a training set, p(y pre ) represents a probability that a predicted label is y pre , and in addition, q(y pre ) is defined as:

[0061]

[0062] wherein y is a real label of an input image, and epsilon is a small constant (set as 0.1) and can prevent the re-identification model from overfitting on the training set, and a total loss of training is:

[0063]

[0064] wherein lambda is a balance weight of the center loss and is set as 0.001 in the application.

[0065] In order to verify the effectiveness of the method of the application, the application verifies three data sets commonly used in the field of cross-resolution pedestrian re-identification, namely CAVIAR, MLR-VIPeR and MLR-Market-1501, and selects four cross-resolution pedestrian re-identification methods based on deep learning as comparison methods, and the comparison methods are as follows:

[0066] Method 1: INTACT proposed by Cheng et al., reference “Cheng Z, Dong Q, Gong S, et al. Inter-Task Association Critic for Cross-Resolution Person Re-Identification [C] IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2020: 2602-2612.”

[0067] Method 2: PS-HRNet proposed by Zhang et al., reference “Zhang G, Ge Y, Dong Z, et al. Deep High-Resolution Representation Learning for Cross-Resolution Person Re-Identification [J]. IEEE Transactions on Image Processing, 2021, 30: 8913-8925.”

[0068] Method 3: JBIM proposed by Zheng et al., reference “Zheng W S, Hong J, Jiao J, et al. Joint Bilateral-Resolution Identity Modeling for Cross-Resolution Person Re-Identification [J]. International Journal of Computer Vision, 2022, 130(1): 136-156.”

[0069] Method 4: LRAR proposed by Wu et al., reference “Wu L Y, Liu L, Wang Y, et al. Learning Resolution-Adaptive Representations for Cross-Resolution Person Re-Identification [J]. IEEE Transactions on Image Processing, 2023, 32: 4800-4811.”

[0070] As shown in Table 1, the performance of the method proposed in the application on three data sets with Rank1, Rank5 and Rank10 as evaluation indexes has significant advantages compared with the other four methods.

[0071] Table 1 Comparison of Rank1, Rank5 and Rank10 indexes of the method with other methods

[0072]

[0073] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the application, but not to limit it; although the application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the application.

Claims

1. A cross-resolution person re-identification method based on dual-stream input feature reconstruction, characterized in that, Includes the following steps: (1) Construct cross-resolution pedestrian image pairs. First, obtain batch HR images from the standard dataset according to pedestrian identity labels. Then, randomly downsample the HR images with a scaling factor of {2,3,4}. Use the obtained LR images and HR images as image pairs for training. (2) Input the pedestrian image into the "dual-stream input and feature reconstruction" network. During this process, different input branches are selected according to the image resolution. The low-resolution branch will adaptively reconstruct the features of the low-resolution image. The two branches respectively obtain and Feature degradation loss was used during the training process. and reconstruction losses Combine a lightweight residual decoder RLDecoder to learn feature reconstruction capabilities; (3) and The input weight-shared Siamese network structure first passes through the fourth and fifth residual blocks of the backbone to obtain preliminary embedded resolution-invariant features. Then, it is input into the feature enhancement module to obtain pedestrian depth-invariant features of different scales and fine granularities. and (4) Use the maximum average fusion pooling operation to... and Pooling is performed to obtain vectorized pedestrian representations corresponding to different features. and These features are then concatenated to form the final feature vector for pedestrian matching. During the training process, cross-resolution triplet loss, cross-resolution center loss, and identity loss are used to constrain the distribution of pedestrian features and reduce the distance between the features of the same pedestrian in images of different resolutions.

2. The cross-resolution person re-identification method based on dual-stream input feature reconstruction according to claim 1, characterized in that, The "dual-stream input and feature reconstruction" network structure in step (2); The backbone network is divided into 5 residual blocks {R1, R2, R3, R4, R5}. The output of the last layer of each residual block is defined as {f1, f2, f3, f4, f5}, where... And d represents the number of channels in the corresponding feature map. Under this setting, the feature extraction and reconstruction function of the dual-stream input branch of this method can be expressed as: in The CBAM module enhances the feature extraction capability of the LR branch, making feature reconstruction more stable and reliable. Furthermore, the weights of the residual blocks in the HR and LR branches are independent and not shared. Given an HR image x... h And a corresponding LR image x l The output is and 3. The cross-resolution person re-identification method based on dual-stream input feature reconstruction according to claim 1, characterized in that, Feature degradation loss in step (2) and reconstruction losses To ensure that the LR branch has the ability to reconstruct and recover features, two losses are used to constrain the output features, namely the feature degradation loss. and reconstruction losses in Directly to f hr and f lr To impose constraints, and Indirect constraints through RL Decoder It can be represented as: Where N represents the number of HR and LR sample pairs in the current batch, i represents the i-th sample pair, stopgrad represents the gradient cutoff operation, and f is the output of the LR input branch. lr Indirect computation via RL Decoder It is expressed as follows: in This is the RL Decoder module.

4. The method according to claim 3, characterized in that, The reconstruction loss The network structure of the RLDecoder used in the computation is as follows: Input feature map f lr After lightweight convolutional blocks and dimensionality-reduced f lr The images are then summed, followed by bilinear interpolation upsampling, nonlinear activation, and finally pixel shuffling to achieve upsampling again and restore the image channels, thus obtaining the reconstructed image.

5. The method for cross-resolution person re-identification based on dual-stream input feature reconstruction according to claim 1, characterized in that, The feature enhancement module in step (3) has a network structure represented by Botteleneck as follows: Reduction is expressed as It consists of 1×1 convolutional layers, with Bottleneck as the basic convolutional block of the backbone, and Reduction compresses the feature map along the channel dimension; Bottleneck processes the input features to create high-level features, while the unprocessed features are used as low-level features. The high-level features are horizontally segmented into local-scale features, and the entire feature map is used as a global-scale feature map. The low-level features are then compressed by channel dimension to obtain the low-level global features. in For input features, To obtain the low-level global features, C in and C glo These represent the number of input feature channels and the number of compressed feature channels, respectively. High-level features undergo different processing to obtain high-level global features and local features. First, the high-level global feature f... hl Similar to f ll It is also obtained through channel dimension compression: As for the local relation feature f loc First, the feature map is sliced ​​horizontally, then stitched together along the channel dimension. A 1×1 convolutional layer is used to establish the correlation between features of different parts of the pedestrian, as shown below: in This indicates the "slicing and splicing" operation. A "stacked" operation was used to make its size consistent with the global features, and then a 1×1 convolutional layer was used to stack f. ll f hl and f loc This is achieved through fusion, thereby establishing its relationship with global features: Cat[·] indicates concatenation by channel dimension. This indicates a stacking operation along the height direction of the feature map. This is the result of interactive fusion of features at different levels and scales.

6. The cross-resolution person re-identification method based on dual-stream input feature reconstruction according to claim 1, characterized in that, In step (4), the maximum average pooling is used simultaneously with adaptive average pooling (AAP) and adaptive maximum pooling (AMP) operations to ensure information preservation and extraction of salient features, respectively. Finally, a 1×1 convolutional layer is used for adaptive fusion to obtain the final feature vector Seq: Where Seq t ,t∈{ll,fus,loc,hl} represent f respectively ll ,f fus ,f loc ,f hl The corresponding feature vector, n controls the output size of the adaptive pooling layer, C out Control the dimension of the final feature vector.

7. The method for cross-resolution person re-identification based on dual-stream input feature reconstruction according to claim 1, characterized in that, The cross-resolution triplet loss in step (4) Cross-resolution center loss In each training batch, P pedestrian categories are randomly selected, and K HR and LR images are selected for each pedestrian category, for a total of 2PK images. The formula is shown below: Where D(·) is the Euclidean distance, [z] + Equivalent to max(z,0), where t represents different eigenvectors and anchor points. Select from the set of HR and LR features. It is a positive sample of the same category as the anchor point. These are negative samples that are different from the anchor category. The selection of all difficult positive and negative samples takes into account the feature spaces of HR and LR, rather than just selecting samples in a single-resolution feature space. Regarding cross-resolution center loss The feature centers of each pedestrian class are established between the HR and LR feature spaces to better constrain the distribution of pedestrian features. Simultaneously, pedestrian features at different resolution levels are penalized to make their distribution more compact, thereby improving cross-resolution pedestrian matching. This is represented as: in and The pedestrian features obtained by the model at the end y is the processing function of the entire network model for the input. i It is the label of the i-th image pair in the current batch. The y-th element is established between the HR and LR feature spaces. i The center of pedestrian-like image features.

Images