A pedestrian re-identification method and system for damaged images

Abstract

This invention discloses a method and system for pedestrian re-identification of damaged images. The invention employs an original pedestrian re-identification network for damaged images, capable of outputting pedestrian representations that remain invariant to the original, undamaged images. The invention filters out style variations in the domain through smoothing style normalization and instance normalization in the restoration module. Simultaneously, the smoothing maximal unit enables this module to more fully extract pedestrian-related features from the deleted information and restore them to the network, mitigating style differences caused by damaged images. Furthermore, a global attention mechanism captures significant pedestrian features across three dimensions by focusing on the interaction between channels and space, reducing the loss of cross-dimensional information. Ultimately, this enables the model to more fully capture pedestrian representations that remain invariant to the original images when faced with damaged pedestrian images that do not lose their original semantic information.

Description

Technical Field

[0001] This invention belongs to the technical fields of computer vision technology, digital image processing, deep learning, etc., and relates to a pedestrian re-identification method and system for damaged images. In particular, it relates to a pedestrian re-identification method for damaged pedestrian re-identification images by extracting features through style normalization and global attention. Background Technology

[0002] Pedestrian re-identification (ReID) aims to retrieve specific pedestrians across cameras, time, and location. Thanks to the rapid development of deep learning, the field of computer vision has achieved a qualitative leap. Consequently, deep learning-based pedestrian re-identification has been widely applied in video surveillance, intelligent security, and unmanned supermarkets, becoming a popular research area.

[0003] Pedestrian images collected by researchers in real-world scenarios are easily corrupted by factors such as noise, blur, weather, and digital artifacts. Furthermore, current pedestrian re-identification models only consider performance on clean (normal) datasets during evaluation, neglecting pedestrian images under various corrupted conditions. Therefore, in 2019, Hendrycks et al. established two new benchmarks, ImageNet-C and ImageNet-P, to evaluate and improve model performance on corrupted images, providing new insights for pedestrian re-identification research. To further improve robustness to corrupted images, in 2020, Mu et al. proposed a data augmentation technique called AugMix, which significantly improved performance on ImageNet-C by mixing multiple augmented images. To comprehensively study the recognition capabilities of models on damaged pedestrian images, in 2021, Chen et al. first used a clean dataset and the ImageNet-C algorithm, introducing a new damage type—"rain"—to generate pedestrian damage datasets covering 20 common damage types in real life: Market1501-C, CUHK03-C, and MSMT17-C, to evaluate the robustness of the models. They established a new baseline, CIL, for pedestrian re-identification on damaged images by improving random erasure, BNNeck, and ID Loss. Although the model's performance on damaged images was significantly improved compared to previous mainstream models, the overall performance metrics remained low, showing a considerable gap compared to the clean dataset. For example, on the damaged test set of the Market1501 dataset, the mINP metric was only 1.76%, a difference of 56.14% compared to the clean test set. Meanwhile, activation functions play a crucial role in the training of person re-identification networks. However, the ReLU non-linear activation function directly sets the value to 0 in negative space, resulting in a zero derivative and causing parameters to fail to update, leading to neuron death during training. Therefore, in 2013, the LeakyReLU function was proposed, which updates parameters by giving a very small coefficient to make the derivative in negative space close to but not equal to zero. In 2017, Google proposed the Swish non-linear activation function to improve upon the aforementioned problems of ReLU. However, LeakyReLU and Swish are not very stable in practical use, resulting in poor performance when dealing with complex and damaged images.

[0004] In recent years, an increasing number of attention mechanisms have been applied to pedestrian re-identification to improve model performance. For example, Wang et al. used an encoder-decoder residual attention module to enhance the ability to extract pedestrian features; Sang et al. used channel attention and spatial attention mechanisms sequentially and ran BAM in parallel, which can extract significant pedestrian attention features from both channel and spatial dimensions. However, due to the reduction of information and the separation of dimensions, the module ignores the interaction between channel, spatial width, and spatial height, losing cross-dimensional information. Summary of the Invention

[0005] To effectively address the challenges of extracting significant pedestrian features and lacking "global" cross-dimensional information when dealing with complex and stylistically diverse damaged images, this invention proposes a pedestrian re-identification method and system based on style normalization and global attention for damaged images, building upon the CIL network.

[0006] The technical solution adopted by the method of the present invention is: a pedestrian re-identification method for damaged images, which inputs the damaged image into a damaged image pedestrian re-identification network to effectively extract pedestrian representations that are unchanged from the original undamaged image;

[0007] The damaged image pedestrian re-identification network uses ResNet50 as its backbone network. After each of the four layers of the backbone network, a Style Normalization and Restoration Module (SM-SNR) with Smooth Maximal Units (SMUs) is added. The SM-SNR is based on the Style Normalization and Restoration Module (SNR), and the ReLU layer after average pooling in the module is replaced with SMUs to form the Smooth Style Normalization and Restoration Module. After the last SM-SNR, a Global Attention Module (GAM) is set, and the pooling layer and fully connected layer at the end of ResNet50 are removed. The stride of the last layer, layer 4, is set to 1. The output is a pedestrian representation that remains unchanged from the original undamaged image.

[0008] The technical solution adopted by the system of the present invention is: a pedestrian re-identification system for damaged images, comprising:

[0009] One or more processors;

[0010] A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the pedestrian re-identification method for damaged images.

[0011] Compared with existing algorithms, the significant advantages and effects of this invention are as follows:

[0012] 1) The Smooth Style Normalization and Recovery Module SM-SNR proposed in this invention enables the network model to filter out interference from various damaged images and capture pedestrian-related features from the filtered information to recover them into the network. This ensures that pedestrian ID features are not lost while eliminating image style differences. It also solves the gradient vanishing problem, enabling the module to stably extract pedestrian features. This significantly improves the model's discrimination ability when facing target domains with diverse styles.

[0013] 2) This invention reduces information diffusion by using Global Attention (GAM) to amplify the interaction between channels and spatial dimensions, enabling the model to capture salient pedestrian features in three dimensions. When faced with damaged pedestrian images that do not lose their original semantic information, the model can capture pedestrian representations that are relatively unchanged from the original image more fully.

[0014] 3) Unlike previous algorithms that only focused on performance on clean (normal) datasets, this invention is designed for pedestrian images with various damage conditions. The invention demonstrates significant superiority over current mainstream algorithms in various metrics on the damaged test set. Specifically, compared with the 2021 CIL model using the CUHK03 dataset, the results show that on the Corrupted Eval, Rank-1, mAP, and mINP are improved by 15.18%, 15.75%, and 11.65%, respectively. Attached Figure Description

[0015] Figure 1 This is a diagram of the pedestrian re-identification network structure for damaged images, as described in an embodiment of the present invention.

[0016] Figure 2 This is a structural diagram of the Style Normalization and Recovery Module (SM-SNR) according to an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the SMU function and ReLU function in an embodiment of the present invention;

[0018] Figure 4 This is a structural diagram of the Global Attention Module (GAM) according to an embodiment of the present invention. Detailed Implementation

[0019] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0020] The present invention provides a pedestrian re-identification method for damaged images, which inputs the damaged image into a damaged image pedestrian re-identification network and outputs a pedestrian representation that remains unchanged relative to the original undamaged image;

[0021] Please see Figure 1 The damaged image pedestrian re-identification network in this embodiment uses ResNet50 as its backbone network. After each of the four layers of the backbone network, a style normalization and restitution module (SM-SNR) with smooth maximum unit-style normalization and restitution (SM-SNR) is added. This module normalizes features through instance normalization to weaken the style differences between damaged and clean images, while recovering pedestrian-related information lost during normalization. A global attention mechanism (GAM) module is added after the last SM-SNR, using the output of this module as input. By reducing the loss of global information and improving the interaction of global features, the network's recognition performance is improved. Then, the pooling layer and fully connected layer at the end of ResNet50 are removed, and the stride of the last layer (layer 4) is changed from 2 to 1. By increasing the size of the GAM output features, fine-grained features are enriched, resulting in a pedestrian representation that remains unchanged from the original undamaged image.

[0022] After GAM (Generative Mean Pooling), a global feature F of size 192×2048×1×1 is obtained through a generalized mean pooling (GeM) layer. Because the statistical information (mean and variance) of damaged and clean images differs significantly, using the BNNecked features for inference would reduce the model's robustness. Therefore, this network uses the global feature F for triplet loss metric learning. Finally, F is processed through a BNNeck layer and normalized to obtain a feature f of size 192×2048. f is then processed by Consistent Identity Loss (Consistent ID Loss) to optimize the distance between samples in the feature space and supervise the network training. Consistent ID Loss utilizes the original image sample x... o The JS divergence between the posterior distribution of the augmented image samples makes the pedestrian re-identification network for damaged images respond more smoothly to damaged images.

[0023] Please see Figure 2 In this embodiment, the SM-SNR is based on the Style Normalization and Recovery Module (SNR), and the ReLU layer after average pooling in this module is replaced with a Smooth Maximal Unit (SMU) to form the Smooth Style Normalization and Recovery Module; the SM-SNR uses the feature map F0∈R C×H×W As input, As output, C, H, and W represent the number of channels, height, and width of the feature map, respectively;

[0024] Because real-world damage types are numerous and unpredictable, fully supervised pedestrian re-identification methods often suffer from poor robustness due to domain differences. To address this issue, this embodiment introduces a Style Normalization and Restoration (SNR) module, replacing the ReLU layer after average pooling in this module with a Smooth Maximum Unit (SMU), thus forming the Smooth Style Normalization and Restoration (SM-SNR) module proposed in this invention. This ensures that pedestrian ID features are not lost while eliminating image style differences, and overcomes the shortcomings of traditional activation functions, improving the model's discriminative ability when facing target domains with diverse styles.

[0025] This embodiment first uses the SNR module. Instance normalization (IN) eliminates the style of the current domain to reduce the variability between pedestrian samples, thereby enhancing the robustness of the network. However, IN inevitably removes some discriminative information, weakening the network's discriminative ability. Therefore, this module can extract task-specific discriminative features from the removed IN information and restore them.

[0026] This embodiment first performs style normalization on the feature map, and then reduces the input features using an instance normalization layer IN(.

[0027] The differences between domains are used to obtain the features after style normalization.

[0028] While IN reduces style differences and improves model robustness, the deterministic and task-independent nature of mathematical operations inevitably leads to the loss of some discriminative pedestrian information. This embodiment uses a residual feature extraction layer to combine F0 with... The residual feature R is obtained by subtraction, and the ID-related feature R is adaptively extracted from the residual feature R through the channel attention layer. + And restore it to the network;

[0029] Finally, the captured R is processed through a fusion layer. + Added to style normalization features Then, the restored features are obtained.

[0030] This embodiment fully utilizes instance normalization (IN) through the SNR module to reduce the style difference between the damaged image in the target domain and the clean dataset in the source domain. Furthermore, it recovers the lost and discriminative features from the residuals and normalized features of the original information, thereby greatly improving the network's ability to identify candidate pedestrians in damaged images.

[0031] To enable the SNR module to more fully capture discriminative information from the residual features, this embodiment replaces the ReLU nonlinear activation function after global average pooling with a Smooth Maximum Unit (SMU). This smooth activation function is implemented through a smooth approximation of the Maximum function, using xerf(μx) as the smoothing function to approximate the Maximum function. Because the SMU's value is not zero when the input x ≤ 0, it avoids parameter not updating during training, solving the gradient vanishing problem. The function graphs of SMU and ReLU are shown below. Figure 3 As shown, applying the SMU activation function to the SNR module can better perform nonlinear mapping and improve the expressive power of the entire module.

[0032] The Global Attention (GAM) module in this embodiment differs from previous attention methods in that it can enhance "global" cross-dimensional interactions by focusing on channel and spatial information. This attention employs a sequential channel-space attention mechanism, such as... Figure 4 As shown, let the output feature after the fourth SM-SNR module be F1∈R C×H×W Using this as the input to GAM, F2 represents the features in the intermediate state, and F3 represents the final output features, defined as follows:

[0033]

[0034]

[0035] Among them, M C It is a channel attention map, M S It is a spatial attention map. This indicates element-wise multiplication. In channel attention, a three-dimensional arrangement is used to retain information in three dimensions; then, a multilayer perceptron (MLP) is used to amplify cross-dimensional channel-space dependencies. The MLP is an encoder-decoder structure similar to BAM, with a dimensionality reduction ratio of r. The F1 and M... C (F1) Element-wise multiplication yields F2. Next, to capture spatial information, spatial attention uses F2 as input and employs two 7×7 Convs for spatial information fusion. Simultaneously, since max pooling reduces the use of dimensional information, max pooling is removed to retain more pedestrian-related features. Finally, F2 is then combined with M... S (F2) After performing element-wise multiplication, feature F3 with multi-dimensional information is obtained.

[0036] This invention applies GAM to the fourth layer of ResNet50, amplifying the cross-dimensional interaction of the output features of the last SNR module, enabling the network to capture salient pedestrian features in three dimensions, thereby improving the model's discriminative power and robustness.

[0037] The damaged image pedestrian re-identification network in this embodiment is a pre-trained damaged image pedestrian re-identification network. This embodiment uses the damaged datasets Market1501-C, CUHK-03-C, and MSMT17-C to train and evaluate the damaged image pedestrian re-identification network. For example, Market1501-C is generated by the algorithm artificially damaging the Market1501 test set. The damaged image types are divided into four main categories: noise, blur, weather, and numbers. Each type of damage has five different levels of severity, resulting in 100 different levels and types of damaged pedestrian images. Simultaneously, to more fully evaluate the robustness of the model, one type and one level of damage are randomly selected from each image in the clean test set to create the following three damaged test sets: Corrupted query and gallery (Corrupted Eval), where query and gallery are image libraries for damaged types; Corrupted query, a damaged query sample; and Corrupted gallery, a damaged image library.

[0038] The loss function during training is L total =L cid +L t Among them, L cid For consistent ID loss, L t For triplet loss;

[0039] L cid for:

[0040]

[0041] in, p o =p(y|x o ), p a1 =p(y|x a1 ), p a2 =p(y|x2); x o For an original image with label y, p o This means using the softmax function to measure x o The predicted probability of being identified as category y; p a1 With p a2This represents the predicted probability of classifying the original images a1 and a2 as category y using the softmax function; KL represents the KL divergence function, which expresses x by taking the average of the sums of three different KL divergences. o With expanded sample x a1 x a2 JS divergence between posterior distributions; augmented sample x a1 x a2 These were generated from the original images a1 and a2 using data augmentation methods.

[0042] The loss of the triplet is:

[0043] L t =(d A,P -d A,N +β) + (4)

[0044] in(·) + This indicates taking the absolute value; for any chosen image sample A, positive sample P, and negative sample N, β is the threshold for the similarity between positive and negative samples, and d A,P and d A,N These represent the similarity between image sample A and its positive and negative samples, respectively.

[0045] This embodiment uses the damaged datasets Market1501-C, CUHK-03-C, and MSMT17-C to evaluate the model. For example, Market1501-C generates pedestrian damaged images by artificially damaging the Market1501 test set using the algorithm. The damaged image types are divided into four main categories: noise, blur, weather, and numbers. Each type of damage has five different levels of severity, resulting in 100 different levels and types of pedestrian damaged images. Simultaneously, to more fully evaluate the model's robustness, three types of damaged test sets are created by randomly selecting one type and one level of damage from each image in the clean test set: Corrupted query and gallery (Corrupted Eval), where query and gallery are image libraries representing damaged types; Corrupted query, a sample of damaged images; and Corrupted gallery, a library of damaged images.

[0046] To evaluate the performance of the damaged image person re-identification network in this embodiment, three person re-identification evaluation metrics were used: CMC-k (cumulative matching characteristics), also known as rank-k; mAP (mean average precision); and mINP (mean inverse negative penalty). This invention uses mINP to measure the model's retrieval ability on the most difficult correct match, expressed as follows:

[0047]

[0048] In the formula Let |Z be the rank value of the hardest match. i | represents the total number of correctly matched samples for the i-th image.

[0049] This embodiment uses the PyTorch 1.8.0 deep learning framework and Ubuntu 20.04 as the operating system. The hardware configuration is: GeForce GTX Titan GPU, Intel(R) Core(TM) i7-10700F CPU, 32GB VRAM, and 16GB RAM. Two novel local data augmentation methods are employed: Self-Patch Mixing and Soft Random Erasing, to help the model uncover more diverse fine-grained information and improve its robustness to damaged images. This embodiment performs ten evaluations on the Market1501 and CUHK03 datasets and three evaluations on the large-scale MSMT17 dataset, taking the average of the final results. The batch sizes are set to 64 and 128 for training and testing, respectively, with a training duration of 150 epochs.

[0050] This invention demonstrates significant superiority over current mainstream algorithms in various metrics on the corrupted test set. Specifically, compared with the 2021 CIL model using the CUHK03 dataset, this invention improves Rank-1, mAP, and mINP by 15.18%, 15.75%, and 11.65%, respectively, on the Corrupted Eval.

[0051] This invention employs a style normalization and restoration module to filter out interference from various damaged images and capture pedestrian-related features from the filtered information to restore them to the network. Simultaneously, a maximal smoothing unit is added to this module instead of ReLU to solve the gradient vanishing problem, enabling the module to stably extract pedestrian features. Finally, a global attention mechanism is used to reduce information diffusion and amplify the "global" cross-dimensional interaction, allowing the model to more fully capture pedestrian representations that are relatively unchanged from the original image when faced with damaged pedestrian images that do not lose their original semantic information.

[0052] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for pedestrian re-identification in damaged images, characterized in that: The damaged image is input into the damaged image pedestrian re-identification network to extract pedestrian representations that are invariant to the original undamaged image; The damaged image pedestrian re-identification network uses ResNet50 as its backbone network. After each of the four layers of the backbone network, a Style Normalization and Restoration Module (SM-SNR) with Smooth Maximal Units (SMUs) is added. The SM-SNR is based on the Style Normalization and Restoration Module (SNR), but replaces the ReLU layer after average pooling in the module with SMUs. After the last SM-SNR, a Global Attention Module (GAM) is added. At the same time, the pooling layer and fully connected layer at the end of ResNet50 are removed, and the stride of the last layer, layer 4, is set to 1. The output is a pedestrian representation that remains unchanged from the original undamaged image. The SM-SNR uses feature maps As input, As output, where C , H and W These represent the number of channels, height, and width of the feature map, respectively. First, style normalization is performed on the feature maps. An instance normalization layer (IN) is used to reduce the variability of the input feature domain, resulting in style-normalized features. ; Then, through the residual feature extraction layer, F 0 and Subtraction yields residual characteristics R And adaptively extract residual features through a channel attention layer. R Extract ID-related features R + And restore it to the network; Finally, the captured R is processed through a fusion layer. + Added to style normalization features Then, the restored features are obtained. ; The Smooth Maximum Unit (SMU) is implemented using a smooth approximation of the Maximum function, employing... As a smoothing function, it aims to approximate the Maximum function; The Global Attention Module (GAM) uses the output features after the fourth SM-SNR module. For input, These are the characteristics when the state is in the intermediate state. For the final output features; （1） （2） in, It is a channel attention map. It is a spatial attention map. This indicates element-wise multiplication; in channel attention, a three-dimensional arrangement is used to retain information in three dimensions; then, a multilayer perceptron (MLP) is used to amplify cross-dimensional channel-space dependencies; and After performing element-wise multiplication, we get... Spatial attention As input, spatial information is fused using two 7×7 Conv arrays; simultaneously, max pooling is removed to retain more pedestrian-related features; finally, and After performing element-wise multiplication, features with multi-dimensional information were obtained. .

2. The pedestrian re-identification method for damaged images according to claim 1, characterized in that: The global features are obtained after GAM by a generalized mean pooling layer GeM. , used for triplet loss metric learning; the Features of a preset size are obtained through normalization processing using the set BNNeck layer. , The training of the network is supervised by optimizing the distance between samples in the feature space through consistent ID loss; The consistency ID Loss utilizes the original image samples. The JS divergence between the posterior distribution of the augmented image samples makes the pedestrian re-identification network for damaged images respond more smoothly to damaged images.

3. The pedestrian re-identification method for damaged images according to any one of claims 1-2, characterized in that: The loss function of the damaged image pedestrian re-identification network; the loss function during the training process is... ;in, For consistent ID loss, For triplet loss; for: （3） in, , , , ; For those with tags An original image, This indicates that the softmax function is used to... Identified as a category The predicted probability; and This indicates that the softmax function is used to convert the original image... a 1 、a 2 is identified as a category The predicted probability; KL represents the KL divergence function, which is expressed by averaging the sums of three different KL divergences. With expanded sample , JS divergence between posterior distributions; sample enlargement , Original image a 1 、a 2. Generated through data augmentation methods; The loss of the triplet is: （4） in This indicates taking the absolute value; for any selected image sample A Positive samples P and negative samples N , This is the threshold for the similarity between positive and negative samples. and Representing image samples respectively A Similarity with its positive and negative samples.

4. A pedestrian re-identification system for damaged images, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the pedestrian re-identification method for damaged images as described in any one of claims 1 to 3.

Images