Pedestrian re-identification method based on multi-branch attitude perception and attention weighted fusion

Abstract

The invention discloses pedestrian re-identification based on multi-branch attitude perception and attention weighted fusion, relates to the technical field of computer vision and artificial intelligence, and aims to solve the problems of poor visual angle attitude adaptability, insufficient shielding scene robustness and the like in the prior art. The method comprises the following steps: firstly, carrying out size normalization, illumination normalization, noise removal and data enhancement preprocessing on a cross-camera pedestrian image; then, a multi-branch feature extraction network with improved ResNet-50 as a backbone is constructed, a channel point affinity attention module is embedded in the backbone network, the local detail extraction capability of the network is enhanced, and global features and attitude sensing features driven by an attitude estimation sub-network are synchronously extracted; weight coefficients of the three types of features are adaptively calculated through a feature attention module, and weighted fusion is carried out to obtain fused features; and finally, accurate matching of the queried pedestrian and the candidate library is realized through cosine similarity calculation. According to the method, the accuracy of pedestrian re-identification in a complex scene is remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, in particular to a pedestrian re-identification and system based on multi-branch pose perception and attention weighted fusion, which realizes accurate tracking and identity matching of specific pedestrians under cross-camera and cross-scene. BACKGROUND

[0002] Pedestrian re-identification technology aims to accurately identify the same pedestrian in images or video sequences captured by different cameras at different times, different locations, and different camera devices, and its core goal is to solve the cross-camera tracking problem. With the acceleration of urbanization and the increasing demand for security, pedestrian re-identification technology has become a key component of intelligent security systems, and also has wide application prospects in intelligent transportation (such as tracking of traffic accident escape personnel), commercial retail (such as customer behavior analysis), and smart park (such as personnel access management).

[0003] However, existing pedestrian re-identification technology still faces many challenges, resulting in difficulty in meeting the demand for recognition accuracy and robustness in actual application. Specifically, the existing technology has the following shortcomings:

[0004] The occlusion problem is difficult to solve: in actual scenarios, pedestrians are often occluded by other pedestrians, vehicles, buildings, and other objects, and partial or complete occlusion can cause the loss of key features of pedestrians (such as clothing texture and body contour). Existing models rely on global feature extraction, which is less adaptable to occluded scenes and easily confuses occluded pedestrians with other pedestrians.

[0005] Light and environmental interference are significant: the installation environment of different cameras (such as indoor and outdoor, day and night) has obvious light differences (such as strong light, weak light, and backlight), and the environmental background (such as streets, shopping malls, and parks) is complex and diverse. The robustness of existing models to light changes and environmental noise is insufficient, resulting in unstable feature expression of the same pedestrian in different environments.

[0006] Low efficiency of feature fusion: some existing technologies attempt to fuse global features and local features (such as head, torso, and leg features) of pedestrians, but use simple concatenation or weighted summation methods, which do not fully exploit the correlation between global features and local features, and easily introduce redundant information during feature fusion, resulting in increased model computational complexity and decreased inference speed, making it difficult to meet real-time requirements. SUMMARY

[0007] To solve the above problems, the field needs a pedestrian re-identification technology with high robustness, high accuracy, high generalization ability, and real-time requirements to adapt to complex and variable actual application scenarios.

[0008] The application aims to provide a pedestrian re-identification method based on multi-branch pose perception and attention weighted fusion.

[0009] S1: obtaining target visible light images and infrared images to be matched, including image data of pedestrians and text descriptions corresponding to the images;

[0010] S2: performing data enhancement on pedestrian re-identification data, the data enhancement is used to process image data lacking of text descriptions, generate text descriptions corresponding to the image data, and construct a multi-branch feature extraction network;

[0011] S3: using an attention weighted fusion strategy to perform weighted summation on global feature vectors, local feature vectors and pose perception feature vectors to obtain a fusion feature vector;

[0012] S4: using a Meta-Learning framework to jointly train the multi-branch feature extraction network and the feature attention module for cross-scene adaptive training;

[0013] S5: comparing cosine similarities between pedestrian fusion feature vectors calculated above with preset values to perform pedestrian re-identification.

[0014] According to an aspect of the embodiment of the application, a combination of adaptive histogram equalization (CLAHE) and gray mean normalization is used to eliminate the light difference between different images, specifically: first, the image is converted to gray, the mean and standard deviation of the gray image are calculated, the gray image is adjusted to a preset mean (such as 127) and a preset standard deviation (such as 64), and then the adjusted gray image is mapped back to the color image space.

[0015] According to an aspect of the embodiment of the application, a combination of Gaussian filtering and median filtering is used to remove Gaussian noise and salt and pepper noise in the image, wherein the size of the convolution kernel of the Gaussian filtering is adaptively adjusted according to the image noise intensity (such as setting the convolution kernel size to 5x5 when the noise intensity is greater than a preset threshold, otherwise setting it to 3x3).

[0016] According to an aspect of the embodiment of the application, a combination of random horizontal flipping, random rotation (-10°-10°), random cropping (cropping ratio is 0.8-1.0) and color jittering (brightness, contrast, saturation adjustment range is 0.8-1.2) is used to expand the pedestrian image dataset and improve the generalization ability of the model.

[0017] According to an aspect of the embodiment of the present application, a multi-branch feature extraction network is constructed, which is based on an improved ResNet-50 network as a basic backbone network, and includes a global feature extraction branch, a local feature extraction branch, and a pose perception feature extraction branch. The global feature extraction branch is connected with a global average pooling layer (GAP) and a batch normalization (BN) layer after the last convolutional layer of the backbone network, extracts a global feature vector (dimension 2048) of the pedestrian image, and the global feature vector reflects global information such as the overall contour and clothing color of the pedestrian. The local feature extraction branch introduces a spatial attention module (SAM) after the third convolutional layer of the backbone network, highlights key local areas (such as the head, torso, and legs) of the pedestrian by calculating the attention weight of each pixel of the image, and then performs average pooling on each key local area to obtain multiple local feature vectors (such as three local feature vectors, each with a dimension of 1024). The pose perception feature extraction branch accesses a lightweight pose estimation subnetwork (based on MobileNetV2) after the second convolutional layer of the backbone network, the pose estimation subnetwork outputs 14 key pose points (such as the head vertex, shoulders, elbows, waist, knees, and ankles) of the pedestrian, and divides the pedestrian image into multiple pose regions (such as the head region, torso region, upper limb region, and lower limb region) according to the key pose points, and performs convolution and pooling operations on each pose region to obtain a pose perception feature vector (dimension 1024).

[0018] According to an aspect of the embodiment of the present application, a feature attention module (FAM) is constructed, which calculates weight coefficients (the sum of the weight coefficients is 1) of global features, local features, and pose perception features through a fully connected layer and a sigmoid activation function, and the weight coefficients are adaptively adjusted according to the importance of the features in different scenes (such as in the occlusion scene, the weight coefficients of the local features and the pose perception features are increased, and the weight coefficient of the global features is decreased). According to the weight coefficients, the global feature vector, the local feature vector, and the pose perception feature vector are weighted and summed to obtain a fusion feature vector (dimension 2048). The fusion feature vector is subjected to L2 normalization processing to eliminate the scale difference of the feature vector and improve the stability of the feature.

[0019] The pedestrian re-identification and system based on multi-branch posture perception and attention weighted fusion provided by the application have the following beneficial effects: first, strong perspective and posture robustness: by constructing a posture perception feature extraction branch, combining a lightweight posture estimation subnetwork, extracting the posture perception features of pedestrians, effectively eliminating the influence of perspective changes and posture dynamic changes on feature extraction, and improving the adaptability of the model to perspective and posture changes. Second, introduce a spatial attention module and a local feature extraction branch to highlight the key local area of the pedestrian. Even in the case of occluded pedestrians, accurate identification can still be achieved through local features, significantly improving the robustness of the model in occluded scenes. Third, high resistance to light and environmental interference: using light normalization and cross-scene adaptive training methods, combined with an adversarial training mechanism, the model can adapt to different light conditions and complex environments, eliminating the influence of light differences and environmental noise on feature expression, and improving the stability of the features. Fourth, the model has good generalization ability: cross-scene adaptive training based on a meta-learning framework enables the model to quickly adapt to the feature distribution of new scenes, effectively addressing the difference between training data and actual application scenarios, and significantly improving the generalization ability of the model. Fifth, balance between recognition efficiency and real-time performance: using an attention weighted fusion strategy avoids the introduction of redundant features, reduces the computational complexity of the model, and selects an improved ResNet-50 and a lightweight posture estimation subnetwork as the base network to improve the inference speed of the model while ensuring recognition accuracy, meeting the real-time requirements of actual applications. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 The flowchart of the pedestrian re-identification based on multi-branch posture perception and attention weighted fusion in the embodiments of the present application;

[0022] Figure 2 The structure diagram of the multi-branch feature extraction network in the embodiments of the present application;

[0023] Figure 3 The structure diagram of the feature attention module in the embodiments of the present application;

[0024] Figure 4 The framework diagram of the cross-scene adaptive training in the embodiments of the present application. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0026] As shown in Figure 1 The present embodiment provides a pedestrian re-identification method based on multi-branch pose perception and attention weighted fusion, comprising the following steps:

[0027] Step S1: Multi-modal data acquisition. Obtain the visible light image and infrared image to be matched, and collect the text description corresponding to the image (such as pedestrian clothing, posture, scene attribute) at the same time. Visible light and infrared are multi-modal complementary: visible light is rich in details under normal illumination, and infrared can still capture the outline of pedestrians in night, low light, and smoke scenes, solving the scene limitations of a single modality. The visual features extracted by ResNet-50 can be associated with text information, improving the semantic distinguishability of the features.

[0028] Step S2: Data augmentation and multi-branch feature network construction. Data is augmented, and a multi-branch feature extraction network with ResNet-50 as the backbone is constructed. Data augmentation solves the problem of insufficient pedestrian re-identification samples (such as random flipping, occlusion simulation, etc.), and text description is supplemented to align visual features with semantic information. The residual structure of ResNet-50 can avoid gradient vanishing in deep network, and is suitable for extracting complex visual features. Multi-branch is to split global branch, local branch (extracting features of head, torso, legs, etc.), and posture branch (extracting pedestrian action features) at different levels of ResNet-50, covering the overall-local-dynamic dimensions of pedestrians, and solving the problem of insufficient distinguishability of single feature.

[0029] Step S3: Attention weighted feature fusion. The global feature vector + local feature vector + posture feature vector output by the ResNet-50 multi-branch is weighted and fused after calculating the weight using the attention mechanism to obtain the final fused feature. The multi-branch of ResNet-50 has extracted multi-dimensional features, but the importance of the features is different in different scenes. Attention weighting is to dynamically allocate weights (such as increasing the weight of local features when occluded) to avoid redundancy / invalid features caused by simple concatenation, and to make the fused features more accurately match the pedestrian identity.

[0030] Step S4: Meta-learning cross-scene training. Use the Meta-Learning framework to jointly train the multi-branch ResNet-50 attention module to achieve cross-scene adaptation. The core difficulty of pedestrian re-identification is the poor cross-scene generalization. Meta-learning can quickly adapt ResNet-50 to the feature distribution of a new scene, and at the same time, jointly train the attention module to make the feature fusion weight also adapt to different scenes, solving the problem of weak generalization ability of ResNet-50 under single training.

[0031] Step S5: Cosine similarity matching. Calculate the cosine similarity between the query pedestrian fusion features and the candidate library pedestrian fusion features, and compare it with the preset threshold (such as 0.85) to output the matching result. The features after ResNet-50 and attention fusion are high-discrimination identity features. Cosine similarity is a classic method for measuring feature vector similarity, and through threshold screening, it can quickly locate pedestrians with the same identity and complete the final matching of re-identification.

[0032] As shown in Figure 2 and Figure 3 , first input the preprocessed normalized image into the ResNet-50 backbone network, extract the basic features from shallow texture to deep semantics through the hierarchical structure (avoid gradient disappearance through residual structure), then split out three types of features from different levels of the network, namely global, local and pose features (cover the scenes of "whole-local-dynamic" such as occlusion and pose change), after the three types of features are collected and spliced into high-dimensional vectors, through dynamic weight calculation (adjust the weight of each feature according to the scene, such as increasing the weight of local features in occlusion), finally weighted fusion and L2 normalization to get robust fusion features; This process solves the problem of single feature dimension with ResNet-50 multi-branch, and solves the problem of poor scene adaptability with "dynamic attention fusion", which improves the pedestrian re-identification ability of the model under complex conditions such as occlusion, pose change and cross-scene.

[0033] As shown in Figure 4As shown, the present example provides a pedestrian re-identification cross-scene adaptive training process using a ResNet multi-branch network. First, through the cross-scene training dataset support set (70%) and query set (30%) and dataset division module, the pedestrian dataset containing scenes such as streets and shopping malls is divided into a support set (adapt single scene feature) and a query set (verify generalization ability) in a ratio of 7:3, providing a data basis for meta-learning scene adaptation. Then, enter the meta-learning inner loop, relying on the "3.1 model fine-tuning" and "3.2 loss calculation" modules, fine-tune the model 5 times on the support set with a low learning rate of 0.0005, and update the local parameters through the triplet loss (α = 0.3) to let the ResNet-50 multi-branch network adapt to the feature distribution of the current scene. Then, through the 4.1 model push, 4.2 loss calculation, and 4.3 outer loop output modules of the meta-learning outer loop, the query set is used to extract the ResNet-50 multi-branch fusion features, calculate the total loss L total = 0.4L triplet + 0.4L ce + 0.2L adv containing triplet loss, cross-entropy loss, and adversarial loss, and output global parameter updates to let the model learn the common identity features of different scenes. At the same time, with the help of the scene discriminator (3-layer fully connected) of the "adversarial training branch" and the adversarial loss module, the ResNet-50 multi-branch network is trained as a generator and a discriminator, through L adv to make the fusion features hide the scene information and retain the identity features. After that, the model main module multi-branch feature extraction network and the feature attention module receive parameter updates from the inner and outer loops to continuously optimize the multi-branch feature extraction and dynamic weighting fusion capabilities of ResNet-50. Finally, the "training control module" stabilizes the training pace with a dynamic learning rate (initially 0.001, reduced to 0.5 to 1e -6 -10 every 10 epochs) and outputs the model. This process is designed around the ResNet-50 multi-branch feature extraction, uses meta-learning double-loop to solve the problem of scene feature distribution difference, uses adversarial training to solve the problem of scene interference, and finally makes the multi-branch features of ResNet-50 retain the global-local-pose information of pedestrian identity and adapt to complex scenes, improving the accuracy of cross-scene pedestrian re-identification.

[0034] It should be noted that the above content is only a preferred exemplary embodiment of the present application, and is not intended to limit the implementation of the present application. Those skilled in the art can easily make corresponding modifications or modifications based on the main concept and spirit of the present application, and the protection scope of the present application should be determined by the protection scope claimed in the claims.

Claims

1. A pedestrian re-identification based on multi-branch pose perception and attention weighted fusion, comprising the following steps: S1: obtaining target visible light images and infrared images to be matched, including image data of pedestrians and text descriptions corresponding to the images; S2: performing data enhancement on the pedestrian re-identification data, the data enhancement being used to process image data lacking text descriptions, generate text descriptions corresponding to the image data, and construct a multi-branch feature extraction network; S3: using an attention weighted fusion strategy to perform weighted summation on global feature vectors, local feature vectors, and pose perception feature vectors to obtain a fusion feature vector; S4: using a meta-learning framework to jointly train the multi-branch feature extraction network and the feature attention module for cross-scene adaptive training; S5: comparing the cosine similarity between the fusion feature vectors of the pedestrians calculated above with a preset value to perform pedestrian re-identification.

2. The pedestrian re-identification method based on multi-branch pose perception and attention weighting fusion according to claim 1, characterized in that, In S2, the data enhancement process is as follows: A lightweight pose guiding branch is added in ResNet-50, and through feature alignment geometric transformation, enhanced samples of the same identity and different poses are generated to optimize the adaptability of the multi-branch network to pose changes. The pose guiding branch includes a grid network and a bilinear sampler, and enhanced samples of the same identity and different poses are generated through pose feature alignment geometric transformation. The geometric transformation is realized based on 6 affine parameters, and the cosine similarity between the pose features of the enhanced samples and the original image is ≥0.

9.

3. The pedestrian re-identification method based on multi-branch pose perception and attention weighting fusion according to claim 1, characterized in that, In step S2, the implementation of the global feature extraction branch is as follows: a global average pooling layer and a batch normalization layer are connected in series after the layer5 convolution layer of ResNet-50, and a 2048-dimensional global feature vector is output. In step S2, the implementation of the local feature extraction branch is as follows: a spatial attention module is connected after the layer3 convolution layer of ResNet-50, a 1x1 convolution is used for dimension reduction, a channel number of 1 is output, and a Sigmoid activation is used to generate a pixel-level attention weight map. After multiplication with the original feature map, it is divided into 3 local regions of head, torso, and leg. After 3x3 convolution and average pooling, 3 local feature vectors of 1024 dimensions are output, and a total of 3 local feature vectors of 1024 dimensions are obtained. In step S2, the implementation of the pose perception feature extraction branch is as follows: a pose estimation subnetwork based on MobileNetV2 is connected after the layer2 convolution layer of ResNet-50. The subnetwork includes 17 convolution layers, contains 3 bottleneck structures, and has an expansion coefficient of 6. A heat map of 14 pedestrian pose points is output. According to the pose point coordinates, 4 pose regions of head, torso, upper limbs, and lower limbs are divided. After convolution and pooling of the feature of each region, a 1024-dimensional pose perception feature vector is obtained.

4. The pedestrian re-identification method based on multi-branch pose perception and attention weighting fusion according to claim 1, characterized in that, In step S3, the specific implementation process of the feature attention module is as follows: (1) Feature concatenation: 2048-dimensional global feature vectors, 3 1024-dimensional local feature vectors (concatenated into 3072-dimensional), and 1024-dimensional pose perception feature vectors are concatenated to obtain a 6144-dimensional joint feature vector; (2) weight generation: the joint feature vector is generated by 1x1 convolution (output channel number 256), ReLU activation, full connection layer, input 256 dimensions, output 4 dimensions and Softmax activation, generating 4 weight coefficients W1, W2, W3, W4, satisfying W1+W2+W3+W4=1, wherein W1 is the global feature weight, and W2, W3 and W4 are respectively the head, torso and leg local feature weights.

5. The pedestrian re-identification method based on multi-branch pose perception and attention weighting fusion according to claim 2, characterized in that, Dataset division: divide the cross-scene dataset into at least 4 scene subsets, and divide each scene subset into a support set and a query set in a 7:3 ratio; Inner loop training: minimize the triplet loss on the support set of each scene subset, with a learning rate of 0.0005, 5 iterations, a margin value α=0.3, and update the local model parameters; Outer loop training: compute total loss on query sets of each scenario subset, update global model parameters, the total loss: L total = 0.4L triplet + 0.4L ce + 0.2L adv where L triple t is the triplet loss, L ce is the cross-entropy loss, L adv is the adversarial loss.

6. The pedestrian re-identification method based on multi-branch pose perception and attention weighting fusion according to claim 1, characterized in that, The adversarial loss L adv The scene discriminator is implemented as a 3-layer fully connected network, with an input of a 2048-dimensional fusion feature vector and an output of a probability distribution of 4 scene categories. The feature extraction network and the scene discriminator are alternately updated during training, with 1 update of the discriminator for every 2 updates of the generator, and a learning rate of 0.0001.

7. The pedestrian re-identification method based on multi-branch pose perception and attention weighting fusion according to claim 1, characterized in that: The training in step S4 is performed using an AdamW optimizer with a weight decay coefficient of 0.0001, β1=0.9, β2=0.999, a training batch size of 64, a total training epoch number of 100, an initial learning rate of 0.001, and a decay of 0.5 of the original value every 10 epochs until it is reduced to 1e -6 ; The preset threshold in step S5 is 0.85, if there are multiple images in the candidate image library whose cosine similarity is greater than the threshold, the top 5 matching results are output in descending order of similarity; if the cosine similarity of all images is less than the threshold, output "no matching result".

8. The pedestrian re-identification system based on multi-branch pose perception and attention weighting fusion according to any one of claims 1-7, characterized in that, It includes: Image acquisition module: deploy cross-scene cameras to collect pedestrian image data and associate text descriptions; Data preprocessing and enhancement module: perform image normalization, illumination adjustment, noise removal, and pose-guided image enhancement and text completion; Multi-branch feature extraction module: contains global feature branch, local feature branch, and pose perception feature branch, outputs three types of feature vectors; Feature fusion module: calculate weights and perform weighted fusion through feature attention module, output normalized fusion feature vector; Model training module: based on meta-learning framework and adversarial training mechanism, jointly train feature extraction module and fusion module; Matching identification module: calculate the cosine similarity of the target image and the candidate library image, and output the matching result after comparing with the preset threshold; Result output module: display the matching image, similarity value and ranking result in a visual way.

9. An electronic device, comprising: It includes at least one control processor and a memory connected in communication with the at least one control processor; The memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the pedestrian re-identification method based on multi-branch pose perception and attention weighted fusion according to any one of claims 1 to 5.

10. A computer readable storage medium characterized by, The computer readable storage medium stores computer executable instructions for causing a computer to execute the pedestrian re-identification method based on multi-branch pose perception and attention weighted fusion according to any one of claims 1 to 5.

Images