A pedestrian re-identification method based on deep learning

Abstract

The application relates to the technical field of operation and maintenance, and discloses a pedestrian re-identification method based on deep learning, which comprises the following steps: acquiring personnel behavior distribution dataset of a security operation and maintenance scene; performing data preprocessing, data augmentation based on image processing and camera style migration based on CycleGAN combined with VAE on the personnel behavior distribution dataset to determine the personnel behavior dataset of the security operation and maintenance scene; improving the TransReID based on ViT to construct a pedestrian re-identification model based on the personnel behavior dataset of the security operation and maintenance scene; and performing simulation experiment, training and verification on the pedestrian re-identification model to obtain the prediction result of the pedestrian re-identification model. The application proposes a pedestrian re-identification model of a base station operation and maintenance scene, verifies the effectiveness of data augmentation and a shuffling module, and simultaneously uses Grad-CAM to perform data visualization work.

Description

Technical Field

[0001] This invention relates to the field of operation and maintenance technology, and in particular to a pedestrian re-identification method based on deep learning. Background Technology

[0002] Typical pedestrian re-identification focuses on street scenes with ample lighting, where images are captured by multiple cameras at various angles. Base station maintenance scenarios present unique challenges. Image quality is affected by factors such as the overhead view of outdoor cameras, varying sizes of people being photographed, insufficient ambient light, and occlusion between equipment and people. Hardware challenges include limited field-of-view coverage of pre-positioned cameras, frequent cross-border movement of people, individual pedestrians appearing only in a single camera frame, and fixed image angles. These challenges are difficult to address directly with CNN-based models like ResNet, Aligned-ReID, and OSNet. Furthermore, CNN-based algorithms are limited by the sliding convolutional kernel window in feature extraction, resulting in insufficient long-range dependencies and global information extraction. In complex scenarios, such as pedestrian occlusion, background interference, and varying lighting conditions, the features extracted by CNN-based models may be inaccurate or lack robustness, thus affecting the accuracy of pedestrian re-identification. Therefore, a deep learning-based pedestrian re-identification method is urgently needed to improve accuracy. Summary of the Invention

[0003] The purpose of this invention is to provide a deep learning-based pedestrian re-identification method to achieve full-link, full-element analysis of personnel identity, time, location, and behavior inside and outside the base station equipment room. The most critical aspect is to solve the problem of personnel identity recognition through pedestrian re-identification technology. By accurately identifying personnel identities, it is possible to effectively distinguish between intrusions by maintenance personnel and strangers, damage to the tower structure, violations by maintenance personnel, etc., and accurately trace the responsibility for security risks.

[0004] This invention provides a deep learning-based person re-identification method, the method comprising:

[0005] Obtain a dataset of personnel behavior distribution in security check operation and maintenance scenarios;

[0006] The personnel behavior distribution dataset is preprocessed, image-based data augmentation is performed, and camera style transfer based on CycleGAN combined with VAE is performed to determine the personnel behavior dataset for security inspection and maintenance scenarios.

[0007] Based on the personnel behavior dataset of the security inspection and maintenance scenario, an improved pedestrian re-identification model is constructed using ViT's TransReID.

[0008] The pedestrian re-identification model was simulated, trained, and validated to obtain the prediction results of the pedestrian re-identification model.

[0009] Preferably, the human behavior distribution dataset includes source data in the form of 1920*1080 JPG or PNG images and their corresponding XML tags, wherein the XML tags mark the detection boxes of pedestrians in the JPG or PNG images;

[0010] The data preprocessing steps include: cropping, naming, filtering, and segmenting JPG or PNG images according to their tags;

[0011] Image processing-based data augmentation includes: geometric transformations, random erasure, Gaussian blurring, and color transformations, wherein the geometric transformations include affine transformations and focus transformations;

[0012] Camera style transfer based on CycleGAN combined with VAE includes: modifying the CycleGAN network in CamStyle, replacing the generator in CycleGAN with VAE, and constructing a new structure of CycleGAN combined with VAE to perform camera style transfer.

[0013] Preferably, the dataset of personnel behavior distribution in the security check and maintenance scenario includes:

[0014] To obtain the types of personnel behavior in security check and maintenance scenarios;

[0015] Based on the pre-defined correspondence between personnel behavior types and personnel categories, personnel categories are determined according to the personnel behavior types, resulting in a personnel behavior distribution dataset.

[0016] Preferably, the types of personnel behavior include both regular security inspection and maintenance operations and non-security inspection and maintenance behaviors.

[0017] Preferably, the personnel category includes maintenance personnel and strangers.

[0018] Preferably, based on the personnel behavior dataset of the security check and maintenance scenario, an improved version of ViT's TransReID is constructed to build a pedestrian re-identification model, including:

[0019] The image patches are linearized, and the positional information is embedded. They then pass through N Transformer layers into the shuffling module. The input vector of the last Transformer layer is divided into three blocks, rearranged, and connected to F... g To F n As a feature map.

[0020] Preferably, the input vector of the last Transformer layer is divided into three blocks and rearranged, including:

[0021] Let the above equation be the input vector of the Lth layer, where Z 0 l-1 It is a CLS token; process the remaining N columns.

[0022]

[0023] After shuffling and swapping the input vectors, the embedded sequences are grouped and shuffled as follows:

[0024]

[0025] Z l-1 Divide into 3 vectors, as shown in the following expression, and rearrange them by connecting the vectors;

[0026] Z l-1 =[Feature2,Feature3,Feature1].

[0027] Preferably, the simulation experiment includes four aspects: data augmentation, skeleton model, improvement module, and optimizer.

[0028] Preferably, the verification of the pedestrian re-identification model includes:

[0029] Visualize the output of DeiT using Grad-CAM;

[0030] Grad-CAM calculates the contribution of each token to the classification result from the output and gradient of any Transformer encoder, and maps the contribution back to the spatial location of the original image to form a heatmap.

[0031] Choose a Transformer encoder from the penultimate layer of ViT as the target layer;

[0032] Determine if the Transformer encoder notices occluded areas in the image.

[0033] Compared with existing technologies, the beneficial effects of this invention are as follows: For data augmentation, it designs affine and perspective transformation image transformation methods to address the problem of target tilting from a top-down perspective; for the problem of insufficient ambient light, it designs color enhancement image transformation methods; and for the problem of mutual occlusion between equipment and personnel, it designs random erasure image transformation methods. To address the issue of differences between images from different cameras, it designs a style transfer method combining VAE and CycleGAN. Simultaneously, this invention introduces pre-trained ViT and DeiT into the model, performs numerous hyperparameter adjustments, and designs a shuffling module for grouping randomly arranged vectors. The shuffling module adds Gaussian noise and zero masking to further analyze the model's performance. This invention proposes a pedestrian re-identification model for a base station operation and maintenance scenario, verifies the effectiveness of the data augmentation and shuffling modules, and uses Grad-CAM for data visualization. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a deep learning-based pedestrian re-identification method according to the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0037] like Figure 1 As shown, this invention provides a deep learning-based person re-identification method, comprising:

[0038] Obtain a dataset of personnel behavior distribution in security check and maintenance scenarios.

[0039] The personnel behavior distribution dataset is preprocessed, augmented with image processing, and subjected to camera style transfer based on CycleGAN combined with VAE to determine the personnel behavior dataset for security inspection and maintenance scenarios.

[0040] Based on the personnel behavior dataset of the security inspection and maintenance scenario, an improved version of ViT's TransReID is used to construct a pedestrian re-identification model.

[0041] The pedestrian re-identification model was simulated, trained, and validated to obtain the prediction results of the pedestrian re-identification model.

[0042] In some embodiments of this application, the personnel behavior distribution dataset includes 1920*1080 JPG or PNG images and their corresponding XML tags, wherein the XML tags annotate the detection boxes of pedestrians in the JPG or PNG images.

[0043] The data preprocessing steps include: cropping, naming, filtering, and segmenting JPG or PNG images according to their tags.

[0044] Image processing-based data augmentation includes geometric transformations, random erasure, Gaussian blur, and color transformations, wherein the geometric transformations include affine transformations and focus transformations.

[0045] Camera style transfer based on CycleGAN combined with VAE includes: modifying the CycleGAN network in CamStyle, replacing the generator in CycleGAN with VAE, and constructing a new structure of CycleGAN combined with VAE to perform camera style transfer.

[0046] In some embodiments of this application, obtaining a personnel behavior distribution dataset for a security check operation and maintenance scenario includes: obtaining the types of personnel behavior in the security check operation and maintenance scenario; determining the personnel category based on the pre-defined correspondence between personnel behavior types and personnel categories, thereby obtaining a personnel behavior distribution dataset.

[0047] In some embodiments of this application, the types of personnel behavior include formal security inspection and maintenance operations and non-security inspection and maintenance behaviors.

[0048] In some embodiments of this application, the personnel category includes maintenance personnel and strangers.

[0049] In some embodiments of this application, based on the personnel behavior dataset of the security inspection and maintenance scenario, an improved version of ViT's TransReID is used to construct a pedestrian re-identification model, including: linearizing image patches, embedding location information, passing the data through N Transformer layers into a shuffling module, dividing the input vector of the last Transformer layer into three blocks, rearranging them, and connecting them to F... g To F n As a feature map.

[0050] In some embodiments of this application, the input vector of the last Transformer layer is divided into three blocks and rearranged, including:

[0051] Let the above equation be the input vector of the Lth layer, where Z 0 l-1 It is a CLS token; process the remaining N columns.

[0052]

[0053] After shuffling and swapping the input vectors, the embedded sequences are grouped and shuffled as follows:

[0054]

[0055] Z l-1 Divide into 3 vectors, as shown in the following expression, and rearrange them by connecting the vectors;

[0056] Z l-1 =[Feature2,Feature3,Feature1].

[0057] In some embodiments of this application, the simulation experiment includes four aspects: data augmentation, skeleton model, improvement module, and optimizer.

[0058] In some embodiments of this application, the pedestrian re-identification model is validated by: visualizing the output of DeiT using Grad-CAM; calculating the contribution of each token to the classification result using the output and gradient of any Transformer encoder using Grad-CAM, and mapping the contribution back to the spatial location of the original image to form a heatmap; selecting a Transformer encoder in the penultimate layer of ViT as the target layer; and determining that the Transformer encoder notices occluded parts in the image.

[0059] To further illustrate the technical solution of the present invention, the present invention will now be described in detail:

[0060] This invention enables pedestrian re-identification, specifically including the following steps:

[0061] The following steps are taken: First, determine the appropriate data preprocessing method for the prediction model, including image processing-based data augmentation and camera style transfer based on CycleGAN combined with VAE. Second, determine the personnel behavior dataset for security inspection and maintenance scenarios. Third, linearize image patches according to the determined data preprocessing method, embed location information, and pass them through N Transformer layers into the shuffling module. The input vector of the last Transformer layer is divided into three blocks, rearranged, and then connected to F... g To Fn As a feature map, the simulation experiment realizes data processing and data augmentation based on the operation and maintenance scenario, pedestrian re-identification model design, training and verification, and the prediction model will give the corresponding prediction results.

[0062] The specific method for constructing a security inspection and maintenance scenario personnel behavior dataset includes: source data consisting of 1920*1080 JPG or PNG images and their corresponding XML tags; the XML tags mark the detection boxes for pedestrians in the images; data preprocessing steps involve cropping, naming, filtering, and segmenting the images according to the tag information; fully considering the characteristics of the dataset itself, image processing-based data augmentation is used, mainly including geometric transformation, color transformation, pixel-level transformation, cropping, and padding methods, to augment the images; based on CycleGAN combined with VAE camera style transfer, the model is trained using multiple cameras of Market1501 to obtain the transfer model from Camera1 to Camera2~6; the trained model is used to perform style transfer on the base station maintenance scenario dataset; and the output is the security inspection and maintenance scenario personnel behavior dataset mytt2zg1.

[0063] The specific implementation of the data augmentation method includes the following steps: obtaining a base station operation and maintenance scenario dataset, the Train set; calculating the mean and standard deviation of the dataset as normalization parameters through dataset statistics; performing normalization operations on the dataset; designing data augmentation units to perform data augmentation operations on the dataset; and applying data augmentation by giving each method a certain probability to perform data augmentation.

[0064] The data augmentation unit includes: a geometric transformation unit, which addresses the issue of overhead view in the data by performing data augmentation through affine transformation and focus transformation; a random erasure unit, which addresses the issue of mutual occlusion between equipment and people in the data by performing data augmentation through random erasure; a Gaussian blur unit, which addresses the issue of blurred parts of the image in the data by performing data augmentation through Gaussian blur; and a color transformation unit, which addresses the issue of insufficient ambient light in the data by adjusting the brightness, saturation, and contrast of the image through color transformation.

[0065] The specific implementation of a deep learning-based person re-identification model includes the following steps: Based on the ViT-based TransReID, a new shuffling mechanism is proposed to improve the model; image patches are linearized according to a determined data preprocessing method, embedding location information; data enters the shuffling module through N Transformer layers, introducing DeiT to improve the data efficiency and performance of the ViT model; the shuffling module divides the input vector of the last Transformer layer into three blocks and then rearranges them; connecting F...g To F n As a feature map.

[0066] The specific implementation methods for building the shuffling module include:

[0067] Read the corresponding data from the dataset;

[0068] Let Z be the input vector of the Lth layer, where Z 0 l-1 It is a CLS token; process the remaining N columns.

[0069]

[0070] Inspired by ShuffleNet's grouped convolution and channel shuffling, the vectors are shuffled and grouped to shuffle the embedded sequence as follows:

[0071]

[0072] The above formula will Z l-1 Divided into 3 vectors;

[0073] As shown in the following equation, the new shuffling mechanism divides the input vector of the last Transformer layer into 3 blocks and rearranges them by vector concatenation;

[0074] Z l-1 =[Feature2,Feature3,Feature1];

[0075] We added Gaussian noise and zero-occlusion modules after the shuffling module to verify the effectiveness of the improvement and to increase the model's robustness to occlusion and image blur.

[0076] The validation of the deep learning-based person re-identification method includes the following steps:

[0077] Experiments were conducted in four areas: data augmentation, skeleton model, improvement module, and optimizer. The experiments on skeleton model, improvement module, and optimizer were based on the premise that there were sufficient images in the dataset.

[0078] Experiments were conducted with and without a limited number of images in the dataset. The dataset had approximately 15,000 images in the training set, while the dataset had approximately 30,000 images in the training set.

[0079] The feasibility of the model is analyzed by comparing and contrasting data augmentation, skeleton model, improvement modules, and optimizer improvements.

[0080] Visualize the output of DeiT using Grad-CAM.

[0081] Grad-CAM calculates the contribution of each token to the classification result from the output and gradient of any Transformer encoder, and then maps these contributions back to the spatial location of the original image to form a heatmap.

[0082] Since the last layer of ViT only has category labels, a Transformer encoder in the penultimate layer is selected as the target layer.

[0083] The experiment shows that the Transformer encoder mainly pays attention to occluded parts in the image, verifying the interpretability of the model.

[0084] The specific implementation of the experiment from four aspects—data augmentation, skeleton model, improvement module, and optimizer—includes the following steps:

[0085] Data augmentation experiments were conducted under conditions of limited and sufficient dataset images. The experiments show that the improvements made based on ViT are effective, and the data augmentation method can indeed improve accuracy when the dataset is limited.

[0086] By introducing DeiT and ViT for skeleton model comparison, experimental results show that the improvement of DeiT over ViT comes at the cost of some performance.

[0087] The module was improved by designing a new shuffling mechanism. The input vector of the last Transformer layer was divided into three blocks and then rearranged. The Sh-ViT model was proposed. Experimental results show that the model has significant advantages over the original ViT model algorithm.

[0088] An optimizer called Adan for Transformers was introduced, which uses default parameters for optimization. Based on the DeiT model, it was compared with the default optimizer SGD. The experiment showed that the Adan optimizer with default parameters did not improve the model.

[0089] The experimental results are analyzed comprehensively to explain the feasibility of the model.

[0090] This invention addresses the issue of target tilt in a top-down view by designing affine and perspective transformation image augmentation methods for data augmentation; it also addresses the potential problem of insufficient ambient light by designing color enhancement image augmentation methods; and it addresses the issue of mutual occlusion between equipment and personnel by designing random erasure image augmentation methods for data augmentation. Furthermore, it addresses the issue of differences between images from different cameras by designing a style transfer method combining VAE and CycleGAN.

[0091] This invention introduces pre-trained ViT and DeiT into the model, performs numerous hyperparameter adjustments, and designs a shuffling module that groups the vectors after randomizing them. The shuffling module incorporates Gaussian noise and zero masking to further analyze the model's performance.

[0092] In summary, this invention proposes a pedestrian re-identification model for base station operation and maintenance scenarios, verifies the effectiveness of data augmentation and shuffling modules, and uses Grad-CAM for data visualization.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

[0094] The system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be further decomposed or combined. For example, the modules in the above embodiments can be merged into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the various modules or steps and are not considered as an improper limitation of the present invention.

[0095] Those skilled in the art will recognize that the modules and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. The programs corresponding to the software modules and method steps can be placed in random access memory (RAM), main memory, read-only memory (ROM), electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art. To clearly illustrate the interchangeability of electronic hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention.

Claims

1. A deep learning-based pedestrian re-identification method, characterized in that, The method comprises: obtaining a personnel behavior distribution data set of a security operation and maintenance scene; performing data preprocessing, image processing-based data augmentation, and CycleGAN combined with VAE-based camera style migration on the personnel behavior distribution data set to determine a security operation and maintenance scene personnel behavior data set, wherein the data augmentation comprises geometric transformation and random erasure; improving a TransReID based on ViT to construct a pedestrian re-identification model based on the security operation and maintenance scene personnel behavior data set; performing simulation experiments, training, and verification on the pedestrian re-identification model to obtain a prediction result of the pedestrian re-identification model; The improvement of the TransReID based on ViT to construct the pedestrian re-identification model based on the security operation and maintenance scene personnel behavior data set comprises: Based on the security operation and maintenance scene personnel behavior data set, input data of the first N layers of the Transformer layer are generated, the input data are input into the first N layers of the Transformer layer, and the output vector of the first N layers of the Transformer layer is used as the input vector of the last layer of the Transformer layer, wherein the input vector comprises a plurality of column vectors; The plurality of column vectors are grouped in order to obtain three vector groups, and the three vector groups are rearranged to obtain a new input vector of the last layer of the Transformer layer; Correspondingly, after the simulation experiments, training, and verification on the pedestrian re-identification model to obtain the prediction result of the pedestrian re-identification model, the method further comprises: Taking any Transformer encoder in the second-to-last layer of ViT as a target layer, using Grad-CAM to calculate the contribution of each token to the classification result based on the output and gradient of the target layer, and then mapping the contribution back to the spatial position of the original image to form a heat map to determine the occluded part in the image that the Transformer encoder pays attention to. 2.The deep learning based pedestrian re-identification method of claim 1, wherein, The personnel behavior distribution dataset includes source data in the form of jpg pictures or png pictures and corresponding xml labels, wherein the xml labels label the detection boxes of pedestrians in the jpg pictures or png pictures. The data preprocessing step comprises: cropping, naming, screening, and dividing jpg pictures or png pictures according to labels; The image processing-based data augmentation further comprises Gaussian blur and color transformation, and the geometric transformation comprises affine transformation and focal point transformation; The CycleGAN combined with VAE-based camera style migration comprises: modifying the CycleGAN network in CamStyle, replacing the generator in CycleGAN with VAE, and constructing a new structure of CycleGAN combined with VAE to perform conversion between camera styles. 3.The deep learning based pedestrian re-identification method of claim 1, wherein, Obtaining a personnel behavior distribution data set of a security operation and maintenance scene comprises: obtaining the behavior categories of personnel in the security operation and maintenance scene; determining the personnel categories based on the behavior categories of personnel according to the correspondence between the pre-set behavior categories of personnel and the personnel categories to obtain the personnel behavior distribution data set. 4.The deep learning based pedestrian re-identification method of claim 3, wherein, The behavior categories of personnel include security operation and maintenance normal operation categories and non-security operation and maintenance behavior categories. 5.The deep learning based pedestrian re-identification method of claim 3, wherein, The personnel categories include maintenance personnel and strangers. 6.The deep learning based pedestrian re-identification method of claim 1, wherein, Based on the security operation scene personnel behavior data set, an improved pedestrian re-identification model is constructed based on the TransReID of ViT, including: Linearize the image block, embed the position information, pass through N layers of Transformer layers into the shuffle module, divide the input vector of the last layer of Transformer layers into 3 blocks, rearrange, and connect to as Feature map feature map.

7. The deep learning-based pedestrian re-identification method of claim 6, wherein, The input vector of the last layer of the Transformer layer is divided into 3 blocks and rearranged, including: Let the above formula be the input vector of the Lth layer, where is the CLS token, and the remaining N columns are processed; ； After shuffling and grouping the input vector, the embedding sequence is shuffled as follows: ； The are split into 3 vectors, expressed as follows, and rearranged by vector concatenation; 。 8.The deep learning based pedestrian re-identification method of claim 1, wherein, The simulation experiment includes four aspects of data augmentation, skeleton model, improved module and optimizer.

Images