Pedestrian re-identification method based on deep learning

Abstract

The invention relates to the technical field of operation and maintenance, and discloses a pedestrian re-identification method based on deep learning, and the method comprises the steps: obtaining a personnel behavior distribution data set of a security check operation and maintenance scene; performing data preprocessing on the personnel behavior distribution data set, performing data augmentation based on image processing and camera style migration based on combination of CycleGAN and VAE, and determining a security check operation and maintenance scene personnel behavior data set; based on the security check operation and maintenance scene personnel behavior data set, improvement is carried out on the basis of TransReID of ViT, and a pedestrian re-identification model is constructed; and performing simulation experiment, training and verification on the pedestrian re-identification model to obtain a prediction result of the pedestrian re-identification model. According to the method, a pedestrian re-identification model of a base station operation and maintenance scene is provided, the validity of a data augmentation and shuffling module is verified, and meanwhile, Grad-CAM is used for data visualization work.

Description

Technical Field

[0001] The present invention relates to the field of operation and maintenance technology, and in particular to a pedestrian re-identification method based on deep learning. Background Art

[0002] General pedestrian re-identification focuses on street scenes with sufficient light, and the images are taken by multiple cameras at multiple angles. The scene of base station operation and maintenance is specific. From the perspective of the image, it faces the problems of outdoor camera looking down, different scales of shooting personnel, insufficient ambient light, and mutual occlusion between equipment and personnel. From the perspective of hardware shooting, it faces the problems of small coverage of the camera preset point field of view, frequent cross-border personnel, single pedestrians appearing in only one camera, and fixed photo angles. These scene difficulties are difficult to be directly completed on CNN-based related models such as ResNet, Aligned-ReID, OSNet and other open source algorithms, and in feature extraction, the CNN-based algorithm is limited by the sliding of the convolution kernel window, and the model is insufficient in long-distance dependence and global information extraction. In complex scenes, such as pedestrian occlusion, background interference, and lighting changes. These factors will cause the features extracted by the CNN-related model to be inaccurate or not robust enough, thereby affecting the accuracy of pedestrian re-identification. Therefore, a pedestrian re-identification method based on deep learning is urgently needed to improve the accuracy of pedestrian re-identification. Summary of the invention

[0003] The purpose of the present invention is to provide a pedestrian re-identification method based on deep learning, so as to realize the full-link and full-factor analysis of the identity-time-location-behavior of personnel inside and outside the base station room. The most critical thing is to solve the problem of personnel identity through pedestrian re-identification technology, and then through accurate identification of personnel identity, it is possible to effectively distinguish the intrusion behavior of maintenance personnel and strangers, the destruction of the tower body, the violation of regulations by maintenance personnel, etc., and accurately hold accountable for safety hazards.

[0004] The present invention provides a pedestrian re-identification method based on deep learning, the method comprising:

[0005] Obtain the personnel behavior distribution data set of security inspection and operation and maintenance scenarios;

[0006] Perform data preprocessing, data augmentation based on image processing, and camera style migration based on CycleGAN combined with VAE on the personnel behavior distribution dataset to determine the personnel behavior dataset for security inspection and operation and maintenance scenarios;

[0007] Based on the personnel behavior dataset of the security inspection and operation and maintenance scenario, we improved ViT's TransReID and built a pedestrian re-identification model.

[0008] The pedestrian re-identification model is subjected to simulation experiments, training and verification to obtain the prediction results of the pedestrian re-identification model.

[0009] Preferably, the personnel behavior distribution data set includes a jpg picture or a png picture with a source data style of 1920*1080 and its corresponding xml tag, wherein the xml tag marks the detection box of the pedestrian in the jpg picture or the png picture;

[0010] The data preprocessing step includes: cropping, naming, screening and dividing the jpg pictures or png pictures according to the tags;

[0011] Data augmentation based on image processing, including: geometric transformation, random erasing, Gaussian blur and color transformation, wherein the geometric transformation includes affine transformation and focus transformation;

[0012] Camera style transfer based on CycleGAN combined with VAE, including: modifying the CycleGAN network in CamStyle, replacing the generator in CycleGAN with VAE, and building a new structure of CycleGAN combined with VAE to perform conversion between camera styles.

[0013] Preferably, obtaining a personnel behavior distribution data set of a security inspection and operation and maintenance scenario includes:

[0014] Obtain the types of behaviors of personnel in security inspection and operation and maintenance scenarios;

[0015] Based on the preset correspondence between the behavior types of personnel and the personnel categories, the personnel categories are determined according to the behavior types of personnel, and a personnel behavior distribution data set is obtained.

[0016] Preferably, the types of personnel behavior include regular security inspection and maintenance operations and non-security inspection and maintenance behavior.

[0017] Preferably, the personnel categories include maintenance personnel and strangers.

[0018] Preferably, based on the personnel behavior dataset of the security inspection and operation and maintenance scenario, improvements are made on ViT's TransReID to build a pedestrian re-identification model, including:

[0019] The image block is linearized, the position information is embedded, and it enters the shuffle module through N layers of Transformer layers. The input vector of the last layer of Transformer layer is divided into 3 blocks, rearranged, and connected to F g To F n As a Feature map feature map.

[0020] Preferably, the input vector of the last Transformer layer is divided into three blocks and rearranged, including:

[0021] Assume the above formula is the input vector of the Lth layer, where Z 0 l-1 It is CLS token, and the remaining N columns are processed;

[0022]

[0023] After shuffling the input vectors and exchanging them, the embedded sequences are shuffled into groups as follows:

[0024]

[0025] Z l-1 Divided into 3 vectors, the expressions are as follows, and rearranged through vector connection;

[0026] Z l-1 =[Feature2,Feature3,Feature1].

[0027] Preferably, the simulation experiment includes four aspects: data augmentation, skeleton model, improvement module and optimizer.

[0028] Preferably, verifying the pedestrian re-identification model includes:

[0029] Visualize the output of DeiT using Grad-CAM;

[0030] Grad-CAM calculates the contribution of each token to the classification result based on the output and gradient of any Transformer encoder, and maps the contribution back to the spatial position of the original image to form a heat map.

[0031] Select a Transformer encoder in the penultimate layer of ViT as the target layer;

[0032] Determines which occlusions in the image the Transformer encoder notices.

[0033] Compared with the prior art, the beneficial effect of the present invention lies in that, in terms of data, the present invention designs an affine transformation and a perspective transformation image transformation method to perform data augmentation in response to the problem of target tilt in a bird's-eye view; designs a color-enhanced image transformation method to address the problem of insufficient ambient light; and designs a random-erasing image transformation method to address the problem of mutual occlusion between equipment and personnel. In response to the problem of image differences between cameras, a style transfer method combining VAE with CycleGAN is designed. At the same time, the present invention introduces ViT and DeiT based on pre-training to the model, adjusts many hyperparameters of the model, and designs a shuffling module that groups vectors after random arrangement. The shuffling module adds Gaussian noise and zero masking to further analyze the performance of the model. The present invention proposes a model for pedestrian re-identification in a base station operation and maintenance scenario, verifies the effectiveness of data augmentation and shuffling modules, and uses Grad-CAM for data visualization. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0035] Figure 1 It is a flowchart of a pedestrian re-identification method based on deep learning in the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] like Figure 1 As shown, the present invention provides a pedestrian re-identification method based on deep learning, comprising:

[0038] Obtain the personnel behavior distribution dataset for security inspection and operation and maintenance scenarios.

[0039] The personnel behavior distribution dataset is subjected to data preprocessing, data augmentation based on image processing, and camera style migration based on CycleGAN combined with VAE to determine the personnel behavior dataset for security inspection and operation and maintenance scenarios.

[0040] Based on the personnel behavior dataset of the security inspection and operation and maintenance scenario, improvements are made on ViT's TransReID to build a pedestrian re-identification model.

[0041] The pedestrian re-identification model is subjected to simulation experiments, training and verification to obtain the prediction results of the pedestrian re-identification model.

[0042] In some embodiments of the present application, the personnel behavior distribution data set includes a jpg image or a png image with a source data style of 1920*1080 and its corresponding xml tag, wherein the xml tag marks the detection box of the pedestrian in the jpg image or the png image.

[0043] The data preprocessing step includes: cropping, naming, screening and dividing the jpg pictures or png pictures according to the tags.

[0044] Data augmentation based on image processing includes: geometric transformation, random erasing, Gaussian blur and color transformation, wherein the geometric transformation includes affine transformation and focus transformation.

[0045] Camera style transfer based on CycleGAN combined with VAE, including: modifying the CycleGAN network in CamStyle, replacing the generator in CycleGAN with VAE, and building a new structure of CycleGAN combined with VAE to perform conversion between camera styles.

[0046] In some embodiments of the present application, obtaining a personnel behavior distribution data set of a security inspection and operation and maintenance scenario includes: obtaining the behavior types of personnel in the security inspection and operation and maintenance scenario; determining the personnel category according to the personnel behavior types based on the correspondence between pre-set personnel behavior types and personnel categories, and obtaining the personnel behavior distribution data set.

[0047] In some embodiments of the present application, the types of personnel behavior include regular security inspection and maintenance operations and non-security inspection and maintenance behavior.

[0048] In some embodiments of the present application, the personnel categories include maintenance personnel and strangers.

[0049] In some embodiments of the present application, based on the personnel behavior dataset of the security inspection and operation scene, an improvement is made on the basis of ViT's TransReID to construct a pedestrian re-identification model, including: linearizing the image block, embedding the position information, entering the shuffle module through N layers of Transformer layers, dividing the input vector of the last layer of Transformer layer into 3 blocks, rearranging, and connecting F g To F n As a Feature map feature map.

[0050] In some embodiments of the present application, the input vector of the last Transformer layer is divided into three blocks and rearranged, including:

[0051] Assume the above formula is the input vector of the Lth layer, where Z 0 l-1 It is CLS token, and the remaining N columns are processed;

[0052]

[0053] After shuffling the input vectors and exchanging them, the embedded sequences are shuffled into groups as follows:

[0054]

[0055] Z l-1 Divided into 3 vectors, the expressions are as follows, and rearranged through vector connection;

[0056] Z l-1 =[Feature2,Feature3,Feature1].

[0057] In some embodiments of the present application, the simulation experiment includes four aspects: data augmentation, skeleton model, improvement module, and optimizer.

[0058] In some embodiments of the present application, the pedestrian re-identification model is verified, including: using Grad-CAM to visualize the output of DeiT; Grad-CAM calculates the contribution of each token to the classification result based on the output and gradient of any Transformer encoder, and maps the contribution back to the spatial position of the original image to form a heat map; selects a Transformer encoder in the penultimate layer of ViT as the target layer; and determines that the Transformer encoder notices the occluded parts of the image.

[0059] To further illustrate the technical solution of the present invention, the present invention is now described in detail:

[0060] The present invention realizes pedestrian re-identification, which specifically includes the following steps:

[0061] Determine the corresponding data preprocessing method of the prediction model, data augmentation based on image processing, and camera style transfer based on CycleGAN combined with VAE; determine the personnel behavior data set of the security inspection and operation scene; linearize the image block according to the determined data preprocessing method, embed the position information, enter the shuffle module through N layers of Transformer layers, divide the input vector of the last layer of Transformer layer into 3 blocks, and then rearrange them and connect F g To Fn As the feature map; through simulation experiments, data processing and data augmentation based on operation and maintenance scenarios, pedestrian re-identification model design, training and verification are realized, and the prediction model will give the corresponding prediction results.

[0062] The specific method of constructing a data set of personnel behavior in security inspection and operation and maintenance scenarios includes: the source data style is a 1920*1080 jpg or png picture and its corresponding xml tag; the xml tag marks the detection box of the pedestrian in the picture; the data preprocessing step crops, names, filters, and divides the image according to the label information; fully considers the characteristics of the data set itself, and uses image processing-based data augmentation, mainly including geometric transformation, color transformation, pixel-level transformation, cropping and filling methods to augment the image data; based on CycleGAN combined with VAE camera style transfer, the model is trained using multiple cameras of Market1501 to obtain a migration model from Camera1 to Camera2~6; the trained model is used to transfer the style of the base station operation and maintenance scenario data set; and the security inspection and operation and maintenance scenario personnel behavior data set mytt2zg1 is output.

[0063] The specific implementation method of constructing the data augmentation method includes the following steps: obtaining the base station operation and maintenance scenario data set and the Train set; calculating the mean and standard deviation of the data set as normalization parameters through data set statistics; normalizing the data set; designing a data augmentation unit to perform data augmentation operations on the data set; performing data augmentation applications, giving each method a certain probability, and executing data augmentation.

[0064] Among them, the data augmentation units include: a geometric transformation unit, which augments data through affine transformation and focus transformation in geometric transformation to address the problem of overhead perspective shooting in the data set; a random erasure unit, which augments data through random erasure to address the problem of mutual occlusion between equipment and personnel in the data set; a Gaussian blur unit, which augments data using Gaussian blur to address the problem of partial image blur in the data set; and a color transformation unit, which adjusts the brightness, saturation and contrast of the image through color transformation to address the problem of insufficient ambient light in the data set.

[0065] The specific implementation method of building a pedestrian re-identification model based on deep learning includes the following steps: improving on the basis of TransReID using ViT, proposing a new shuffling mechanism to combine with building a pedestrian re-identification model; linearizing the image block according to the determined data preprocessing method, and embedding the position information; the data enters the shuffling module through N layers of Transformer layers, and DeiT is introduced to improve the data efficiency and performance of the ViT model; the shuffling module divides the input vector of the last layer of Transformer layer into 3 blocks, and then rearranges them; connecting Fg To F n As a Feature map feature map.

[0066] The specific implementation methods of building the shuffle module include:

[0067] Read the corresponding data in the data set;

[0068] Let the following be the input vector of the Lth layer, where Z 0 l-1 It is CLS token, and the remaining N columns are processed;

[0069]

[0070] Inspired by ShuffleNet group convolution and channel shuffle, the vectors are grouped and shuffled to shuffle the embedding sequence as follows:

[0071]

[0072] The above formula converts Z l-1 Divided into 3 vectors;

[0073] As shown in the following formula, the new shuffling mechanism divides the input vector of the last Transformer layer into three blocks and rearranges them through vector concatenation;

[0074] Z l-1 =[Feature2,Feature3,Feature1];

[0075] Gaussian noise and zero masking modules are added after the shuffling module to verify whether the improvement is effective and to increase the model's robustness to occlusion and image blur.

[0076] The verification of building a deep learning-based person re-identification method includes the following steps:

[0077] Experiments were conducted in four aspects: data augmentation, skeleton model, improvement module, and optimizer. The experiments on skeleton model, improvement module, and optimizer were based on the situation where there were sufficient images in the dataset.

[0078] Experiments were conducted under the conditions of limited dataset images and sufficient dataset images. The limited dataset images had approximately 15,000 training images, and the sufficient dataset images had approximately 30,000 training images.

[0079] The feasibility of the model is analyzed by comparing data augmentation, skeleton model, improved module, and optimizer improvements.

[0080] Visualize the output of DeiT using Grad-CAM.

[0081] Grad-CAM calculates the contribution of each token to the classification result based on the output and gradient of any Transformer encoder, and then maps these contributions back to the spatial position of the original image to form a heat map.

[0082] Since the last layer of ViT only has category labels, a Transformer encoder in the penultimate layer is selected as the target layer.

[0083] From the experiment, it can be found that the Transformer encoder mainly pays attention to the occluded parts in the image, verifying the interpretability of the model.

[0084] The specific implementation of the experiment from four aspects, data augmentation, skeleton model, improvement module, and optimizer, includes the following steps:

[0085] Data augmentation experiments were conducted when the dataset had limited images and when the dataset had sufficient images. The experiments showed that the improvements based on ViT were all effective improvements. When the dataset was limited, the data augmentation method could indeed improve the accuracy.

[0086] DeiT and ViT are introduced to compare skeleton models. The experimental results show that the improvement of DeiT over ViT comes at the cost of certain performance.

[0087] The module was improved, a new shuffling mechanism was designed, the input vector of the last Transformer layer was divided into three blocks, and then rearranged, and the Sh-ViT model was proposed. Experimental results showed that the model had obvious advantages over the original ViT model algorithm.

[0088] We introduced Adan, an optimizer for Transformer, and used default parameters for optimization. We compared the DeiT model with the default optimizer SGD. The results showed that the Adan optimizer with default parameters did not improve the model.

[0089] Comprehensively analyze the experimental results and explain the feasibility of the model.

[0090] In terms of data, the present invention designs affine transformation and perspective transformation image transformation methods to augment data in order to solve the problem of target tilt when looking down. In order to solve the problem of insufficient ambient light, a color enhancement image transformation method is designed. In order to solve the problem of mutual occlusion between equipment and personnel, a random erasure image transformation method is designed to augment data. In order to solve the problem of image differences between cameras, a style transfer method combining VAE and CycleGAN is designed.

[0091] The present invention introduces ViT and DeiT based on pre-training to the model, adjusts many hyperparameters of the model, and designs a shuffling module that groups vectors after shuffling them in random order. The shuffling module adds Gaussian noise and zero masking to further analyze the performance of the model.

[0092] In general, the present invention proposes a pedestrian re-identification model for base station operation and maintenance scenarios, verifies the effectiveness of data augmentation and shuffling modules, and uses Grad-CAM for data visualization.

[0093] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

[0094] The system provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not regarded as improper limitations of the present invention.

[0095] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs or any other form of storage medium known in the technical field. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

Claims

1. A pedestrian re-identification method based on deep learning, characterized in that: The method comprises: Obtain the personnel behavior distribution data set of security inspection and operation and maintenance scenarios; Perform data preprocessing, data augmentation based on image processing, and camera style migration based on CycleGAN combined with VAE on the personnel behavior distribution dataset to determine the personnel behavior dataset for security inspection and operation and maintenance scenarios; Based on the personnel behavior dataset of the security inspection and operation and maintenance scenario, we improved ViT's TransReID and built a pedestrian re-identification model. The pedestrian re-identification model is subjected to simulation experiments, training and verification to obtain the prediction results of the pedestrian re-identification model.

2. The pedestrian re-identification method based on deep learning according to claim 1, characterized in that: The personnel behavior distribution dataset includes jpg or png images with a source data style of 1920*1080 and their corresponding xml tags, where the xml tags mark the detection boxes of pedestrians in the jpg or png images; The data preprocessing step includes: cropping, naming, screening and dividing the jpg pictures or png pictures according to the tags; Data augmentation based on image processing, including: geometric transformation, random erasing, Gaussian blur and color transformation, wherein the geometric transformation includes affine transformation and focus transformation; Camera style transfer based on CycleGAN combined with VAE, including: modifying the CycleGAN network in CamStyle, replacing the generator in CycleGAN with VAE, and building a new structure of CycleGAN combined with VAE to perform conversion between camera styles.

3. The pedestrian re-identification method based on deep learning according to claim 1, characterized in that: Obtain the personnel behavior distribution data set of the security inspection and operation and maintenance scenario, including: Obtain the types of behaviors of personnel in security inspection and operation and maintenance scenarios; Based on the preset correspondence between the behavior types of personnel and the personnel categories, the personnel categories are determined according to the behavior types of personnel, and a personnel behavior distribution data set is obtained.

4. The pedestrian re-identification method based on deep learning according to claim 3, characterized in that: The types of personnel behavior include regular security inspection and maintenance operations and non-security inspection and maintenance behavior.

5. The pedestrian re-identification method based on deep learning according to claim 3, characterized in that: The personnel categories include maintenance personnel and strangers.

6. The pedestrian re-identification method based on deep learning according to claim 1, characterized in that: Based on the personnel behavior dataset of the security inspection and operation and maintenance scenario, we improved ViT's TransReID and built a pedestrian re-identification model, including: The image block is linearized, the position information is embedded, and it enters the shuffle module through N layers of Transformer layers. The input vector of the last layer of Transformer layer is divided into 3 blocks, rearranged, and connected to F g To F n As a Feature map feature map.

7. The method for pedestrian re-identification based on deep learning according to claim 6, characterized in that: The input vector of the last Transformer layer is divided into three blocks and rearranged, including: Assume the above formula is the input vector of the Lth layer, where Z 0 l-1 It is CLS token, and the remaining N columns are processed; After shuffling the input vectors and exchanging them, the embedded sequences are shuffled into groups as follows: Z l-1 Divided into 3 vectors, the expressions are as follows, and rearranged through vector connection; Z l-1 =[Feature2,Feature3,Feature1]。 8. The method for pedestrian re-identification based on deep learning according to claim 1, characterized in that: The simulation experiment includes four aspects: data augmentation, skeleton model, improvement module and optimizer.

9. The method for pedestrian re-identification based on deep learning according to claim 1, characterized in that: The pedestrian re-identification model is verified, including: Visualize the output of DeiT using Grad-CAM; Grad-CAM calculates the contribution of each token to the classification result based on the output and gradient of any Transformer encoder, and maps the contribution back to the spatial position of the original image to form a heat map. Select a Transformer encoder in the penultimate layer of ViT as the target layer; Determines which occlusions in the image the Transformer encoder notices.

Images