Pedestrian attribute recognition method based on multi-scale self-attention attribute positioning

Abstract

The application discloses a pedestrian attribute recognition method based on multi-scale self-attention attribute positioning, and comprises the following steps: preparing a pedestrian attribute recognition training set, including a human body attribute data set and a face attribute data set; designing a pedestrian attribute recognition network based on multi-scale self-attention attribute positioning, training a human body attribute recognition model and a face attribute recognition model; adopting a pedestrian attribute recognition network architecture based on multi-task learning, sharing main network parameters, and identifying each attribute by using independent branches; constructing a shared network structure, taking a pedestrian image including a human body image and a face image as input, and outputting shared features of each attribute; designing a pedestrian attribute positioning branch structure based on multi-scale self-attention, taking the shared features of each attribute output by the shared network as input of each branch attribute, and outputting attribute features corresponding to each branch attribute; and integrating the human body attribute recognition model and the face attribute recognition model, and fusing output pedestrian attribute features.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a pedestrian attribute recognition method based on multi-scale self-attention attribute positioning. BACKGROUND

[0002] With the development of smart cities and the growth of public safety needs, the number of intelligent video surveillance cameras is increasing, and it is far from enough to process the data generated by hundreds of millions of surveillance cameras through manual video monitoring, so intelligent video monitoring technology has emerged, and pedestrian attribute structured recognition in the monitoring scene is an important part of intelligent video monitoring technology. The technology has been widely used in smart security, advertising marketing and commercial retail fields.

[0003] Due to the complex environment in the monitoring scene and the fine-grained attribute labels, the pedestrian attribute recognition task in the monitoring scene has the following challenges: (1) different attributes: the attributes of pedestrians are different, some need shallow features, some need deep features, some need local features, and some need global features. How to extract a feature containing different attributes is crucial; at the same time, for some fine-grained attributes such as hats and hairstyles, detailed positioning is needed, and how to accurately position small features is particularly important. (2) Low resolution: the resolution of the pedestrian image captured at a distance in the monitoring scene is often small, mostly around 100*200. (3) Occlusion problem: the occlusion of some parts of the pedestrian will affect attribute recognition. SUMMARY

[0004] In view of the above problems, the application provides a pedestrian attribute recognition method based on multi-scale self-attention attribute positioning, which learns the positioning information of different granularity attributes, so that even for low-resolution pictures and small attributes, the key area of the attribute can be accurately positioned, and the accuracy of pedestrian attribute recognition is improved.

[0005] To solve the above technical problems, the application adopts the following technical solutions:

[0006] A pedestrian attribute recognition method based on multi-scale self-attention attribute positioning, comprising the following steps:

[0007] Step 1, prepare a pedestrian attribute recognition training set, including a human attribute data set and a face attribute data set; the human attribute data set includes a public data set PETA data set and a self-made monitoring scene human data set; the face attribute data set includes a public data set CelebA, an AFAD age data set, an EmotioNet expression data set and a self-made monitoring scene face data set;

[0008] Step 2, a pedestrian attribute recognition network based on multi-scale self-attention attribute positioning is designed, and a human attribute recognition model and a face attribute recognition model are trained;

[0009] Step 3, a pedestrian attribute recognition network architecture based on multi-task learning is adopted, and the main network parameters are shared, and each attribute is identified by an independent branch;

[0010] Step 4, a shared network structure is constructed, taking a pedestrian image including a human image and a face image as input, and outputting shared features of each attribute;

[0011] Step 5, a pedestrian attribute positioning branch structure based on multi-scale self-attention is designed, taking the shared features output by the shared network as the input of each branch attribute, and outputting the attribute features corresponding to each branch attribute;

[0012] Step 6, the human attribute recognition model and the face attribute recognition model are integrated, and the output pedestrian attribute features are fused.

[0013] In a possible implementation manner, the self-made human data set in a monitoring scene is taken from monitoring videos including subway, shopping mall and community scenes, a human body frame is detected by using a pedestrian target detector, and data cleaning and attribute labeling are performed after detection.

[0014] In a possible implementation manner, the self-made human data set in a monitoring scene is taken from monitoring videos including subway, shopping mall and community scenes, a human body frame is detected by using a pedestrian target detector, and data cleaning and attribute labeling are performed after detection.

[0015] In a possible implementation manner, the process of training the human attribute recognition model in step 2 is as follows:

[0016] The pedestrian image is detected by a human body detector to obtain a human image, and is scaled to 128*256 as input; the human image passes through a shared convolutional network, and a residual neural network Resnet is used to extract common feature maps of each attribute; the common feature maps enter attribute branches 1, 2, …, N respectively, N representing the number of identified attributes, an independent self-attention module is arranged in each attribute branch for attribute positioning, and a balanced pyramid is used for multi-scale positioning; the attribute features of each attribute branch are superimposed with positioning information, input into respective fully connected layers, and attribute prediction is performed, and each attribute is a two-class or multi-class classification.

[0017] In a possible implementation manner, the self-attention module adopts a spatial domain self-attention mechanism, and the process is as follows:

[0018] The original feature layer X is subjected to feature decomposition to obtain three matrices Q, K and V, wherein Q and K are different feature representation matrices; a sampling method is adopted to sample the three matrices; Q*K obtains a correlation matrix between different features, and the correlation matrix is multiplied by V to obtain an output.

[0019] In a possible implementation manner, the multi-scale positioning of the balanced pyramid comprises scaling, integration, refinement and strengthening steps, four different level features C2, C3, C4 and C5 of Resnet output are taken, C4 and C5 are up-sampled, C2 and C3 are down-sampled by pooling, the four levels are scaled to the same size, superimposed and integrated into a balanced semantic feature map, after the integration, a refinement layer is entered for further refinement of the features, the refinement layer adopts an embedded Gaussian non-local attention module, finally, each layer is re-scaled to the original size, superimposed with the original features for strengthening to obtain new feature maps P2, P3, P4 and P5.

[0020] In a possible implementation manner, step 6 specifically comprises: inputting the pedestrian body image and the face image into a body attribute recognition model and a face attribute recognition model respectively, outputting body attribute features feature and corresponding attribute quality scores feature_value and face attribute features featue_face and featue_face_value; performing quality score comparison on a common part of the two attribute features to perform weight weighting calculation, and simultaneously calculating a relationship matrix between different attributes, and outputting by weight weighting through the relationship matrix.

[0021] The application has the following beneficial effects: through the self-attention mechanism and the multi-scale mechanism, the focusing positions of different attributes can be positioned more accurately, pedestrian attribute information can be extracted more effectively, the accuracy of pedestrian attribute recognition in a monitoring scene is improved, great convenience is provided for pedestrian structured attribute search and positioning; and through learning of positioning information of different granularity attributes, even for low-resolution pictures and small attributes, the key regions of the attributes can be positioned accurately, and the accuracy of pedestrian attribute recognition is improved. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 A step flowchart of the pedestrian attribute recognition method based on multi-scale self-attention attribute positioning of the embodiment of the application is shown in the figure.

[0023] Figure 2 A structure schematic diagram of the body attribute recognition model in the embodiment of the application is shown in the figure.

[0024] Figure 3 A structure schematic diagram of a shared network structure of the pedestrian attribute recognition network in the embodiment of the application is shown in the figure.

[0025] Figure 4A spatial domain self-attention mechanism diagram of an embodiment of the present application is shown in FIG. 1.

[0026] Figure 5 A multi-scale positioning structure diagram of balancing the feature pyramid structure for each attribute in an embodiment of the present application is shown in FIG. 2. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0028] Reference Figure 1 A step flowchart of a pedestrian attribute recognition method based on multi-scale self-attention attribute positioning according to an embodiment of the present application is shown in FIG. 3, including the following steps:

[0029] Step 1, preparing a pedestrian attribute recognition training set, including a human attribute data set and a face attribute data set; the human attribute data set includes a public data set PETA data set and a self-made monitoring scene human data set; the face attribute data set includes a public data set CelebA, an AFAD age data set, an EmotioNet expression data set, and a self-made monitoring scene face data set.

[0030] In a specific application example, the human attribute data set includes a public data set PETA data set and a self-made monitoring scene human data set. PETA is a long-distance pedestrian attribute recognition data set, containing 19,000 images with a resolution range of 17x39 to 169x365 pixels. The 19,000 images include 8,705 individuals, each with 61 binary attributes and 4 multi-class attributes. The self-made monitoring scene data set is taken from monitoring videos of scenes including subways, shopping malls, and communities, etc. A pedestrian target detector is used to detect the human frame, such as the yolov5 target detection method. After detection, data cleaning and attribute labeling are performed, including but not limited to gender, age, hairstyle, backpack, handheld items, shirt style, shirt color, pants style, pants color, umbrella, handheld umbrella, posture, etc.

[0031] The face attribute dataset includes a public dataset CelebA, an AFAD age dataset, an EmotioNet expression dataset, and a self-made monitoring scene face dataset. CelebA has 20w data and 40 attribute labels. The AFAD dataset is an Asian face attribute set with 16w+ data. EmotioNet is an expression dataset, including 7 expressions such as smile, anger, fear, fear, surprise, disgust, and no expression. The self-made face attribute dataset is taken from monitoring videos in scenes such as subways, shopping malls, and communities, and uses a pedestrian target detector to detect face frames, including attributes such as gender, age, whether wearing a mask, whether wearing glasses, whether wearing a hat, expression, hairstyle, etc.

[0032] Step 2, design a pedestrian attribute recognition network based on multi-scale self-attention attribute positioning, train a human body attribute recognition model and a face attribute recognition model.

[0033] The structure of the human body attribute recognition model is as shown in Figure 2 The specific process is as follows: the pedestrian image is detected by a human body detector to obtain a human body image, which is scaled to 128*256 as input; the human body image passes through a shared convolutional network, and a residual neural network (Resnet) network is used to extract common feature maps of each attribute; the common feature maps enter attribute branches 1, 2,..., N, where N represents the number of recognized attributes, and 20 attributes are recognized in the example; each attribute branch has its own independent self-attention module for attribute positioning, and a balanced pyramid is used for multi-scale positioning, which can more effectively position attributes of different granularities; the attribute features of each attribute branch are superimposed with positioning information and input into respective fully connected layers for attribute prediction. Each attribute is a binary or multi-class classification, such as gender binary classification or hairstyle multi-class classification such as short hair, long hair, and baldness.

[0034] The pedestrian attribute recognition network based on multi-scale self-attention attribute positioning uses a multi-task learning architecture, and is equipped with a multi-scale self-attention module to improve attribute positioning accuracy and improve pedestrian attribute recognition accuracy.

[0035] The model training method is as follows: the human body data training set has a picture size of 128*256, and the face dataset has a picture size of 72*72. First, the model is pre-trained using public datasets, and then complete attribute training is performed using self-made datasets. The labeled data is divided into a training set, a validation set, and a test set in a ratio of 7:2:1 for model training and effect verification and testing.

[0036] The face attribute recognition model structure is similar to the human body attribute recognition model.

[0037] Step 3, a pedestrian attribute recognition network architecture based on multi-task learning is adopted, the main network parameters are shared, and each attribute is identified by an independent branch.

[0038] Pedestrian attribute recognition involves the simultaneous recognition of multiple attributes. To more efficiently perform recognition, a basic framework of multi-task learning is adopted, that is, the main network parameters are shared, and each attribute is identified by an independent branch. By sharing the feature expression between attributes, the model has better generalization performance in attribute recognition.

[0039] Step 4, a shared network structure is constructed, taking pedestrian images including human body images and face images as input, and outputting shared features of each attribute.

[0040] The shared network structure of the pedestrian attribute recognition network adopts Resnet. In a specific application example, Resnet50 is used to extract pedestrian attribute main features. The human attribute recognition network structure is as shown in Figure 3 Resnet50 contains five stages: Conv1, Conv2_x, Conv3_x, Conv4_x, and Conv5_x. Conv1 contains a 7*7 convolution, a batch normalization (BN) layer, and a Relu activation function layer. The main structure of the last four stages is a residual structure unit composed of a 1*1 convolution, a 3*3 convolution, and a 1*1 convolution. The main difference is the number of channels and the number of residual structure units in each stage. The residual structure of Resnet mainly adds the original input into the next stage in addition to the network output after each unit. This structure effectively solves the degradation problem of deep neural networks, that is, when the depth of the neural network increases to a certain extent, the effect of deeper network stacking becomes worse. This is because when the number of neural network layers is large, the correlation between the gradients becomes worse and finally approaches white noise. Such gradients are not beneficial to the model and instead randomly fluctuate. Resnet can effectively strengthen the connection between different layers and increase the correlation between gradients, better extracting deep network feature information.

[0041] Step 5, a pedestrian attribute positioning branch structure based on multi-scale self-attention is designed, taking the attribute shared features output by the shared network as the input of each branch attribute, and outputting the attribute features corresponding to each branch attribute.

[0042] Many existing methods identify pedestrian attributes as a multi-label problem, and only extract attribute features from a picture. Such methods rely on overall features, but pedestrian attributes are different, some need shallow features, some need deep features, some need local features, and some need global features. The region feature is more useful for fine-grained attribute classification. The commonly used attention mechanism is to first extract a relevant vector for generating attention scores from the feature maps of the previous layer using pooling or convolution, and then input it into a lightweight fc (full connection) layer or conv (convolution) layer to obtain the attention scores. After that, the attention scores are multiplied by the original feature maps. However, such attention scores only rely on a fully connected layer for training and learning, and the attention scores are not supervised learning, so the learned attention scores cannot fully reflect the relationship between features.

[0043] In a specific application example of the present application, a spatial domain self-attention mechanism is used, as shown in Figure 4 , and each attribute has an independent self-attention module. The attention of the spatial domain is relatively intuitive, and the human visual system has the feature of focusing on a certain area in the field of view to help quickly capture important information. For example, to observe the hairstyle, focus on the head and shoulders, and to observe the style and color of the lower body clothes, focus on the lower body.

[0044] The specific process of the spatial domain self-attention mechanism of the embodiment of the present application is as follows:

[0045] (1) The original feature layer X is decomposed to obtain three matrices Q (Query), K (Key) and V (Value). Q and K are different feature representation matrices, so Q*K can represent the relationship between different features. (2) In order to reduce the calculation amount of Q*K, the sampling method is adopted to sample the three matrices based on the idea of local linearization. (3) Q*K obtains the correlation matrix between different features, and finally the correlation matrix and V are multiplied to obtain the output.

[0046] The spatial domain self-attention mechanism used in the present application has stronger modeling ability for the correlation of different parts of the space, can better mine the relationship between features, and fully play the attention mechanism, because it directly calculates the correlation between each feature. At the same time, based on the idea that the features extracted in a local region are similar to the correlation of another region, the sampling method is used to reduce the operation amount of inner product.

[0047] In an embodiment of the present application, in order to more accurately locate the attributes, an embodiment of the present application uses a balanced feature pyramid structure to perform multi-scale positioning of each attribute, as shown in Figure 5As shown, including scaling, integration, refinement and strengthening, taking the 4 different levels of features C2, C3, C4 and C5 of the Resnet output, up-sampling C4 and C5, pooling and down-sampling C2 and C3, 4 levels are scaled to the same size and superimposed to balance the semantic feature map, after integration, entering the refine layer for further refinement of the features, the refine layer preferably uses an embedded Gaussian non-local attention module, finally each layer is rescaled to the original size, superimposed with the original features to strengthen and get new feature maps P2, P3, P4 and P5. Unlike the FPN method of integrating multi-level features using horizontal connection, the balanced feature pyramid uses multi-level feature information to enhance the expression ability of each level feature map, and strengthens the semantic information and positioning information of each layer.

[0048] Step 6, integrate the human body attribute recognition model and the face attribute recognition model, and fuse the output pedestrian attribute features.

[0049] In a specific application example, the human body image and the face image of the pedestrian are respectively input into the human body attribute recognition model and the face attribute recognition model, and the human body attribute feature feature and the corresponding attribute quality score feature_value and the face attribute feature featue_face and featue_face_value are output. The common part of the two attribute features is compared in quality score and weighted calculation, and the relationship matrix between different attributes is calculated, and the weighted output is output through the relationship matrix. The human body attribute and the face attribute have common attributes, such as gender and age, which can be judged from the face and also from the human body clothing, and the two are complementary, and different attributes are related, such as male and baldness attribute, female and long hair attribute, through the correlation between different attributes, the pedestrian attribute feature is more accurately output.

[0050] The above pedestrian attribute recognition method based on multi-scale self-attention attribute positioning has at least the following beneficial effects: (1) a pedestrian attribute recognition network architecture based on multi-task learning is adopted, i.e., main network parameter sharing, and each attribute is identified by an independent branch. Through this shared feature expression between attribute recognition, the model has better generalization performance in attribute recognition. (2) A pedestrian attribute positioning branch structure based on multi-scale self-attention is designed. The spatial domain self-attention mechanism is adopted to directly calculate the correlation between each feature, which has stronger modeling capability for the correlation of different parts of features in space and can better mine the correlation between features. At the same time, based on the idea that the correlation between the features extracted in the local region and another region is similar, a sampling method is used to reduce the computational amount of inner product. (3) A balanced feature pyramid structure is used for multi-scale positioning of each attribute, and multi-level feature information is used to enhance the expression ability of each level of feature map, and the semantic information and positioning information of each layer are enhanced. (4) The human body attribute recognition model and the face attribute recognition model are integrated, and the output pedestrian attribute features are fused, and through the correlation between different attributes, the pedestrian attribute features are more accurately output. (5) Through the self-attention mechanism and the multi-scale mechanism, the focusing position of different attributes can be more accurately positioned, and the pedestrian attribute information can be more effectively extracted, the accuracy of pedestrian attribute recognition in the monitoring scene is improved, and great convenience is provided for the structured attribute search and positioning of pedestrians. And by learning the positioning information of different granularity attributes, even for low-resolution pictures and small attributes, the key area of the attribute can be accurately positioned, and the accuracy of pedestrian attribute recognition is improved.

[0051] It should be understood that the example embodiments described herein are illustrative and not restrictive. Although one or more embodiments of the application are described in conjunction with the attached figures, it should be understood that various changes in form and detail can be made without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. A pedestrian attribute recognition method based on multi-scale self-attention attribute positioning, characterized by, The method comprises the following steps: Step 1, preparing a pedestrian attribute recognition training set, including a human body attribute data set and a face attribute data set; the human body attribute data set comprises a public data set PETA data set and a self-made monitoring scene human body data set; the face attribute data set comprises a public data set CelebA, an AFAD age data set, an EmotioNet expression data set and a self-made monitoring scene face data set; Step 2, designing a pedestrian attribute recognition network based on multi-scale self-attention attribute positioning, training a human body attribute recognition model and a face attribute recognition model; Step 3, adopting a pedestrian attribute recognition network architecture based on multi-task learning, sharing main network parameters, and identifying each attribute by using independent branches; Step 4, constructing a shared network structure, taking a pedestrian image comprising a human body image and a face image as input, and outputting shared features of each attribute; Step 5, designing a pedestrian attribute positioning branch structure based on multi-scale self-attention, taking the shared features of each attribute output by the shared network as input, and outputting attribute features corresponding to each attribute branch; Step 6, integrating the human body attribute recognition model and the face attribute recognition model, and fusing output pedestrian attribute features; In step 2, the process of training the human body attribute recognition model is as follows: a pedestrian image is detected by a human body detector to obtain a human body image, which is scaled to 128*256 as input; the human body image is input into a shared convolutional network, and a residual neural network Resnet is used to extract common feature maps of each attribute; the common feature maps are respectively input into attribute branches 1, 2,..., N, wherein N represents the number of attributes to be recognized, and an independent self-attention module is arranged in each attribute branch to perform attribute positioning, and a balanced pyramid is used for multi-scale positioning; the attribute features of each attribute branch are input into respective fully connected layers after being superimposed with positioning information, and attribute prediction is performed, and each attribute is a binary classification or multi-classification; The self-attention module adopts a spatial domain self-attention mechanism, and the process is as follows: The original feature layer X is decomposed to obtain three matrices Q, K and V, wherein Q and K are different feature representation matrices; a sampling method is used to sample the three matrices; the correlation matrix between different features is obtained by Q*K, and the correlation matrix is multiplied by V to obtain the output; The multi-scale positioning of the balanced pyramid comprises scaling, integrating, refining and strengthening steps; four different level features C2, C3, C4 and C5 output by the Resnet are scaled, C4 and C5 are up-sampled, C2 and C3 are down-sampled by pooling, the four levels are scaled to the same size and integrated into a balanced semantic feature map, the integrated feature is input into a refinement layer for further refinement, the refinement layer adopts an embedded Gaussian non-local attention module, and finally each layer is rescaled to the original size, the original feature is superimposed to strengthen the new feature map P2, P3, P4 and P5; The step 6 specifically comprises: inputting the pedestrian human body image and the human face image into a human body attribute recognition model and a human face attribute recognition model respectively, outputting human body attribute features feature and corresponding attribute quality scores feature_value and human face attribute features feature_face and feature_face_value; comparing the quality scores of the common parts of the two attribute features, performing weight weighting calculation, simultaneously calculating a relationship matrix between different attributes, and performing weight weighting output through the relationship matrix.

2. The pedestrian attribute recognition method based on multi-scale self-attention attribute positioning of claim 1, wherein, The self-made monitoring scene human body dataset is taken from monitoring videos including subway, shopping mall and community scenes, a pedestrian target detector is used to detect a human body frame, data cleaning and attribute labeling are performed after detection. 3.The pedestrian attribute recognition method based on multi-scale self-attention attribute positioning of claim 1, wherein, The self-made monitoring scene human face dataset is taken from monitoring videos including subway, shopping mall and community scenes, a pedestrian target detector is used to detect a human face frame, data cleaning and attribute labeling are performed after detection.

Images