A method for identifying imbalanced pedestrian attributes based on ResNeSt and online mining of hard samples

Abstract

This invention belongs to the field of computer vision, and particularly relates to pedestrian attribute recognition. Specifically, it relates to a pedestrian imbalance attribute recognition method based on ResNeSt and online hard sample mining, aiming to solve the problem of low attribute recognition accuracy caused by the imbalance of positive and negative pedestrian samples. This invention includes: using a ResNeSt50 network model as a feature extraction network to mine higher-granularity pedestrian features; using a channel attention mechanism to further salientize pedestrian semantic features; using an online hard sample mining algorithm to mine hard samples online; further improving the inter-class imbalance problem through a loss balance weighting strategy; and using a label smoothing strategy to alleviate the model's dependence on sample data. This invention can be applied to pedestrian attribute recognition tasks in various scenarios and can significantly improve the performance of pedestrian attribute recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and particularly relates to the field of pedestrian attribute recognition. Background Technology

[0002] The purpose of pedestrian attribute recognition is to mine the attribute features of pedestrians in a given image, classify these attributes, and ultimately predict the attribute labels of pedestrians in the test set, such as gender, clothing color, hat, and clothing style. Pedestrian attribute recognition is currently a popular field in computer vision, with wide applications in many areas, such as autonomous driving, intelligent transportation, and intelligent security. With the deepening development of artificial intelligence, pedestrian attribute recognition methods have gradually transitioned from primarily traditional machine learning methods to today's deep learning methods. Currently, the main methods for pedestrian attribute recognition tasks include attribute recognition methods based on conventional networks, attribute recognition methods based on local segmentation, attribute recognition methods based on attention mechanisms, and attribute recognition methods based on time series prediction, all of which have achieved good recognition accuracy. However, existing pedestrian attribute datasets exhibit a certain degree of attribute imbalance, meaning that some pedestrian attributes correspond to a large proportion of samples, while other pedestrian attributes correspond to a very small proportion, thus affecting the recognition performance of the algorithm. Especially in practical applications, attribute imbalance in training samples is frequently encountered; when the attribute imbalance is severe, the performance of the network model deteriorates significantly. This is a significant challenge currently facing pedestrian attribute recognition tasks, and it has not yet been well resolved. Therefore, designing a pedestrian attribute recognition method that can effectively solve this problem is of great importance. Summary of the Invention

[0003] (a) Technical problems to be solved:

[0004] Currently, the main methods for pedestrian attribute recognition tasks include attribute recognition methods based on conventional networks, attribute recognition methods based on local segmentation, attribute recognition methods based on attention mechanisms, and attribute recognition methods based on time series prediction, which have achieved good recognition accuracy. However, in practical applications, attribute imbalance in the training samples is frequently encountered, affecting the recognition performance of the algorithm, especially when the attribute imbalance is severe, the performance of the network model degrades significantly. This invention addresses this pressing problem by proposing a pedestrian imbalance attribute recognition method based on ResNeSt and online hard sample mining.

[0005] (II) Technical Solution:

[0006] To achieve the above objectives, this invention proposes a pedestrian imbalance attribute recognition model based on ResNeSt and online hard sample mining. This model consists of five parts: data augmentation, a feature extraction network, a channel attention mechanism, online mining of hard pedestrian samples, and a loss-balanced weighting strategy. Data augmentation utilizes random flipping, random rotation, random mirror flipping, affine transformation, and random pruning methods. Considering that the ResNeSt network can analyze deeper pedestrian attributes and better uncover relationships between attributes, this invention selects the ResNeSt50 network as the feature extraction network. A Channel Attention Merge (CAM) mechanism is used to organize the extracted semantic features of pedestrians and suppress unnecessary features, such as the influence of background noise, enabling the extraction of more detailed pedestrian semantic features. Then, the saliency-enhanced features are subjected to Global Average Pooling (GAP) to obtain the final features. Attribute prediction is then performed, setting a threshold for selecting hard samples and ranking them according to confidence level for online mining. Finally, the loss value corresponding to each attribute category is calculated using a loss function, and the total loss is obtained through a loss weighting strategy. The schematic diagram of this method is attached. Figure 1 .

[0007] 1. ResNeSt network and attention mechanism

[0008] Generally, attributes with a small sample size are local attributes that rely on fine-grained features (e.g., glasses, shoes, handbags, smoking, etc.) and are located in a relatively small area within pedestrian images. To better distinguish these attributes, more efficient feature extraction networks and attention mechanisms are needed to mine finer-grained semantic features. This invention uses the ResNeSt50 network as the feature extraction network to mine finer-grained features, combined with a channel self-attention mechanism (CAM) to organize the extracted semantic features of pedestrians and suppress certain unnecessary features, such as the influence of background noise, to extract more detailed pedestrian features.

[0009] The ResNeSt network is based on ResNet and adds a distributed attention module, as shown in the attached image. Figure 2 As shown in the diagram, this is a schematic of the distributed attention mechanism. In the distributed attention module, each input value is first weighted and balanced, then globally pooled, and then processed using an attention mechanism. Next, an r-softmax classifier is used for classification, and finally, the results are linearly combined with the output of the residual module for output. The formula for the r-softmax function in the distributed attention module is as follows:

[0010]

[0011] in, The weights of each slice are represented by , k represents the k-th cardinality group, R is the cardinality hyperparameter, representing the number of segments in the cardinality group, c represents the number of slices in each Cardinal, and s... k This indicates how many channels each slice has.

[0012] As attached Figure 3 The diagram shown illustrates the channel self-attention mechanism, where the input feature X∈R C×H×W After dimensionality reduction and compression of X, we obtain A, B, C ∈ R. C×N Multiplying the transposes of A and B gives W∈R C×C Then, the coefficient matrix D∈R is generated using Softmax. C×C The coefficient matrix primarily considers the relationships between channels. Since the generated A, B, and C do not undergo convolution operations, their relationship with the channel map is well preserved. The coefficient calculation formula is as follows.

[0013]

[0014] The attention feature map M∈R is obtained from C and D. C×H×W M then obtains the final channel domain output Y∈R by weighted summation of the learnable parameters θ and the original input X. C×H×W .

[0015] Y = θM + X (3)

[0016] 2. Online algorithm for difficult pedestrian sample mining

[0017] The algorithm flow for online mining of difficult pedestrian samples is as follows:

[0018] First, inputting the sample images of pedestrians into the pedestrian attribute recognition network model will yield the predicted label of the j-th attribute of the i-th pedestrian sample.

[0019] The obtained predicted label values ​​are then used to calculate the cross-entropy loss function value.

[0020] Finally, the predicted labels of the negative samples during the model training process are arranged in descending order of confidence. At the same time, the predicted labels of the negative samples are taken out according to the ratio of positive to negative samples 1:m, and the loss value is calculated by the cross-entropy loss function, where the value of m is greater than 1, and is generally around 3.

[0021] When the number of positive samples for a certain attribute category of pedestrians is too small, i.e., n < N min When / (1+m), where N min / (1+m) is the set threshold. Then, the predicted labels of the positive samples of the current attribute category are substituted into the loss function for calculation, and N is selected in descending order of confidence. min We take 1+m negative samples and select m of them in descending order of confidence to serve as positive samples in the loss calculation. This makes the distribution of positive and negative samples more balanced and can also uncover some hard samples, accelerating the effective training of the model. If the number of positive samples is too large, i.e., (1+m)*n exceeds the preset minimum number of negative samples, then all negative samples are taken during the iteration process. Here, n represents the number of positive samples corresponding to a certain attribute category of the pedestrian, and N is the number of positive samples. min The minimum number of negative samples to be selected is set to m, which is 3. N can be flexibly selected based on the model's performance. min / (1+m).

[0022] 3. Loss-balanced weighted strategy

[0023] By introducing a loss balancing weighting strategy, the loss function of each pedestrian attribute category is weighted and averaged, further improving the problem of inter-class imbalance. The final attribute classification loss function formula is as follows:

[0024]

[0025] Where w j The definition is as follows.

[0026]

[0027] Where: N represents the number of samples in the dataset; w j y represents the weight of the j-th attribute in the loss balance weighted sum strategy. It is a hyperparameter that needs to be set before training the model. In the pedestrian multi-attribute recognition task, if the classification effect of a certain attribute is not good, the weight of that class needs to be increased. The sum of the corresponding weights of all attribute classes is equal to 1. ij The true label represents the j-th attribute of the i-th pedestrian sample. In this chapter, the labels of all pedestrian samples are encoded using one-hot encoding. If the label exists, it is recorded as 1; otherwise, it is recorded as 1. σ is the hyperparameter that needs to be set during training. In this chapter, it is set to 1. In formula (4) The predicted label representing the j-th attribute of the i-th pedestrian sample is calculated as follows:

[0028]

[0029] (III) Beneficial Effects:

[0030] This invention provides a method for identifying imbalanced pedestrian attributes based on ResNeSt and online mining of hard samples. It has the following advantages:

[0031] 1. This invention uses the ResNeSt50 network as the feature extraction network to further mine more fine-grained pedestrian features, and combines it with a channel self-attention mechanism to organize the extracted semantic features of pedestrians, which can extract more detailed pedestrian semantic features and thus more effectively distinguish local hard attributes.

[0032] 2. This invention employs an online method for mining difficult pedestrian samples. During training, a portion of difficult samples are mined online according to a pre-set threshold, reducing the noise impact on model training when there are too many mislabeled samples. When the number of positive samples in a certain category is less than the set threshold, a certain number of negative samples are selected by confidence ranking to serve as positive samples for loss calculation. This results in a more balanced distribution of positive and negative samples during training, accelerating model convergence.

[0033] 3. This invention further improves the problem of class imbalance by using a loss balance weighting strategy, and alleviates the model's dependence on sample data by using a label smoothing strategy. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0035] Figure 1 The attached figure is a schematic diagram of the pedestrian imbalance attribute identification method based on ResNeSt and hard sample online mining;

[0036] Figure 2 The attached diagram illustrates the mechanism of distraction.

[0037] Figure 3 The attached diagram is a schematic diagram of the channel attention mechanism;

[0038] Figure 4 The attached figure is a flowchart illustrating the system implementation of the present invention;

[0039] Figure 5 The attached figure is a flowchart of the online sampling process for difficult pedestrian samples according to the present invention; Detailed Implementation

[0040] This invention is a method for identifying imbalanced pedestrian attributes based on ResNeSt and online mining of hard samples. The system implementation flowchart is attached. Figure 4 The specific implementation steps are as follows:

[0041] Step S1: Load the dataset to be used and select the corresponding data augmentation method according to different use cases.

[0042] Step S2: Input the preprocessed image data into the ResNeSt50 pre-trained model on ImageNet for training, and extract pedestrian features with certain semantic information.

[0043] Step S3: Using the channel self-attention mechanism to weight the pedestrian features extracted by the backbone network can obtain more detailed features.

[0044] Step S4: Obtain 2048-dimensional features through the last convolutional layer of the network, and then obtain the final features through global average pooling.

[0045] Step S5: Perform multi-attribute joint training on the network. That is, the classification layer of the network is a multi-branch classification network, each attribute has its own classifier, and the output of the network is the result of the combination of multiple classifiers.

[0046] Step S6: Obtain positive and negative pedestrian prediction labels through forward propagation of the network.

[0047] Step S7: Use the pedestrian difficult sample mining algorithm to perform online mining of pedestrian difficult samples. The process is as follows, and the flowchart of the algorithm can be found in [link to flowchart]. Figure 5 .

[0048] In step S7, the sample images of pedestrians are first input into the pedestrian attribute recognition network model, which will then yield the predicted label of the j-th attribute of the i-th pedestrian sample.

[0049] In step S7, the obtained predicted label values ​​are then used to calculate the cross-entropy loss function value.

[0050] In step S7, the predicted labels of the negative samples during the model training process are arranged in descending order of confidence. At the same time, the predicted labels of the negative samples are taken out according to the ratio of positive to negative samples 1:m, and the loss value is calculated by the cross-entropy loss function, where the value of m is greater than 1, and is generally around 3.

[0051] In step S7, when the number of positive samples for a certain attribute category of pedestrians is too small, i.e., n < N min When / (1+m), where N min / (1+m) is the set threshold. Then, the predicted labels of the positive samples of the current attribute category are substituted into the loss function for calculation, and N is selected in descending order of confidence. minWe take 1+m negative samples and select m of them in descending order of confidence to serve as positive samples in the loss calculation. This makes the distribution of positive and negative samples more balanced and can also uncover some hard samples, accelerating the effective training of the model. If the number of positive samples is too large, i.e., (1+m)*n exceeds the preset minimum number of negative samples, then all negative samples are taken during the iteration process. Here, n represents the number of positive samples corresponding to a certain attribute category of the pedestrian, and N is the number of positive samples. min The minimum number of negative samples to be selected is set to m, which is 3. N can be flexibly selected based on the model's performance. min / (1+m).

[0052] Step S8: By introducing a loss balancing weighting strategy, the loss function of each pedestrian attribute category is weighted and averaged to obtain the final attribute classification loss function formula as follows:

[0053]

[0054] Where w j The definition is as follows.

[0055]

[0056] Where: N represents the number of samples in the dataset; w j y represents the weight of the j-th attribute in the loss balance weighted sum strategy. It is a hyperparameter that needs to be set before training the model. In the pedestrian multi-attribute recognition task, if the classification effect of a certain attribute is not good, the weight of that class needs to be increased. The sum of the corresponding weights of all attribute classes is equal to 1. ij The true label represents the j-th attribute of the i-th pedestrian sample. In this chapter, the labels of all pedestrian samples are encoded using one-hot encoding. If the label exists, it is recorded as 1; otherwise, it is recorded as 1. σ is the hyperparameter that needs to be set during training. In this chapter, it is set to 1. In formula (4) The predicted label representing the j-th attribute of the i-th pedestrian sample is calculated as follows:

[0057]

[0058] Step S9: In addition to formula (4), this invention also employs a label smoothing regularization strategy during the training process. The basic idea of ​​label smoothing regularization (LSR) is to assign a small value to the non-true class, rather than zero. By introducing noise, the model will not overly trust the true class during training, thus alleviating the overfitting phenomenon during training.

[0059] Step S10: Use the final loss function obtained in step S9 to perform supervised training on the model, and perform backpropagation optimization on the model according to the iteration rounds.

[0060] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. The above descriptions are merely preferred embodiments of the present invention; any changes or modifications made within the scope defined by the invention's claims are within the protection scope of the present invention.

Claims

1. A method for identifying imbalanced pedestrian attributes based on ResNeSt and hard sample online mining, characterized in that, The specific implementation steps are as follows: Step S1: Load the dataset to be used and select the corresponding data augmentation method according to different use cases; Step S2: Input the preprocessed image data into the ResNeSt50 pre-trained model on ImageNet for training, and extract pedestrian features with semantic information; Step S3: Use the channel self-attention mechanism to weight the pedestrian features extracted by the backbone network; Step S4: Obtain 2048-dimensional features through the last convolutional layer of the network, and then obtain the final features through global average pooling. Step S5: Perform multi-attribute joint training on the network. That is, the classification layer of the network is a multi-branch classification network, each attribute has its own classifier, and the output of the network is the result of the combination of multiple classifiers. Step S6: Obtain pedestrian positive and negative prediction labels through forward propagation of the network; Step S7: Use the pedestrian hard sample mining algorithm to mine pedestrian hard samples online. The process is as follows: First, input the pedestrian sample image into the pedestrian attribute recognition network model to obtain the predicted label of the j-th attribute of the i-th pedestrian sample. The obtained predicted label values ​​are used to calculate the cross-entropy loss function value. The predicted labels of negative samples during model training are arranged in descending order of confidence. Simultaneously, negative sample predicted labels are extracted according to a 1:m ratio of positive to negative samples, and the loss value is calculated using the cross-entropy loss function, where m is set to 3. When the number of positive samples n for a certain attribute category of pedestrians is too small, i.e., n < N... min When / (1+m), where N min / (1+m) is the set threshold, N min To determine the minimum number of negative samples to select, the predicted labels of the positive samples for the current attribute category are substituted into the loss function for calculation, and N samples are selected in descending order of confidence. min The system takes 1+m)*n negative samples and selects m of them in descending order of confidence to serve as positive samples in the loss calculation. If the number of positive samples is too large, i.e. (1+m)*n is greater than the preset minimum number of negative samples, then all negative samples are taken during the iteration process. Step S8: By introducing a loss balancing weighting strategy, the loss function of each pedestrian attribute category is weighted and averaged to obtain the final attribute classification loss function formula as follows: in The definition is as follows: Where: N represents the number of samples in the dataset; The weighting weight of the j-th attribute in the loss balance weighted sum strategy is such that the sum of the corresponding weights of all attribute categories is 1. represents the true label of the j-th attribute of the i-th pedestrian sample. The labels of all pedestrian samples are encoded using one-hot encoding. If the label exists, it is recorded as 1, otherwise it is recorded as 0. σ is a hyperparameter that needs to be set during training, and it is set to 1. The predicted label representing the j-th attribute of the i-th pedestrian sample; The calculation expression is as follows: Step S9: Employ a label smoothing regularization strategy during training; Step S10: Supervised training of the model is performed using the loss function, and backpropagation optimization is performed on the model according to the iteration rounds.

Images