A pedestrian attribute recognition method suitable for complex monitoring scenes

Abstract

A pedestrian attribute recognition method based on removing the redundant attribute co-occurrence bias learned by the model in the inference process in the monitoring scene to achieve better generalization performance in the changing scene. It includes: a new generation method of attribute information decoupling features. In the model's identification and inference process of a certain attribute, by minimizing the mutual information between the specialized features of this attribute and the prediction probability of other attributes, the use of other attribute information is reduced, better accuracy is achieved under the premise of significantly reducing the training time; a new feature interpolation method. The invention proposes a direction-norm separation feature interpolation method, which can better explore and explore the distribution domain of attribute features compared with the original interpolation process, thereby improving the final effect of the model; using a convolutional neural network, the model is trained according to the above algorithm framework with specific training parameter configuration to obtain a pedestrian attribute recognition model. The method can achieve more accurate and stable pedestrian attribute recognition results in the spatio-temporal changing monitoring scene.

Description

Technical Field

[0001] The present invention relates to a pedestrian attribute recognition method suitable for complex monitoring scenes. Background Art

[0002] In the past, security professionals needed to manually search pedestrian surveillance data collected from cameras scattered across city streets and alleyways, or researchers wanted to track and analyze trends in soft biometric characteristics across a broad population at a sociological level using pedestrian images captured by video cameras. Due to the lack of intelligent, structured descriptions of pedestrian images, manual identification and labeling of the massive amount of pedestrian surveillance samples required for retrieval often required significant human and material resources. In recent years, the rapid development of big data, the Internet of Things, and computer vision-based technologies has made intelligent analysis of large-scale pedestrian surveillance data possible. Consequently, pedestrian attribute recognition algorithms have emerged as a key component of intelligent pedestrian analysis technology.

[0003] In engineering practice, deep neural networks (DNNs), with their generalization capabilities far exceeding those of traditional vision algorithms and their relatively well-defined and standardized engineering development and optimization processes, have become a key research tool in the field of general vision, including pedestrian attribute recognition. This technology uses digital simulations to establish a mathematical model of the transmission and activation of electrical signals between neurons in the brain. Using backpropagation algorithms, it acquires the ability to capture and learn underlying statistical regularities and visual patterns in labeled data samples. When deep neural networks were first introduced to pedestrian attribute recognition, the task was simply viewed as a multi-label classification task of individual pedestrian attributes using convolutional neural networks. The overall algorithmic framework is as follows: First, the input image is passed through a convolutional network to extract visual features. Subsequently, these features are passed to a classifier, which, through gradient descent learning, ultimately outputs a prediction of the probability of each attribute being present in the pedestrian image. Subsequently, to better explore and utilize the prior relative positional relationships between attributes and specific human structures, the "localization-first, recognition-later" approach to human attributes has gradually become a major research direction in this field. The main research topic in this direction is how to achieve accurate positioning of attributes without providing attribute space positioning annotations, so as to remove the influence of irrelevant redundant information and noise on the final recognition results.

[0004] Due to their inherently supervised learning nature, deep network-based pedestrian attribute recognition algorithms often require massive amounts of well-labeled pedestrian surveillance images to ensure stable and reliable recognition accuracy. However, due to the significant social security implications of pedestrian surveillance data, extensive collection of such data is often costly and difficult in practical applications. Consequently, current public pedestrian image datasets often only contain pedestrian image data from a specific, fixed scene within a specific time period. The learned visual patterns of pedestrian attributes often fail to generalize well to other scenarios that may vary in real-world applications. Specifically, the correlation and co-occurrence probability between pedestrian attributes may vary across different seasons and scenarios: in midsummer, the clothing combination of "shorts" may more often co-occur with "short-sleeved shirts," while in autumn, the attribute that appears more frequently with "short-sleeved shirts" may be "long pants," not "shorts." Similarly, the co-occurrence relationship between the attributes "children" and "schoolbags," which is common around elementary schools, may be less common on university campuses. However, due to the lack of self-decoupling function between attributes, neural networks tend to apply the attribute co-occurrence patterns learned in a limited dataset to the attribute reasoning process of pedestrian images that the model may face in actual applications, which come from spatiotemporal scenarios completely different from the dataset. This leads to the limited generalization ability of existing methods when facing changing scenarios in practice.

[0005] Currently, pedestrian attribute recognition is primarily used in surveillance scenarios and specific natural settings. However, because the correlation and co-occurrence patterns between attributes vary with time and location, this creates a data bias that can be incorrectly learned and exploited by models, hindering the generalization ability of existing technologies to varying scenarios outside of datasets. Summary of the Invention

[0006] To address the aforementioned issues in existing technologies, this paper presents an efficient pedestrian attribute recognition method suitable for complex surveillance scenarios, based on decoupled attribute information. This method, through an information-theoretic optimization approach, avoids leveraging feature information from other attributes when reasoning about a particular pedestrian attribute. This eliminates attribute co-occurrence bias and achieves pedestrian attribute recognition that generalizes well despite spatiotemporal variations in the scene.

[0007] According to one embodiment of the present invention, a pedestrian attribute recognition method applicable to complex monitoring scenarios includes:

[0008] S1. Generate features after decoupling attribute information. In the recognition and reasoning process of pedestrian attributes based on the ConvNeXt feature extractor model, the linear interpolation of features matches the linear interpolation of labels based on the experimental observation in previous studies. By minimizing the mutual information between the specialized features of different pedestrian attributes, excessive interdependence between the recognition results of different pedestrian attributes is avoided.

[0009] S2. A new feature interpolation method: This invention proposes a feature interpolation method of direction-norm separation. Compared with the original interpolation process, this method can better discover and explore the distribution domain of attribute features, thereby improving the final effect of the model.

[0010] S3. Use a convolutional neural network to train the model according to the above algorithm framework and specific training parameter configuration.

[0011] Wherein, the step S1 specifically includes:

[0012] S1.1) Decompose the mixed attribute information of visual features generated by ResNet-50 or ConvNeXt-base feature extractors into attribute-specific features for single attribute recognition through a single fully connected hierarchical decomposition.

[0013] S1.2) For the attribute-specific features generated in S1.1 for the input sample, perform linear interpolation with the attribute-specific features of another sample in the same training batch, using interpolation coefficients randomly sampled from a uniform distribution of 0-1. This generates new attribute-specific features that have been perturbed by the interpolation within the attribute feature distribution domain. The interpolated attribute-specific feature matrix is ​​summed across the attribute category dimension and directly fed into the classifier, which is instructed to predict the interpolation result for the attribute label corresponding to the attribute-specific feature interpolation coefficient.

[0014] The step S2 comprises:

[0015] S2 is an improvement to the feature interpolation method used in S1. In S2, we propose to interpolate two specialized feature vectors from different samples corresponding to an attribute with the same label twice, using different interpolation coefficients in the norm dimension and the direction vector dimension respectively. The direction vector interpolation result is multiplied by the norm interpolation result to produce the final attribute-specialized vector after interpolation perturbation applied in S1. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The figure is a flowchart of a pedestrian attribute recognition method applicable to complex monitoring scenarios according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The purpose of the present invention is to learn a feature after decoupling of various attribute information so that pedestrian attribute recognition based on this feature satisfies the requirement that the model's prediction of a certain attribute does not depend on any information from other attributes.

[0018] The core function of the present invention is that, during the model training process, we ensure that the semantic features used to identify each attribute are independent of each other through the proposed training method (that is, after the pedestrian attribute recognition model training is completed, the features extracted by the neural network for a certain pedestrian attribute recognition do not contain information that can be used to identify other attributes), that is, there will be no feature factor that affects the recognition results of two or more attributes at the same time. Specifically, we achieve this function by adding an additional training restriction: we hope that if the attribute-specific features used to identify a certain attribute in the model are changed, the model's prediction probability for other attributes will not change. At the same time, using the features after decoupling the attribute information, we randomly combine the attribute features belonging to different pedestrian individuals in the feature space, and use this randomly combined feature and its corresponding randomly combined label to train the model, perform data enhancement at the attribute co-occurrence level, and generate attribute co-occurrence patterns that have not appeared in the training data set.

[0019] We hope that through the proposed method, we can achieve an attribute recognition mechanism that is closer to that of humans in the deep pedestrian attribute recognition model. That is, the recognition result of the intelligent agent for a certain pedestrian attribute should not depend on the presence or absence of other attributes, but only on the information of the attribute itself, thereby avoiding the model's memorization and utilization of the limited attribute co-occurrence patterns in the dataset, and enhancing the model's generalization ability in changing scenarios.

[0020] According to one embodiment of the present invention, a pedestrian attribute recognition method applicable to complex monitoring scenarios includes the following steps:

[0021] S1. Generate features after decoupling attribute information, including:

[0022] In the process of pedestrian attribute recognition and inference based on the ConvNeXt feature extractor, based on the experimental observation that the linear interpolation of features matches the linear interpolation of labels, the mutual information between different pedestrian attribute specialized features is minimized to avoid excessive interdependence between different pedestrian attribute recognition results.

[0023] According to a further embodiment of the present invention, the pedestrian attribute recognition method applicable to complex monitoring scenarios further includes:

[0024] S3. Training step, including training a pedestrian attribute recognition model using a convolutional neural network.

[0025] Symbol Definition

[0026] min: Minimization symbol, indicating that the optimization goal is to minimize the formula after the symbol

[0027] P(): a probability symbol indicating that the classifier in the method of the present invention outputs a set of predictions on the probability of occurrence of each pedestrian attribute after receiving the input in the brackets.

[0028] The summation symbol adds the items defined subsequently from 1 to k according to the ordinal numbers specified by the superscript and subscript.

[0029] Loss: Loss function symbol, refers to the objective function used to generate gradients for backpropagation parameter updates.

[0030] A[:,i,:]: Matrix element representation. This expression, for example, represents the i-th element in the second dimension of the three-dimensional matrix A. If the shape of the matrix A is [A,B,C], then the shape of A[:,i,:] is [A,C].

[0031] A i : Feature matrix number. Indicates that the feature is generated by the i-th sample in the dataset.

[0032] ||||: Vector L2 norm symbol, indicating that the L2 norm of the vector within the symbol is calculated.

[0033] Wherein, the step S1 includes:

[0034] S1.1) As shown in Figure 1, a ResNet-50 or ConvNeXt-base feature extractor is first used to generate visual features with mixed attribute information and shape [N, K], where N is the batch size of the training batch and C is the total number of attributes to be predicted. This visual feature is then processed through a fully connected layer with a weight matrix of size [K, C*K], generating fully connected layer-processed features of size [N, C*K]. This fully connected layer-processed feature matrix A is then rearranged to obtain an attribute-specific feature matrix A of shape [N, C, K]. The feature components in each channel along the second dimension (shape [N, 1, K]) are attribute-specific feature vectors corresponding to a specific attribute prediction. Using this pedestrian attribute recognition method suitable for complex surveillance scenarios, we expect that the attribute-specific feature vector corresponding to A[:, j, :] only contains feature semantics that can be used to identify the jth pedestrian attribute, and does not contain any feature semantics that can reduce the uncertainty of attributes other than the jth attribute. In other words, the attribute-specific feature vectors correspond one-to-one to the attribute to be recognized.

[0035] S1.2) Generate N*C random numbers from a uniform distribution between 0 and 1, rearrange these random numbers into an interpolation weight matrix of shape [N, C, 1], and replicate the last dimension of the interpolation weight matrix K times to obtain an interpolation matrix W of shape [N, C, K]. This is then linearly combined (element-wise multiplication) with the attribute-specific feature matrix A obtained in S1.1 as follows:

[0036]

[0037] Where A' is the rearranged attribute-specialized feature matrix obtained by randomly shuffling A on the first dimension. The random interpolation matrix obtained above is That is the attribute-specialized feature matrix after perturbation and interpolation.

[0038] S1.3) As shown in Figure 1, All attribute-specialized feature vectors in the second dimension are added together and fed directly into the classifier, which is then asked to predict the label results corresponding to the interpolation matrix W used to interpolate the corresponding attribute labels.

[0039] in:

[0040] Add an extra dimension to the attribute label matrix of shape [N, C], and perform K replication operations on the added dimension of the attribute label matrix to obtain a three-dimensional attribute label matrix Y of shape [N, C, K].

[0041] Then, based on the interpolation matrix W generated in S1.2, the following operations are performed to generate the interpolated three-dimensional attribute label matrix

[0042]

[0043] The Y' matrix is ​​the re-shot three-dimensional label matrix after rearranging the first dimension in the same scrambled order as when generating A'. It is the interpolated three-dimensional attribute label matrix with shape [N,C,K].

[0044] After that, the attribute-specialized feature matrix obtained in S1.2 after interpolation is summed in the second dimension to obtain the [N, K]-dimensional summed and reduced feature matrix, and the summed and reduced feature matrix is ​​passed to the classifier. Through the cross entropy loss, the attribute prediction probability output generated by the classifier is the same as the interpolated three-dimensional attribute label matrix obtained in the above formula, that is, the training objective function is

[0045]

[0046] Using the above formula, the predicted probability value of the jth attribute depends only on the attribute-specific feature matrix after interpolation The j-th attribute-specialized feature of the second dimension (generating the attribute-specialized feature matrix after interpolation The interpolation weight of the j-th attribute-specialized feature of the second dimension and the interpolated, three-dimensional attribute label matrix The interpolation weight for the jth attribute label in the second dimension of is the same. ) and is independent of changes in the attribute-specific feature vectors corresponding to positions other than the traversed position j (because they correspond to different interpolation weights). The training objective function above ensures that the attribute-specific feature vectors corresponding to positions other than the traversed position j do not contain feature semantics that affect the prediction result of the jth attribute, ensuring the information decoupling between the prediction probability of a specific attribute and other attribute-specific features.

[0047] According to a further embodiment of the present invention, the pedestrian attribute recognition method applicable to complex monitoring scenarios further includes the following steps:

[0048] S3) Training step, including:

[0049] S3.1) Set the initial learning rate of the pedestrian attribute recognition model to 0.0001. After a certain number of update steps, the learning rate is reduced by 10%. The training process uses the Adam optimizer for multi-batch parameter updates, and the data batch size for a single iteration is 64.

[0050] S3.2) The input image is bilinearly interpolated and resized to 256 x 192. During training, a data batch is constructed using the data augmentation method of random horizontal rotation and random cropping.

[0051] S3.3) Set the maximum number of training iterations, such as 60. After the model reaches the maximum number of training iterations, save the model parameters that have converged for subsequent testing.

[0052] The above disclosure is only a specific embodiment of the present invention. Without departing from the scope of the claims of the present invention, those skilled in the art may make corresponding changes and adjustments based on the basic technical concept and framework provided by this invention.

Claims

1. A pedestrian attribute recognition method suitable for complex monitoring scenarios, characterized by The following steps are involved: S1. Generating features after decoupling attribute information, including: in the process of pedestrian attribute recognition and inference using a pedestrian attribute recognition model based on ConvNeXt as a feature extractor, minimizing the mutual information between different pedestrian attribute specialized features based on experimental observations that linear interpolation of features matches linear interpolation of labels, so as to avoid excessive interdependence between different pedestrian attribute recognition results. The step S1 comprises: S1.1) First, use ResNet-50 or ConvNeXt-base feature extractors to generate visual features with mixed attribute information and shape [N, K], where N is the batch size of the training batch and K is the dimension of the feature vector. After the visual feature is processed by a fully connected layer with a weight matrix size of [K, C*K], a fully connected layer processed feature of size [N, C*K] is generated, where C is the total number of attributes to be predicted. After that, the features after processing the fully connected layer are rearranged to obtain a three-dimensional attribute-specific feature matrix A of shape [N, C, K], where the feature component in each channel on the second dimension of A is the attribute-specific feature vector corresponding to the prediction of a certain attribute, where the shape of the feature component in each channel on the second dimension of A is [N, 1, K], where A[:, i, :] represents the i-th element on the second dimension of the three-dimensional matrix A, and when the shape of A is [A, B, C], the shape of A[:, i, :] is [A, C], then it is expected that the attribute-specific feature vector corresponding to A[:, j, :] only contains feature semantics that can be used to identify the j-th pedestrian attribute, and does not contain any feature semantics that can reduce the uncertainty of other attributes other than the j-th attribute, that is, the attribute-specific feature vector corresponds one-to-one to the attribute to be identified. S1.2) Generate N*C random numbers from a uniform distribution between 0 and 1, rearrange these random numbers into an interpolation weight matrix of shape [N, C, 1], and replicate the interpolation weight matrix K times in its third dimension to obtain an interpolation matrix W of shape [N, C, K]. This is then linearly combined with the attribute-specific feature matrix A, i.e., element-wise multiplication, as follows: Where A' is the rearranged attribute-specialized feature matrix obtained by randomly shuffling A on the first dimension, and the matrix obtained by random interpolation is That is, the attribute-specialized feature matrix after perturbation and interpolation, S1.3) All attribute-specialized feature vectors in the second dimension are added together and fed directly into the classifier, which is then asked to predict the label results corresponding to the interpolation matrix W used to interpolate the corresponding attribute labels. in: Add an extra dimension to the attribute label matrix of shape [N, C], and perform K replication operations on the extra dimension of the attribute label matrix to obtain a three-dimensional attribute label matrix Y of shape [N, C, K]. Then, based on the interpolation matrix W, the following operations are performed to generate the interpolated, three-dimensional attribute label matrix The Y' matrix is ​​the rearranged three-dimensional label matrix after rearranging the first dimension in the same scrambled order as when generating A'. is an interpolated, three-dimensional attribute label matrix of shape [N,C,K], After that, the interpolated attribute is specialized into the feature matrix The sum is performed in the second dimension to obtain the [N, K]-dimensional summed and reduced feature matrix, and the summed and reduced feature matrix is ​​passed into the classifier. Through the cross entropy loss, the attribute prediction probability output generated by the classifier is made the same as the interpolated three-dimensional attribute label matrix Y obtained in the above formula, that is, the training objective function is in: Loss is the symbol of the loss function, which represents the objective function used to generate the gradient of back propagation for parameter update. P() is a probability symbol, which means the classifier outputs a set of predictions on the probability of each pedestrian attribute after receiving the input in the brackets. Using the above formula, the predicted probability value of the jth attribute depends only on the attribute-specific feature matrix after interpolation The j-th attribute-specialized feature of the second dimension, that is, the attribute-specialized feature matrix after interpolation is generated The interpolation weight of the j-th attribute-specialized feature of the second dimension and the interpolated, three-dimensional attribute label matrix The interpolation weight of the j-th attribute label of the second dimension is the same, and is independent of the changes in the attribute specialization feature vectors corresponding to other positions outside the traversed position j. The objective function ensures that the attribute-specific feature vectors corresponding to positions other than the traversed position j do not contain feature semantics that affect the prediction result of the j-th attribute, ensuring the information decoupling between the prediction probability of a specific attribute and other attribute-specific features.

2. The pedestrian attribute recognition method according to claim 1, characterized in that Further including training steps, including: S3.1) Set the initial learning rate of the pedestrian attribute recognition model, After a specific number of update steps, the learning rate is reduced by 10%. The training process uses the Adam optimizer for multi-batch parameter updates, and the data batch size for a single iteration is 64. S3.2) The input image is bilinearly interpolated and resized to 256 x 192, where a data batch is constructed using the data augmentation method of random horizontal rotation and random cropping. S3.3) Set the maximum number of training iterations. After the model reaches the maximum number of training iterations, save the model parameters that have converged for subsequent testing.

3. The pedestrian attribute recognition method according to claim 2, characterized in that: The initial learning rate of the pedestrian attribute recognition model is set to 0.0001.

4. The pedestrian attribute recognition method according to claim 2, characterized in that: Set the maximum number of training iterations to 60.

Images