Pedestrian attribute recognition method adaptive to complex view angles and pedestrian states

Abstract

Pedestrian attribute recognition (PAR) aims to generate structured descriptions of pedestrians and is of great importance in surveillance. Existing methods mainly focus on improving the spatial localization of attributes during training. However, these methods fail to address the difference between the single-view poses in the dataset and the complex-view poses encountered in reality, resulting in poor attribute localization performance in open environments. Therefore, this paper introduces a new multi-view PAR method that uses human keypoint information to adapt to complex-view poses. This method uses key points to model the spatial distribution of attribute features, facilitating weakly supervised localization to simulate the dynamic relationship between attributes and the human body in complex views. In addition, it also combines spatial position encoding to model the human body structure information and implicitly capture the relationship between attributes and the human body. Finally, we apply a feature fusion method to make the spatial position information play a greater role.

Description

Technical Field

[0001] This invention provides a pedestrian attribute recognition method adapted to complex perspectives and pedestrian states, belonging to the field of computer vision. Background Technology

[0002] With the continuous advancement of urban digitalization, security monitoring using surveillance video information has become a fundamental infrastructure for smart society governance, playing an increasingly important role in daily life. In the identification tasks of surveillance, Pedestrian Attribution Recognition (PAR) is a crucial component of analyzing surveillance video content. It primarily involves transforming continuous visual information of the human body captured in images into structured descriptions. This process is essential for applications such as re-identification, pedestrian detection, and personnel retrieval, where the structured description of individuals plays a key role.

[0003] An attribute is a fixed property or characteristic possessed by an object. For pedestrian attributes, a pedestrian's attributes remain constant over a period of time. However, for static images, the state of the pedestrian varies in each image, leading to different ways attributes are presented. This makes attribute recognition difficult, as it involves finding common features from multiple states. Existing research methods mainly adapt to state changes in two ways: methods that focus on attribute feature extraction without considering the relationships between attributes, and methods that focus on utilizing the semantic relationships between attributes. Both of these methods show limitations in adapting to the spatial complexity of attributes.

[0004] We categorize the states of static pedestrian images into two types: changes originating from the camera and changes inherent to the pedestrian themselves. For the spatial characteristics of attributes, these correspond to changes in viewpoint and pose, respectively. These two types of changes alter the spatial representation of attributes, thus creating a bottleneck for attribute recognition. Currently, the viewpoint and pose representations on datasets are relatively fixed, while real-world applications involve complex changes in viewpoint and pose, leading to poor generalization performance of the model.

[0005] For video surveillance in real-world scenarios, existing algorithms cannot meet the needs of complex perspectives and pedestrian states, mainly due to the following problems:

[0006] 1) Existing methods cannot adapt to the differences between complex viewpoint poses in real-world applications and single viewpoint poses in datasets. Current methods, while recognizing attributes, cannot actually recognize human anatomy. Therefore, attribute localization is often learned through the statistical characteristics of spatial distribution. Consequently, due to the influence of fixed viewpoint poses in the dataset, different attributes are easily bound to fixed positions. However, in complex viewpoint poses, the spatial position of attributes changes with the pose, leading to deviations in the model's attribute localization.

[0007] 2) Existing methods cannot adapt to changes in the relationship between attribute space characteristics and human structure as perspective and posture change with complex angles and poses when utilizing human structure information. Current methods, when recognizing attributes and considering the use of human structure information, typically bind the attribute's feature space to a relevant part of a specific human body region. However, with changes in perspective and posture, this relative position also changes accordingly, and existing methods still experience positioning errors in practical applications. Summary of the Invention

[0008] To address the challenges of complex perspective states in real-world applications and the differences in pose from a single perspective in a dataset, this invention proposes a pedestrian attribute recognition algorithm adapted to complex perspectives and pedestrian states. The algorithm comprises: first, extracting keypoint information from pedestrian slices using a Mask-RCNN model; then, integrating the keypoints into positional encoding information and incorporating it into the image features, enabling the model to learn information about human body structure; next, forming region coordinate information based on the differences between regions using the keypoints; truncating the probability distribution function calculation during attention scoring of the horizontal and vertical coordinates to enhance the attention of image attribute features in a weakly supervised manner; and finally, fusing the deep and shallow features guided by the keypoint information and using a classifier to form the attribute judgment result.

[0009] To address the problem of models failing to recognize human body structures, this invention proposes a keypoint-based spatial encoding method. After generating corresponding spatial codes using keypoints, consecutive convolutional and activation layers map the spatial information and the model's image features into the same feature space. The corresponding features are then concatenated and integrated with the texture and spatial information through a 1x1 convolution to form attribute features.

[0010] To address the problem of inadequate adaptation to the dynamic spatial relationship between attributes and human structure under complex conditions, this invention proposes a weakly supervised attribute feature localization method based on keypoint constraints. After passing through a linear layer, attribute features generate importance scores across different dimensions. These scores are treated as probability density vectors and summed using a keypoint information truncation function to obtain a constrained probability distribution function. Finally, the product forms an attention distribution function accurate to the attribute level.

[0011] According to one aspect of the present invention, a pedestrian attribute recognition method adapted to complex perspectives and pedestrian states is provided, characterized by comprising the following steps:

[0012] A) Key point information is obtained using the Mask-RCNN method, and image features are obtained using ResNet-50;

[0013] B) Process the key point information into key point-based spatial location encoding and alignment, then embed it into image features;

[0014] C) Integrate key point information into human body region coordinates and enhance attribute features through weak supervision methods;

[0015] D) The deep and shallow features of the image guided by key points are fused together and the result is obtained by classifying the attribute recognition result;

[0016] in:

[0017] The spatial location encoding method described in step B) includes:

[0018] B1) Incorporating spatial information into feature extraction is crucial for accurately locating attribute features, enabling the model to distinguish spatial differences between similar attributes and enhancing feature localization. We analyze and adapt to variations in keypoint distribution by using coordinates relative to keypoint encoding and a normalization method. This adjustment ensures that spatial information is effectively integrated and balanced in terms of keypoints.

[0019] KPC x =PC x -k x ,

[0020]

[0021] Where, k x The horizontal (x-axis) coordinates (PC) representing key points are part of the coordinate information layer. x This represents the x-axis coordinate layer. KPC is the coordinate information relative to the keypoint encoding. x This represents the x-axis coordinate information. The abs function takes the absolute value, and the max function takes the maximum value of the vector. We can obtain KPC using the same steps. y .

[0022] B2) In addition to the x and y coordinates, we will also use the distance relative to the keypoint as the KPC. r It has been integrated.

[0023]

[0024] KPC r This represents the radius value in polar coordinates. The Max function represents taking the maximum value of the vector.

[0025] B3) We introduce an alignment module after the spatial encoding step to align the spatial encoded features with the attribute features.

[0026] KPC = concat(KPC) x KPC y KPC r ),

[0027] KConv(I)=Conv(concat(I,Align(KPC))),

[0028] f i =KConv(f i )

[0029] Here, Align represents the alignment module, which consists of several convolutional layers. Kconv represents the spatial encoding module. I represents any input feature. The Concat function represents the concatenation operation. Conv represents a convolutional layer. f i The characteristic representing attribute i.

[0030] The keypoint-guided weak supervision method described in step C) includes:

[0031] C1) To integrate keypoints into this framework, we reimagined the feature vector as a feature vector of a probability density distribution, reflecting the importance of attribute features on the x and y axes. Under this design, each attribute requires the computation of two spatial vectors. Therefore, the input feature f i First, through a linear layer, we obtain the probability density distribution F of each attribute on the x and y axes. ix F iy See equation (3.1).

[0032] F ix F iy =linear(f i )

[0033] Where i represents an attribute, f i F represents the characteristic of attribute i. ix F represents the distribution score of attribute i on the x-axis. iy Similarly, Linear represents a linear layer function.

[0034] C2) To truncate the distribution, we segment the softmax process according to the following formula and compute the probability distribution on different coordinate axes. Following the principle of the Expectation-Maximization (EM) algorithm, we avoid computing the probability of invalid points outside the defined region. This leads us to modify the method revealed in the original paper, directly truncating the probability density distribution, rather than just the distribution at the boundary locations.

[0035] p xi =Mask part (exp(F ix )),

[0036]

[0037] Here, `part` represents the body part to which the attribute belongs. The `Mask` function changes the probability density values ​​outside the body part region to 0. These can be considered outliers, and their probability density can be ignored in the calculation. The `Sum` function sums the vectors, P... x This represents the probability density vector obtained after being constrained by the key point region.

[0038] C3) Since we have two boundaries, we need to process the vector twice to obtain p. x1 and p x2 The difference between the two lies in p xi1 Only the left boundary p is captured xi2 Only the right boundary is captured. Based on the probability density function, we can calculate the probability distribution function.

[0039] P xi1 =Cumsum(p xi1 ),

[0040] P xi2 =Cumsum(P xi2 ),

[0041] P xi1 =P xi1 ·(1-P xi2 )

[0042] Here, Cumsum represents the cumulative summation function from left to right. By considering p... x1 By summing the results, we obtain a probability distribution P. x1 The same operation can yield P. x2 P x1 and P x2 These represent two probability density vectors after undergoing the same operation. By multiplying the probability distributions constrained by two boundaries, we can obtain a fraction P constrained to a specific region on the coordinate axis. xi The same steps can be used to obtain P. yi .

[0043]

[0044] P represents the cross product operation between matrices. i Attention graph representing attribute i.

[0045] C4) By multiplying the values ​​of the two coordinate axes, we can obtain a spatial mask matrix P constrained by key points.

[0046] f i =α*f i +(1-α)*f i *Pi

[0047] Among them, f i Let α represent the feature of attribute i. Let α represent the weight. We achieve a refined feature map by softly adding the feature enhanced with spatial masking to the original feature. Therefore, this approach successfully facilitates weakly supervised localization modeling of attribute features guided by keypoints.

[0048] The fusion method described in step D) includes the following steps:

[0049] D1) We strategically use key points to guide shallow and deep features.

[0050] func(x) = KGM(PCM(x)),

[0051] Func represents the main function in this patent. KGM refers to the key point guidance and localization module, while PCM represents the spatial coding module.

[0052] D2) Then, the two features are concatenated and classified through a linear layer to obtain the attributes.

[0053] Attr=FC(concat(func(feat4), func(feat3)))

[0054] Here, feat4 represents the features of the fourth layer, and feat3 represents the features of the third layer. Attr represents the final attribute obtained. Attached Figure Description

[0055] Figure 1 It is a complex perspective and posture diagram of the real world.

[0056] Figure 2 This is a performance comparison chart between the real world and the dataset.

[0057] Figure 3 This is a flowchart of a pedestrian attribute recognition method adapted to complex perspectives and pedestrian states according to an embodiment of the present invention. Detailed Implementation

[0058] According to one aspect of the present invention, a pedestrian attribute recognition method based on complex viewpoint pose adaptation is provided, such as... Figure 3 It includes the following steps:

[0059] A) Key point information is obtained using the Mask-RCNN method, and image features are obtained using ResNet-50;

[0060] B) Process the key point information into key point-based spatial location encoding and alignment, then embed it into image features;

[0061] C) Integrate key point information into human body region coordinates and enhance attribute features through weak supervision methods;

[0062] D) The deep and shallow features of the image guided by key points are fused together and the result is obtained by classifying the attribute recognition result;

[0063] A) Key point and image feature extraction

[0064] The primary objective of this invention is to adapt to complex perspectives and poses in the real world. For example... Figure 1 Real-world pedestrian snapshots exhibit a variety of complex perspectives. For example... Figure 2 Under the influence of complex viewpoints and poses, the model's performance in the real world is significantly lower than its performance on datasets. The main reason is that without human recognition, the model is affected by the single viewpoint pose of the dataset, causing it to locate attribute features in relatively fixed positions, resulting in attribute localization bias. Therefore, we need pose-robust information to guide the model in attribute localization. Our method uses keypoint information to guide the model.

[0065] After training on a large amount of data, the Mask-RCNN model can extract accurate human structural information while recognizing human bodies. We first expand the pedestrian slices to twice their length and width to adapt to the data characteristics of Mask-RCNN. Then, after extracting the key points corresponding to the pedestrian attributes through the model, we normalize the key point data to the [0, 1] size range of the normal image.

[0066] B) Spatial Location Coding

[0067] Incorporating spatial information into feature extraction is crucial for accurate attribute feature localization. This enables the model to discern spatial differences between similar attributes, enhancing feature localization. The CoordConv technique uses additional coordinate channels in convolutional layers to directly access input coordinates. We adapt this concept by using coordinates encoded relative to keypoints, along with a normalization condition to account for variations in keypoint distribution. This adjustment ensures that spatial information is effectively integrated and balanced in terms of keypoints.

[0068] KPC x =PC x -k x ,

[0069]

[0070] Where, k x Represents the horizontal coordinates of key points. PC is the coordinate information layer. xThis represents the x-axis coordinate layer. `Max` is the maximum value function, and `abs` is the absolute value function. `KPC` represents the coordinate information encoded relative to the keypoint. In addition to the x and y coordinates, we also use the distance relative to the keypoint as the KPC. r It has been integrated.

[0071]

[0072] A key aspect is that integrating spatial coordinate information into the data layer, closer to the data layer itself, has proven to yield better results in spatial encoding. We interpret this as spatial information being more closely aligned with the original image data in this way. However, we encounter a significant challenge: directly concatenating spatial encoding at the data layer can interfere with the integrity of the pre-trained model. If we choose to use an additive approach to incorporate spatial encoding into the pre-trained model to preserve the pre-trained parameters, there is a risk of feature imbalance, which may weaken the effectiveness of the method. Furthermore, adding spatial information via concatenation after the classification layer of the pre-trained model leads to significantly worse results. This is likely because the features at this stage are too abstract to be effectively integrated with the spatial encoding. To address this issue, we introduce an alignment module after the spatial encoding step. This module enables the mapping and alignment of spatially encoded features with the attribute feature space.

[0073] KPC = concat(KPC) x KPC y KPC r ),

[0074] KConv(I)=Conv(concat(I,Align(KPC))),

[0075] f i =KConv(f i )

[0076] Here, Align represents the alignment module, which consists of several convolutional layers. Kconv represents the spatial encoding module. I represents arbitrary input features. The Concat function represents the concatenation operation. Conv represents a convolutional layer. f i The characteristic representing attribute i.

[0077] C) Weak supervision method guided by key points

[0078] We decompose the attention mechanism into the boundary distribution probabilities of the model along the vertical and horizontal axes of the image. The cross product of two scoring vectors forms the attention for a specific region. If we can explicitly define this constraint as a specific attribute region within the keypoint area, we can achieve the dual goals of introducing weak supervision of keypoints and refining the constraint to the attribute level. To integrate keypoints into this framework, we reimagine the feature vector as a feature vector of a probability density distribution, reflecting the importance of attribute features along the x and y axes. Due to this design, each attribute only needs to compute two spatial vectors. Therefore, the input feature f i First, through a linear layer, we obtain the probability density distribution F of each attribute on the x and y axes. ix F iy .

[0079] F ix F iy =linear(f i )

[0080] Where i represents an attribute, f represents an image feature, and f i F represents the characteristic of attribute i. ix The distribution score represents attribute i on the x-axis. `Linear` denotes the linear layer function. To truncate the distribution, we segment the softmax process according to the following formula and compute the probability distribution on different coordinate axes. Following the Expectation-Maximization (EM) algorithm, we avoid calculating the probability of invalid points outside the defined region. This leads us to modify the general approach, directly truncating the probability density distribution, rather than just the distribution at the boundary locations.

[0081] P xi =Mask part (exp(F ix )),

[0082]

[0083] Here, `part` represents the body part to which the attribute belongs. The `Mask` function sets the probability density values ​​outside the body part region to 0, and `sum` is the summation function. These can be considered outliers, and their probability density can be directly ignored in the calculation. P xi This represents the probability density vector obtained after being constrained by the keypoint region. Since we have two boundaries, we need to process the vector twice using the steps above to obtain p. xi1 and p xi2 The difference between the two lies in p xi1 Only the left boundary p is captured xi2Only the right boundary is captured. Next, the probability density vector is calculated into a probability distribution:

[0084] P xi1 =Cumsum(p xi1 ),

[0085] P xi2 =Cumsum(p xi2 ),

[0086] P xi1 =P xi1 ·(1-P xi2 )

[0087] Here, Cumsum represents the cumulative summation function from left to right. By considering p... xi1 By summing the results, we obtain a probability distribution P. xi1 The same method can be used to obtain Pxi2. By multiplying the probability distributions of the two boundary constraints, we can obtain a fraction P that is constrained in a specific region on the coordinate axis. xi The same operation can yield P. yi .

[0088]

[0089] This represents the cross product operation between matrices. By multiplying the values ​​of two coordinate axes, we can obtain a spatial mask matrix P constrained by keypoints. i .

[0090] f i =α*f i +(1-α)*f i *P i

[0091] Among them, f i Let i represent the features of attribute i. We achieve a refined feature map by softly adding features enhanced with spatial masks to the original features. Therefore, this approach successfully facilitates weakly supervised localization modeling of attribute features guided by keypoints.

[0092] D) Feature fusion

[0093] The effectiveness of the spatial guidance module applied to shallower layers prompted us to further explore the possibility of guiding features at shallower levels. Furthermore, our experimental results using AlexNet as the backbone are closely competitive with those using ResNet. This indicates that shallow features play a significant role in identifying certain attributes. We strategically use keypoints to guide both shallow and deep features.

[0094] func(x) = KGM(PCM(x)),

[0095] Here, KGM refers to the keypoint guided localization module, while PCM represents the spatial coding module. Then, the last two features of the backbone network are concatenated and classified through a linear layer to obtain the attributes:

[0096] Attr=FC(concat(func(feat4), func(feat3)))

[0097] In this network, feat4 and feat3 are the feature maps of the last two layers of the backbone network, concat is the concatenation function that concatenates the two features, FC is a linear layer, and Attr is the final attribute obtained.

Claims

1. A pedestrian attribute recognition method adapted to complex perspectives and pedestrian states, characterized in that... Includes the following steps: A) Key point information is obtained using the Mask-RCNN method, and image features are obtained using ResNet-50; B) Process the key point information into attribute features after spatial location encoding and alignment based on the key points; C) Integrate key point information into human body region coordinates and enhance attribute features through weak supervision methods; D) The deep and shallow features of the image guided by key points are fused together and the result is obtained by classifying the attribute recognition result; in: Step B) includes: B1) To identify spatial differences between similar attributes and / or enhance feature localization, coordinates relative to the keypoint encoding and a normalization condition are used to characterize changes in keypoint distribution: KPC x =PC x -k x , Where, k x The horizontal axis, or x-axis, represents the key points. PC x KPC represents the x-axis coordinate layer. x This represents the x-axis coordinate information; abs indicates the absolute value; max indicates the maximum value of the vector. Similarly, KPC obtains coordinate information representing the y-axis. y , B2) Based on the x-axis and y-axis information, the distance KPC relative to the keypoint can be calculated. r : Where max represents taking the maximum value. B3) Ensure that the spatial coding features are consistent with the attribute feature space: KPC=concat(KPC x ,KPC y ,KPC r ), KConv(I)=Conv(concat(I,Align(KPC))), f i =KConv(f i ) in: Align refers to the alignment module, which consists of several convolutional layers. Kconv stands for spatial coding module. I represents any input. The Concat function represents a concatenation operation. Conv represents a convolutional layer. f i The characteristic representing attribute i Step C) includes: C1) To augment keypoints into features in a weakly supervised manner, the feature vector is reconstructed as a feature vector of a probability density distribution, reflecting the importance of attribute features on the x and y axes. Each attribute only requires the computation of two spatial vectors, including: First, make the attribute feature f i After passing through a linear layer, we obtain the probability density distribution F of each attribute on the x and y axes. ix F iy : F ix ,F iy =linear(f i ), Among them, F ix F represents the distribution score of attribute i on the x-axis. iy The score represents the distribution of attribute i on the y-axis; Linear represents the linear layer function. C2) To truncate the distribution, the softmax process is segmented, and the probability density function is calculated on different coordinate axes, following the Expectation-Maximization (EM) algorithm principle. This avoids calculating the probability of invalid points outside the defined region, thus directly truncating the probability density distribution. The softmax process is segmented according to the following formula: p xi =Mask part (exp(F xi )), in: `part` represents the body part to which the attribute belongs. The `Mask` function changes the probability density values ​​outside the body part region to 0. These probability density values ​​that are changed to 0 are considered outliers and their probability density is directly ignored in the calculation. p xi This represents the probability density vector obtained after being constrained by the keypoint region. C3) Since there are two boundaries, the Mask function above only extracts the right boundary to obtain p. xi1 p is obtained by only truncating the right boundary. xi2 The probability density function can be used to further calculate the probability distribution function. P xi1 =How much(p xi1 ), P xi2 =How much(p xi2 ), P xi1 =P xi1 ·(1-P xi2 ) Where Cumsum represents the cumulative summation function from left to right, calculated by applying p... xi1 By performing cumulative summation, we obtain a probability distribution P. xi1 By analyzing p xi2 By performing cumulative summation, we obtain Pxi2, P xi1 and P xi2 These represent two probability density vectors after undergoing the same operation. By multiplying the probability distributions constrained by two boundaries, we obtain a fraction P that is restricted to a specific region on the coordinate axis. i : P represents the cross product operation between matrices. i The attention graph representing attribute i. C4) By multiplying the values ​​of the two coordinate axes, a spatial mask Pi constrained by keypoints is obtained: f i =α*f i +(1-a)*f i *P i Among them, f i Let α represent the feature of attribute i, and let α represent the weight. Thus, by adding weighted features enhanced with spatial masks to the original features, a refined feature map was achieved, successfully facilitating weakly supervised localization modeling of attribute features guided by keypoints. The fusion described in step D) includes: D1) To bring spatial information closer to the data source, key points are used to guide shallow and deep features: func(x) = KGM(PCM(x)), KGM refers to the keypoint guided localization module, PCM refers to the spatial coding module, and x represents any input feature. D2) The feature maps feat4 and feat3 from the last two layers of the backbone network are concatenated and classified through a linear layer to obtain the final attribute Attr: Attr=FC(concat(func(feat4), func(feat3))) Here, feat3 and feat4 represent the features of the third and fourth layers, concat is the concatenation function for two features, and FC is a linear layer.

Images