A pedestrian attribute recognition method based on attribute semantic similarity matching

Abstract

The application relates to a pedestrian attribute recognition method based on attribute semantic similarity matching, and relates to the technical field of computer vision. First, an adaptive semantic query module is designed, adaptive queries combined with visual features are used to learn attribute-specific spatial distribution, and semantic information of each attribute is captured. In addition, pedestrian attribute recognition is reconstructed as a semantic matching task, attribute text features are used as semantic anchors, and the distance between the semantic information of the query and the anchors is used to predict the pedestrian attribute. Finally, a dynamic negative semantic learning strategy is proposed, the spatial prior information of the attribute is combined with the learnable parameters to generate the negative semantic information of the attribute, the attention area of the semantic query module to the attribute is constrained, and more accurate pedestrian attribute recognition is realized.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method for pedestrian attribute recognition based on attribute semantic similarity matching. Background Technology

[0002] The purpose of pedestrian attribute recognition is to predict a series of attributes, such as age, gender, and clothing, from a given pedestrian image. Due to the high-level semantics and informational nature of pedestrian attributes, they can provide valuable auxiliary information for other tasks, such as pedestrian re-identification and scene understanding, and have wide applications in fields such as security monitoring and intelligent transportation. Pedestrian attribute recognition is a subset of multi-label classification tasks, but unlike traditional multi-label classification tasks, pedestrian attribute recognition involves learning prior knowledge and semantic structural information from pedestrian images, such as spatial distribution and relationships between attributes.

[0003] Due to the lack of location labels for attributes, existing attribute recognition methods often struggle to learn comprehensive feature representations for fine-grained attribute identification. This results in low accuracy, failing to meet practical application requirements. In recent years, visual language pre-trained models (such as CLIP) have seen rapid development. By pre-training on large-scale image-text datasets, they can capture rich semantic information from different modalities, achieving impressive performance across a range of tasks. Because pedestrian attributes are scarce, the frequency of each attribute in the dataset is uneven. Utilizing pre-trained visual language models to extract semantic information from attributes can alleviate this problem to some extent. However, CLIP's global features tend to emphasize the subject category, which is not conducive to attribute recognition in multi-label classification. Summary of the Invention

[0004] The purpose of this invention is to provide a pedestrian attribute recognition method based on attribute semantic similarity matching, which aims to solve the problems of low accuracy, difficulty in learning comprehensive feature representation, and inability to effectively deal with sample imbalance in existing pedestrian attribute recognition methods. Through innovative module design and algorithm strategy, the accuracy and reliability of pedestrian attribute recognition are improved.

[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution.

[0006] A pedestrian attribute recognition method based on attribute semantic similarity matching includes the following steps:

[0007] 1) Design an adaptive attribute query module to extract attribute-specific semantic information from pedestrian images to capture the semantic features corresponding to each attribute; design a series of learnable parameters as corresponding attribute queries, input pedestrian images into an image encoder to obtain visual vector representations, and interact attribute queries and visual vectors through a cross-attention mechanism to learn the spatial distribution of attributes and capture decoupled attribute semantic information.

[0008] 2) A semantic similarity matching module is proposed to reconstruct pedestrian attribute recognition into a semantic matching task. The attribute text features are used as semantic anchors. The similarity matching between the queried attribute semantic information and the corresponding text features is performed to identify the corresponding pedestrian attributes.

[0009] 3) A dynamic negative semantic learning strategy is proposed. By combining the spatial prior information of the attribute with the learnable parameters, the negative semantic information of the attribute is generated to constrain the semantic query module’s attention area of ​​the attribute. This ensures that even when the corresponding attribute does not exist in the image, the dynamic semantic query module can still pay attention to the local area corresponding to the pedestrian attribute, thus achieving more accurate pedestrian attribute recognition.

[0010] In step 1), the design of an adaptive attribute query module specifically includes the following steps:

[0011] (1) Pedestrian image feature extraction part

[0012] For a given pedestrian image, a pre-trained Visual Transformer is used to extract its corresponding visual vectors; first, the image is divided into a series of fixed-size, non-overlapping blocks, totaling [number missing]. The sequence of blocks is divided into blocks, where P represents the block size; these block sequences are then converted into 1-dimensional vectors through a trainable linear projection and fed into CLIP's visual encoder to obtain the visual vector representation F.

[0013] (2) Attribute semantic query part

[0014] Since CLIP's global features may ignore some fine-grained attribute information, adaptive attribute queries are used to extract the semantic information corresponding to each attribute from the image's feature map. First, a set of learnable attribute queries is constructed. Where N is the number of attributes, the attribute query first learns the association information between attributes through self-attention, and then the attribute query and image features F obtain the semantic information corresponding to the attributes through a cross-attention module;

[0015] (3) Semantic consistency loss

[0016] After obtaining the semantic information corresponding to each attribute, a semantic consistency loss is proposed to decouple attribute semantics and ensure semantic consistency of the same attribute. First, based on whether the semantic features belong to the same attribute, corresponding group labels are generated within the training minimum batch. Then, by calculating the similarity between different semantics, we can obtain the probability that they belong to the same attribute. The final semantic consistency loss is calculated as follows:

[0017]

[0018] In step 2), the semantic similarity matching module aims to use the similarity between attribute text features and attribute semantics to perform pedestrian attribute recognition. By transforming attribute recognition into a semantic matching task, the semantic consistency of the visual language model can be fully utilized, reducing minority sample recognition errors caused by sample imbalance. Specifically, the attribute name is first expanded into a text description, and then the corresponding text features are generated through CLIP's text encoder. This process is defined as follows:

[0019]

[0020] Then, the cosine similarity between the semantic features of the attributes and the text features is calculated to obtain the predicted probability of each attribute. This process is defined as follows:

[0021]

[0022] in, Temperature hyperparameters are used to adjust the model's sensitivity to similar changes; the proposed method is trained using weighted cross-entropy loss, and its mathematical expression is as follows:

[0023]

[0024] in, This represents the unbalanced weight of the j-th attribute. This represents the proportion of the j-th attribute in the training set.

[0025] In step 3), the dynamic negative semantic learning strategy regularizes the query semantics for non-existent attributes, constraining attribute queries to relevant regions and reducing interference from irrelevant regions during attribute queries due to the absence of pedestrian attributes in the image. For each attribute, learnable dynamic negative semantics are obtained by combining its spatial location prior information with learnable embeddings. This process is described as follows:

[0026]

[0027] in, and Represents prefix and suffix embedding. Let i be the learnable embedding corresponding to the i-th attribute; then, the corresponding negative semantic features are obtained through the CLIP text encoder, as detailed below:

[0028]

[0029] Similar to the semantic similarity matching module, the predicted probability for each non-existent attribute can be obtained by calculating the similarity between attribute semantics and dynamic negative semantics. Then, the weighted cross-entropy loss is calculated. This process is represented as follows:

[0030]

[0031] The final overall loss is expressed as:

[0032]

[0033] in, Hyperparameters for controlling negative semantic loss.

[0034] Compared with the prior art, the outstanding technical effects and advantages of the present invention are as follows:

[0035] 1. Through the adaptive attribute query module, the spatial distribution of attributes is learned from the feature map output by CLIP and the semantic information corresponding to each attribute is extracted, thereby achieving the decoupling of attribute semantics. It can learn the spatial distribution specific to the attribute, capture fine-grained attribute semantic information, solve the problem that existing methods are difficult to learn comprehensive feature representations, and thus improve the accuracy of attribute recognition.

[0036] 2. This invention transforms attribute prediction into a semantic similarity matching method, which can maintain a consistent learning objective with the pre-trained model and improve the performance during fine-tuning of downstream tasks. The semantic similarity matching module uses a weighted cross-entropy loss training model, which adjusts the loss weights according to the frequency of attribute occurrence in the training set, effectively improving the model's ability to identify a small number of samples and overcoming the challenges caused by sample imbalance.

[0037] 3. To address the imbalance between positive and negative labels in samples, dynamic negative semantic learning is introduced. Learnable semantic anchors are used to constrain attribute queries to attribute-related regions, improving the accuracy of semantic queries. The dynamic negative semantic learning strategy generates dynamic negative semantics to constrain the semantic query module's focus on the attribute region, ensuring that attribute queries can still be performed accurately even when the corresponding attribute does not exist in the image, further improving the accuracy and reliability of pedestrian attribute recognition. Attached Figure Description

[0038] Figure 1 This is a flowchart of the adaptive attribute query and semantic similarity matching process of the present invention.

[0039] Figure 2 This is a flowchart of the dynamic negative semantic learning process of the present invention. Detailed Implementation

[0040] The following embodiments will further illustrate the present invention with reference to the accompanying drawings.

[0041] The overall process of implementing this invention includes the following steps:

[0042] This invention mainly comprises three parts: an adaptive attribute query module, semantic similarity matching, and dynamic negative semantic learning. First, adaptive attribute query extracts decoupled semantic information from the image. Then, attribute similarity matching is used to calculate the similarity between the attribute semantic information and the corresponding text features to predict pedestrian attributes. Finally, this invention proposes dynamic negative semantic learning to ensure that attribute queries are concentrated in attribute-related regions, thereby achieving better attribute recognition.

[0043] 1. See Figure 1 The adaptive attribute query and semantic similarity matching part specifically includes the following steps:

[0044] 1) Pedestrian image feature extraction section

[0045] For a given pedestrian image, this invention uses a VisualTransformer pre-trained based on contrastive image language to extract its corresponding visual vectors; first, the given pedestrian image is resized to a uniform size, and then divided into fixed-size, non-overlapping blocks, totaling [number missing]. The image is divided into blocks, where P represents the block size. These block sequences are then converted into 1-dimensional vectors through a trainable linear projection layer whose parameters are continuously optimized during training. Next, learnable positional embeddings are added to capture spatial location information between image blocks. The converted 1-dimensional vectors are then fed into CLIP's visual encoder, where the visual vector representation F of the image is obtained by fine-tuning a pre-trained Transformer layer.

[0046] 2) Attribute semantic query part

[0047] Since CLIP's global features may overlook some fine-grained attribute information, this invention extracts the semantic information corresponding to each attribute from the image's feature map through adaptive attribute querying. The adaptive attribute querying consists of a self-attention layer, a cross-attention layer, and a set of learnable parameters. First, a set of learnable attribute queries is constructed. Where N represents the number of attributes; the value of N depends on the specific number of pedestrian attributes. In common pedestrian attribute recognition tasks, attributes such as gender, age range, and clothing type may be included, and the value of N is usually between 20 and 60. Attribute query first interacts through a self-attention layer to learn the association information between attributes, such as long hair and female, which will co-occur with a high probability. Then, attribute query and image features interact through a cross-attention module to obtain the semantic information corresponding to the attribute. In the cross-attention module, the attribute query is the query in the cross-attention layer, and the image features are the key and value in the cross-attention. The semantic information corresponding to the attribute is obtained through matrix operations and attention weight calculation.

[0048] 3) Semantic consistency loss

[0049] After obtaining the semantic information corresponding to each attribute, a semantic consistency loss is proposed to decouple attribute semantics and ensure semantic consistency for the same attribute. First, based on the labels during training, it is determined whether semantic features belong to the same attribute. Then, corresponding group labels are generated for the attribute semantics within the minimum batch. Then, by calculating the cosine similarity between different semantics, we can obtain the probability that they belong to the same attribute. The final semantic consistency loss is calculated as follows:

[0050]

[0051] During training, through continuous iterative optimization, the semantic consistency loss is gradually reduced, thereby achieving decoupling of attribute semantics and consistency of semantics for the same attribute. This method makes the semantics corresponding to the same queried attribute more similar, and the semantic differences between different attributes more obvious, thus improving the ability to understand the semantic information contained in pedestrian images.

[0052] 4) Semantic similarity matching module

[0053] The semantic similarity matching module aims to leverage the powerful feature extraction capabilities of a pre-trained visual-language large model and the semantic consistency between the visual and language modalities to achieve more accurate pedestrian attribute recognition. First, the attribute name is expanded into a text description, for example, "long hair" is expanded to "pedestrians with long hair." Then, the CLIP text encoder generates corresponding text features. This involves first encoding the text to obtain corresponding vectors, and then inputting the text vectors into a pre-trained Transformer layer to obtain the final text features. This process is defined as follows:

[0054]

[0055] Then, the cosine similarity between the semantic features of the attributes and the text features is calculated to obtain the predicted probability of each attribute. This process is defined as follows:

[0056]

[0057] in Temperature hyperparameters are used to adjust the model's sensitivity to similar changes.

[0058] 2. See Figure 2 The dynamic negative semantic learning part of this invention, for each attribute, obtains learnable dynamic negative semantics by combining its spatial location prior information with a learnable embedding. The prior information is a description of the body part corresponding to the attribute, such as "head," "upper body," and "feet," which is encoded and combined with the corresponding learnable embedding. The learnable embedding is continuously optimized during training to generate more accurate negative semantic information. This process is described as follows:

[0059]

[0060] in and Represents prefix and suffix embedding. Let i be the learnable embedding corresponding to the i-th attribute. Then, the corresponding negative semantic features are obtained through the CLIP text encoder, as detailed below:

[0061]

[0062] Similar to the semantic similarity matching module, the predicted probability for each non-existent attribute can be obtained by calculating the similarity between attribute semantics and dynamic negative semantics. Then, the weighted cross-entropy loss is calculated. This process is represented as follows:

[0063]

[0064] The final overall loss is expressed as:

[0065]

[0066] in, Hyperparameters for controlling negative semantic loss.

[0067] During training, the similarity between attribute semantics and dynamic negative semantics is calculated to obtain the predicted probability for each non-existent attribute, and the model parameters are adjusted according to the weighted cross-entropy loss. The model is comprehensively optimized through the overall loss function, and its optimal value is determined through multiple experiments.

[0068] 3. Referring to Table 1, a series of experiments were conducted to verify the superiority of the method of this invention. The experimental datasets used were several publicly available pedestrian attribute recognition datasets, including PETA and PA100K. Comparison methods included traditional pedestrian attribute recognition methods based on convolutional neural networks, as well as some advanced deep learning methods in recent years, such as methods based on convolutional neural networks and methods based on attention mechanisms.

[0069] Evaluation metrics: Accuracy, recall, and F1 score are used as the main evaluation metrics. Accuracy measures the proportion of correctly predicted samples out of the total number of predicted samples; recall measures the proportion of correctly predicted positive samples out of the actual number of positive samples; and the F1 score is the harmonic mean of accuracy and recall.

[0070] Table 1

[0071]

[0072] Experimental results show that on the PETA dataset, the proposed method achieves an accuracy of 83.65%, which is 5.61 percentage points higher than traditional methods based on handcrafted features, 3.27 percentage points higher than methods based on convolutional neural networks, and 2.55 percentage points higher than methods based on attention mechanisms. The recall rate reaches 90.95%, and the F1 score reaches 89.64%, both significantly better than the comparative methods. Similar excellent results are also achieved on the PA100K dataset, with an accuracy of 82.48%, a recall rate of 90.36%, and an F1 score of 89.21%. Experiments demonstrate that the pedestrian attribute recognition method based on attribute semantic similarity matching proposed in this invention has significant advantages in accuracy and reliability, effectively solving the problems of existing methods and providing a more effective solution for the field of pedestrian attribute recognition.

[0073] This invention mainly comprises three parts: an adaptive attribute query module, a semantic similarity matching module, and dynamic negative semantic learning. First, the adaptive attribute query module extracts attribute-specific semantic information from pedestrian images to capture the semantic features corresponding to each attribute. This invention designs a series of learnable parameters as corresponding attribute queries. Then, the pedestrian image is input into an image encoder to obtain a visual vector representation. The attribute query and the visual vector interact through a cross-attention mechanism to learn the attribute-specific spatial distribution and capture decoupled attribute semantic information. Next, this invention proposes a semantic similarity matching module, which performs similarity matching between the queried attribute semantic information and the corresponding text features to identify the corresponding pedestrian attributes. Finally, this invention proposes dynamic negative semantic learning to ensure that even when the corresponding attribute does not exist in the image, the dynamic semantic query module can still focus on the local region corresponding to the pedestrian attribute, thereby improving the accuracy of attribute queries.

[0074] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A pedestrian attribute recognition method based on attribute semantic similarity matching, characterized in that... Includes the following steps: 1) Design an adaptive attribute query module to extract attribute-specific semantic information from pedestrian images in order to capture the semantic features corresponding to each attribute; A series of learnable parameters are designed as corresponding attribute queries. Pedestrian images are input into an image encoder to obtain visual vector representations. The attribute queries and visual vectors are interacted through a cross-attention mechanism to learn the spatial distribution of attributes and capture decoupled attribute semantic information. 2) A semantic similarity matching module is proposed to reconstruct pedestrian attribute recognition into a semantic matching task. The attribute text features are used as semantic anchors. The similarity matching between the queried attribute semantic information and the corresponding text features is performed to identify the corresponding pedestrian attributes. 3) A dynamic negative semantic learning strategy is proposed. By combining the spatial prior information of the attribute with the learnable parameters, the negative semantic information of the attribute is generated to constrain the semantic query module’s attention area of ​​the attribute. This ensures that when the corresponding attribute does not exist in the image, the dynamic semantic query module can still pay attention to the local area corresponding to the pedestrian attribute, thus achieving more accurate pedestrian attribute recognition. The dynamic negative semantic learning strategy regularizes the query semantics of non-existent attributes, constraining attribute queries to relevant regions and reducing interference from irrelevant regions during attribute queries due to the absence of pedestrian attributes in the image. For each attribute, learnable dynamic negative semantics are obtained by combining its spatial location prior information with learnable embeddings. This process is described as follows: in, and Represents prefix and suffix embedding. Let i be the learnable embedding corresponding to the i-th attribute; then, the corresponding negative semantic features are obtained through the CLIP text encoder, as detailed below: Similar to the semantic similarity matching module, the predicted probability for each non-existent attribute is obtained by calculating the similarity between attribute semantics and dynamic negative semantics, and then the weighted cross-entropy loss is calculated. This process is represented as follows: The final overall loss is expressed as: in, Hyperparameters for controlling negative semantic loss.

2. The pedestrian attribute recognition method based on attribute semantic similarity matching as described in claim 1, characterized in that... In step 1), the design of an adaptive attribute query module specifically includes the following steps: (1) Pedestrian image feature extraction part For a given pedestrian image, a pre-trained Visual Transformer is used to extract its corresponding visual vectors; first, the image is divided into a series of fixed-size, non-overlapping blocks, totaling [number missing]. The sequence of blocks is divided into blocks, where P represents the block size; these block sequences are then converted into 1-dimensional vectors through a trainable linear projection and fed into CLIP's visual encoder to obtain the visual vector representation F. (2) Attribute semantic query part Since CLIP's global features ignore some fine-grained attribute information, adaptive attribute queries are used to extract the semantic information corresponding to each attribute from the image's feature map. First, a set of learnable attribute queries is constructed. Where N is the number of attributes, the attribute query first learns the association information between attributes through self-attention, and then the attribute query and image features F obtain the semantic information corresponding to the attributes through a cross-attention module; (3) Semantic consistency loss After obtaining the semantic information corresponding to each attribute, a semantic consistency loss is proposed to decouple attribute semantics and ensure semantic consistency of the same attribute. First, based on whether the semantic features belong to the same attribute, corresponding group labels are generated within the training minimum batch. Then, by calculating the similarity between different semantics, we can obtain the probability that they belong to the same attribute. The final semantic consistency loss is calculated as follows: 。 3. The pedestrian attribute recognition method based on attribute semantic similarity matching as described in claim 1, characterized in that... In step 2), the semantic similarity matching module aims to use the similarity between attribute text features and attribute semantics to perform pedestrian attribute recognition. By transforming attribute recognition into a semantic matching task, the semantic consistency of the visual language model can be fully utilized, reducing minority sample recognition errors caused by sample imbalance. Specifically, the attribute name is first expanded into a text description, and then the corresponding text features are generated through CLIP's text encoder. This process is defined as follows: The cosine similarity between the semantic features of the attributes and the text features is calculated to obtain the predicted probability of each attribute. This process is defined as follows: in, The scaling factor is used to adjust the smoothness of the similarity calculation results distribution, controlling the sensitivity of the semantic similarity output to feature differences. The proposed method is trained using weighted cross-entropy loss, and its mathematical expression is as follows: in, This represents the unbalanced weight of the j-th attribute. This represents the proportion of the j-th attribute in the training set.

Images