A multi-view pedestrian attribute recognition method

Abstract

This invention presents a multi-view pedestrian attribute recognition method. It identifies attributes from three views: semantic relevance, regional relevance, and relevance between semantic regions. Semantic graph convolution is used to learn semantic relevance, and visual graph convolution is used to learn regional relevance. A synthetic graph is constructed by jointly using the semantic and visual graphs. Semantic and regional relevance are learned through graph convolution of the synthetic graph, and attributes are predicted using embedded graph edges that fuse feature information from both graphs. Regarding the loss function, traditional methods use the same weighting strategy for all datasets, leading to either overweighting or underweighting on certain datasets. This method uses different weighting methods for different datasets, effectively alleviating the attribute imbalance problem.

Description

Technical Field

[0001] This invention relates to the field of pedestrian attribute recognition technology for recording media used to identify graphics, and more specifically to a multi-view pedestrian attribute recognition method. Background Technology

[0002] Pedestrian attribute recognition refers to identifying the predetermined attributes of a pedestrian from an image given a pedestrian image and a predefined list of attributes for each dataset. These attributes typically include approximate age, gender, clothing, and behavioral patterns. The attribute information includes not only low-level semantics such as hair length and skirt type, but also high-level semantics such as age and gender. These higher-level semantics differ from the low-level semantics and are more robust to changes in viewpoint and imaging conditions. Due to its widespread application in public safety and smart security, pedestrian attribute recognition has become a hot topic in the field of computer vision.

[0003] Pedestrian attribute recognition methods generally fall into two categories. One approach is more intuitive, directly locating attributes by extracting local features based on prior human knowledge or other models. These local features then assist global feature recognition; for example, attention mechanisms can extract specific attribute features to predict corresponding attributes. However, this method is susceptible to external factors such as lighting, occlusion, and changes in human posture. Furthermore, the visual information corresponding to some high-level semantic attributes is rather abstract, and some fine-grained attributes correspond to small regions, making attribute localization more difficult for the network. The other approach involves learning the correlations between attributes. Pedestrian attributes are numerous, such as gender, age, sunglasses, clothing type, and hairstyle. Some of these attributes are closely related. For example, the attribute "skirt" is often associated with the attribute "female," and clothing type can provide information for age determination. Therefore, attribute correlation can improve the performance of pedestrian attribute recognition. When faced with challenges such as different appearances, occlusion of body parts, and low resolution, it is necessary to infer information about the region based on contextual information. For example, when identifying a person's gender, people often focus on multiple regions, such as the head, body, and the area of ​​carried items, and consider their contextual relationships. CN108921051A groups pedestrian attributes by region and uses an attention mechanism to learn the features corresponding to each group. It then combines this with a recurrent neural network to learn the correlations between features within attribute groups. However, the feature relationships within attribute groups can only be learned through fully connected layers, failing to effectively learn the relationships between all attributes. CN109190472A feeds visual features and attribute features into a long short-term memory network to learn the relationship between attributes and visual features. However, long short-term networks can only learn unidirectional relationships between attributes; the correlations between pedestrian attributes are often bidirectional. For example, the attribute 'skirt' often infers the attribute 'female,' and the 'female' attribute increases the probability of the 'skirt' attribute appearing. CN115482584A discloses a pedestrian attribute recognition method based on a graph network (GCN). It uses pedestrian trajectory data to convert it into a graph representation, which can learn spatial connections, but it cannot learn the correlations between attributes. Meanwhile, pedestrian attribute imbalance is also a challenge. Weighting the loss function and using batch normalization before classification are common methods to alleviate pedestrian attribute imbalance. However, the severity of imbalance varies among different pedestrian attribute datasets, and using a consistent loss function weighting method when training on different datasets is unlikely to yield good results. In addition, low resolution, partial body occlusion, and complex appearance variations are all important problems that need to be addressed in pedestrian attribute recognition. Summary of the Invention:

[0004] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a multi-view pedestrian attribute recognition method that uses graph convolution to learn semantic associations, region associations, and the association between semantics and regions. It constructs a semantic graph using attribute-specific feature vectors and learns semantic relevance using graph convolution; it constructs a visual graph based on visual graph features and learns region relevance using graph convolution; it then combines the visual and semantic graphs to construct a synthetic graph and uses graph convolution to establish semantic and region relevance. Furthermore, this invention uses different weighting strategies for the loss function based on the characteristics of the dataset. This invention overcomes the problems of low pedestrian attribute recognition rates caused by low resolution, partial body occlusion, complex appearance variations, and attribute imbalances across different datasets.

[0005] The technical solution adopted by this invention to solve this technical problem is:

[0006] A multi-view pedestrian attribute recognition method, the recognition method includes the following:

[0007] The public dataset is grouped by attribute. The pedestrian images from the public dataset are processed through the first three layers of a ResNet-101 network before entering the fourth layer, which splits into two branches. The two branches obtain semantic feature maps respectively. and visual feature map The first three layers share weights, while the fourth layer is trained separately and the weights are not shared. After pooling, the feature vector is obtained. Semantic classification prediction is performed using a classifier, and semantic constraint loss is applied. constraint;

[0008] at the same time After passing through multiple fully connected layers, we obtain attribute-specific feature vectors. Each attribute-specific feature vector is a node in the semantic graph; all attribute-specific feature vectors... A semantic graph is constructed, and its adjacency matrix is ​​built based on prior knowledge from a public dataset. The semantic graph and its adjacency matrix are then input into a semantic graph convolutional GCN to learn semantic relevance and obtain a semantic association graph. Semantic Relationship Graph Semantic graph prediction is performed using a classifier, and a semantic graph constraint loss is applied. constraint;

[0009] Multiple visual clustering features were obtained through a soft assignment strategy. Each visual clustering feature corresponds to a region feature for each attribute group; multiple visual clustering features A visual graph is constructed. An adjacency matrix is ​​obtained based on the similarity and topological distance between clusters. The visual graph and its adjacency matrix are then input into a visual graph convolutional GCN to learn the correlations between regions corresponding to attribute groups, resulting in a visual association graph. Visual association diagram Visual graph prediction is performed using a classifier, and the loss is constrained by the visual graph. constraint;

[0010] A composite graph is constructed by combining semantic association graph and visual association graph. The composite graph is inter-embedded with the semantic association graph and the visual association graph through graph edges to guide the learning of each other and build semantic and regional relevance.

[0011] The data was trained using a publicly available dataset and then used for pedestrian attribute recognition.

[0012] In the process of constructing the composite graph, semantic association graphs are utilized. Visual association diagram The edges between graphs are obtained after fusion operations and a fully connected layer. Semantic Relationship Graph Updates are divided into intra-graph updates and inter-graph updates, with inter-graph updates guided by inter-graph edges; semantic association graph In-graph update is performed after a semantic graph convolution;

[0013] Visual association diagram The updates are divided into intra-graph updates and inter-graph updates, visual association graphs In-graph update is performed after a visual graph convolution;

[0014] The inter-graph update process is: semantic association graph or visual association diagram Mapping matrices with semantic association graphs respectively or visual association graph mapping matrix Multiply, then multiply by the edges in the graph. Perform the fusion operation, given the semantic association graph weight matrix W. A,V And the weight matrix W of the visual association graph V,A The result after fusing with the inter-graph edges is then compared with the semantic association graph weight matrix W. A,V Or the weight matrix W of the visual association graph V,A Each performs a matrix multiplication operation to complete the inter-graph update of the visual association graph and the semantic association graph;

[0015] Inter-graph update results of semantic association graph and visual association graph Then, the results are fused with the intra-graph update results of the semantic association graph and the visual association graph, respectively. Finally, the fused results are passed through a fully connected layer to obtain the visual embedding graph. and semantic embedding graph Visual Embedded Image Image edge and semantic embedding graph After a fusion operation and a fully connected layer, the updated edge embedding graph is obtained.

[0016] Updated embedded graph edges Then, the embedding graph edge classifier predicts the embedding graph edges, and the synthetic graph constraint loss is used. constraint.

[0017] The specific steps of the multi-view pedestrian attribute recognition method are as follows:

[0018] The first step is to preprocess and extract features from the pedestrian images:

[0019] After resizing the pedestrian images to 256×128, the training images were enhanced using random horizontal flipping, hue and saturation adjustments, rotation, translation, cropping, erasing, and the addition of random Gaussian blur. The semantic feature map of the i-th image was then obtained using ResNet-101. and visual feature map The first three layers of ResNet-101 share weights, while the fourth layer is divided into two branches and trained separately without sharing weights; the downsampling and channel augmentation operations in the fourth layer of ResNet-101 are removed in the visual graph branch.

[0020] The second step is to learn semantic relevance using semantic graphs:

[0021] Based on the semantic feature map obtained in the first step above We obtain attribute-specific feature vectors by using semantic constraint loss constraints, construct a semantic graph using attribute-specific feature vectors, construct an adjacency matrix using the conditional probabilities of attributes in the training set, perform graph convolution, and then obtain the corresponding attribute probability for each node. We then use the loss function to constrain the training of the semantic graph.

[0022] Step 2.1, using semantic constraint loss to obtain attribute-specific feature vectors:

[0023] Semantic feature maps extracted from ResNet-101 The feature vector is obtained after global average pooling. Where c is The number of channels; The semantic classification prediction result is obtained by a semantic classifier consisting of a fully connected layer with an output dimension equal to the number of attributes, batch normalization, and an activation function. A semantic classifier is mathematically represented as:

[0024]

[0025] Among them W A,1 σ represents the parameters of the fully connected layer, σ is the sigmoid function, and BN represents the batch normalization layer. Semantic classification prediction is constrained using a semantic constraint loss function.

[0026] Will Different feature vectors with specific attributes are obtained by passing through fully connected layers with different output dimensions d. At this point, the number of fully connected layers is the same as the number of attributes, which can be mathematically represented as:

[0027]

[0028] It is the parameter of the j-th attribute in the fully connected layer, and the result is... d is the feature vector Length;

[0029] Step 2.2, Semantic Graph Convolution:

[0030] Use all attribute-specific feature vectors merge into As input to the semantic graph, semantic graph convolution is performed to mine the semantic relationships between pedestrian attributes. This part is an in-graph update. When faced with difficulties such as low resolution that make it difficult to identify some fine-grained attributes, the prediction of other attributes and the semantic relationships between attributes are used to predict these fine-grained attributes, which is expressed by the following formula:

[0031]

[0032] Where A A It is a learnable adjacency matrix. The output of semantic graph convolution is the semantic association graph. Let A be the adjacency matrix. A The matrix formed by the diagonals of the reciprocals and square roots of the matrix is ​​called the identity matrix, I. Adjacency matrix A A It is obtained based on prior knowledge of the pedestrian attributes training set. The conditional probability P of one attribute when another attribute appears in the training set can be obtained by the following formula:

[0033] P ab =M ab / N b (4),

[0034] Where P ab M represents the probability that attribute a occurs when attribute b occurs. ab N represents the number of times attributes a and b appear simultaneously in the training set. bThis represents the number of times attribute b appears. The frequency of an attribute appearing in the training set is used as the probability of that attribute. A A By P ab It is obtained by threshold normalization.

[0035] Semantic graph First, with the weight matrix W A Multiply, then with the normalized adjacent matrix Multiplication achieves graph convolution. W... A Set to size d×d, where d is the length of the feature vector, semantic association graph Size and same. The semantic graph prediction results are obtained through a semantic graph classifier consisting of global average pooling (GAP), batch normalization (BN), and a sigmoid activation function. The semantic graph classifier uses a semantic graph constraint to constrain the loss, and its formula is as follows:

[0036]

[0037] The third step is to learn the region relevance of the visual graph:

[0038] Based on the visual feature maps extracted from ResNet-101 Constructing a visual graph involves using topological relationships and similarity to build an adjacency matrix, followed by visual graph convolution. The specific steps are as follows:

[0039] Step 3.1, visual feature map Constructing a visual graph using a soft allocation strategy:

[0040] To learn the region relationships in an image, visual feature maps extracted from ResNet-101 are used. After being mapped onto various clusters using a soft assignment strategy, each cluster represents a set of attributes that correspond to the same visual clustering features; in the soft assignment strategy, Each local feature has a weight for each cluster, and the sum of the contributions of all local features to the same cluster is 1. The resulting visual clustering features are calculated by weighted summation of the transformed local features; in this way, the network can learn clustering on its own to cope with changes in human pose and camera viewpoint, and the resulting visual clustering features are... Multiple visual clustering features The visual graph is constructed, where m represents the number of nodes in the visual graph (different datasets have different numbers of nodes), and l represents the length of the feature vector of each node. The calculation formula is as follows:

[0041]

[0042] Among them, AVS W is the mapping matrix, representing the normalization performed at each position using softmax; vs Here is the weight matrix; a t,s ∈A VS The calculation formula is as follows:

[0043]

[0044] a t,s ∈A VS Representation of visual feature maps The s-th point Assign to visual image The confidence level of node t. For calculating the trainable weight matrix with assigned weights;

[0045] Step 3.2: Construct an adjacency matrix based on the topological distance and similarity between clusters and then perform graph convolution.

[0046] The visual graph updates based on the distance and similarity between regions corresponding to these attribute groups, thereby learning contextual relationships. When encountering appearance changes or partial occlusion of the body, it can identify the attributes corresponding to hard-to-identify regions through other regions. This part of the graph convolution is an in-graph update, and the formula is as follows:

[0047]

[0048] in The result of convolving a visual graph, i.e., a visual association graph, W V1 ∈R l×l and W V2 ∈R l×l A is a trainable weight matrix. V1 ∈R m×m and A V2 ∈R m×m Let A be an adjacency matrix. V1 This is obtained from the normalized results of the similarity between clusters. First, the pairwise similarity between the two local regions, node k and node p, is calculated:

[0049]

[0050] in and These are transformations of visual features, respectively. Normalization yields A V1 The formula is as follows:

[0051]

[0052] Where m is the number of nodes in the visual graph.

[0053] A V1 This is obtained from the normalized result of the topological distance between clusters. The formula is as follows:

[0054]

[0055] Where d(·) represents the topological distance calculation operation, visual association graph Using a visual graph classifier for prediction, for Each node All of them are processed by a visual graph classifier to obtain visual graph node prediction results. The outputs of all nodes constitute the visual graph prediction. The visual graph classifier also includes a fully connected layer, a batch normalization layer, and a sigmoid function. The output dimension of the fully connected layer in the visual graph classifier is the number of attributes contained in the attribute group corresponding to the node. The formula for the visual graph classifier is as follows:

[0056]

[0057] in, These represent the parameters of the fully connected layer in a visual graph classifier.

[0058] The visual graph classifier is constrained using visual graph constraint loss so that each visual cluster feature can learn the region features of the corresponding attribute group.

[0059] Step 4: Synthesize graph convolution:

[0060] To better learn the visual semantic relationships between attributes, the semantic association map and the visual association map are combined into a composite map. Specifically:

[0061] Step 4.1: Construct a composite graph by combining the semantic association graph and the visual association graph:

[0062] All nodes in the semantic association graph are connected to all nodes in the visual association graph, forming inter-graph edges. The features of graph edges are generated by the two endpoints; graph edges The formula is as follows:

[0063]

[0064] in This is a visual association graph mapping matrix. For a fully connected layer, c(·) is the fusion operation, and the graph edges... R K×m×d K is the number of nodes in the semantic association graph, i.e., the number of attributes; m is the number of nodes in the visual association graph; d is the length of the node vector in the semantic association graph; and the number of edges between graphs is... The number of nodes is the product of the number of nodes in the semantic graph and the number of nodes in the visual graph; composite graph Visual association diagram Semantic Relationship Graph And the edge between the graph composition;

[0065] Step 4.2, Construct the graph convolution strategy for the synthetic graph:

[0066] In the composite image In semantic association graphs and visual association graphs, updates are divided into intra-graph updates and inter-graph updates, with inter-graph updates guided by inter-graph edges. Intra-graph updates of semantic association graphs are performed using semantic graph convolution, while the other part is generated by combining inter-graph edges with the visual association graph. The visual association graph is mapped to d dimensions using a mapping matrix, then fused with inter-graph edges, and finally weighted with a weight matrix. This part aims to embed visual information into the semantic association graph, using region relationships to guide the learning of semantic relationships. The inter-graph update of the semantic association graph is expressed by the following formula:

[0067]

[0068] in The result is obtained from the inter-graph update of the semantic association graph. Let c(·) be the edge between graphs, and c(·) be the merge operation. W is the visual association graph mapping matrix, which maps the dimensions of the visual association graph to the same dimensions as the semantic association graph. V,A For the visual association graph weight matrix, the information obtained from the inter-graph edges and the visual association graph is to be allocated to each node of the semantic association graph. The sum of the contributions of all inter-graph edges connecting the semantic association graph node and the visual association graph node to the update of that node is 1.

[0069] The update formula for the semantic association graph is as follows:

[0070]

[0071] in This is the updated semantic embedding graph. For a fully connected layer with an input dimension of 2d and an output dimension of d, c(·) is the fusion operation, and W A Let be the weight matrix of the semantic graph convolution. The update of the visual association graph is similar to that of the semantic association graph, where the inter-graph update formula for the visual association graph is as follows:

[0072]

[0073] in The result is obtained from the inter-graph update of the visual association graph. Let c(·) be the edge between graphs, and c(·) be the merge operation. W is the semantic association graph mapping matrix. A,V It is the semantic association graph weight matrix;

[0074] The update formula for the visual association graph is as follows:

[0075]

[0076] in This is the updated visual embedding image. For a fully connected layer with input dimension 2l and output dimension l, c(·) is the fusion operation, and W V1 and W V2 This is the weight matrix for the visual graph convolution;

[0077] After the semantic and visual association graphs complete the inter-graph update operation, the inter-graph edges are updated, which consists of the inter-graph edges. Visual Embedded Image and semantic embedding graph The formula is obtained by passing through a fully connected layer with an input dimension of 3l and an output dimension of l, as shown below:

[0078]

[0079] in This is a fully connected layer with an input dimension of 3l and an output dimension of l, where c(·) is the fusion operation. The mapping matrix yields the embedded graph edges. and They are the same size.

[0080] Step 4.2, embedding inter-graph edge prediction attributes:

[0081] The inter-graph edges are embedded and then processed by the inter-graph edge embedding classifier to obtain the predicted inter-graph edges. The embedded graph edge classifier includes global max pooling, global average pooling, batch normalization layers, and activation functions. The formula for the embedded graph edge classifier is as follows:

[0082]

[0083] Where GMP stands for Global Max Pooling, GAP for Global Average Pooling, BN for Batch Normalization, and σ is the activation function. As the final prediction, and with the loss constrained by the synthetic graph. constraint.

[0084] Step 5, weighted loss function:

[0085] The loss function used is the adaptive binary clustering loss function based on the binary clustering loss function (polyloss). There are four losses used to guide the training of the entire network: one is the semantic constraint loss, two are the semantic graph constraint loss and the visual graph constraint loss, and one is the synthetic graph constraint loss. The binary clustering loss function consists of many polynomials, unifying the binary cross-entropy loss function and the binary focus loss function. A better loss weighting method is used based on the imbalance of the pedestrian attribute dataset. An adaptive binary clustering loss function is proposed on the basis of the binary clustering loss function and applied to these four losses (see formula (20)). Taking the loss of attribute features in the semantic graph as an example, it can be written as:

[0086]

[0087]

[0088] in Let be the binary cross-entropy loss function, ∈ be the hyperparameter, n be the number of samples, i and j represent the j-th attribute of the i-th image, and p be the hyperparameter. t The predicted value of an attribute for the target category can be expressed by the following formula:

[0089]

[0090] ρ ij This is a penalty coefficient used to mitigate the imbalanced data problem in pedestrian attribute recognition. Two strategies are used in the network. We use the positive sample rate of an attribute in a dataset to measure attribute imbalance; each attribute has a positive sample rate, where the formula for the positive sample rate of attribute j is as follows:

[0091] r j =N j / N (23),

[0092] Where N j The number of times attribute j appears in the training set is represented by , and N represents the total number of samples in the training set.

[0093] We consider a dataset to be severely imbalanced if the ratio of the maximum to the minimum positive sample rate for all attributes is greater than 100, and not severely imbalanced if it is less than 100. This ratio is as follows:

[0094] r = Max(r1, r2, ..., r j ) / Min(r1,r2,...,r j )

[0095] (twenty four),

[0096] Among the three public datasets for pedestrian attribute recognition, the ratio for the PETA dataset is 48, while the ratios for the RAP and PA-100K datasets are 230 and 1880, respectively. For the PETA dataset (which has a small dataset size and relatively mild attribute imbalance), an exponential strategy is used, employing relatively smooth attribute weights.

[0097]

[0098] Where r j This represents the positive sample ratio for attribute j. For the RAP and PA-100K datasets (which suffer from severe attribute imbalance and have few positive samples), a square root strategy was employed to emphasize attributes with rarer positive samples:

[0099]

[0100] The losses for semantic graphs, visual graphs, and synthetic graphs are calculated in a similar manner, respectively. as well as The total loss for training the network is as follows:

[0101]

[0102] To reduce the influence of easily negative samples, p = max(p - 0.15, 0) is set before calculating the loss. These losses are jointly and simultaneously optimized to fully learn the attributes and contextual relationships of pedestrian attribute recognition. To effectively learn semantic and visual relationships in the synthetic graph, the semantic and visual graphs fed into the synthetic graph need to have good semantic and visual features. Therefore, in the first 15 iterations of training, only the visual association graph and semantic association graph acquisition parts are trained. The prediction results of the visual association graph and semantic association graph acquisition parts are the average of the semantic graph prediction and the visual graph prediction. for: Used to evaluate the accuracy of the acquired part; after the first 15 iterations, the entire network is trained, and the network's prediction results are the prediction results of the inter-edge embeddings. Predicting results using embedded graph edges Evaluate the accuracy of the entire network.

[0103] The present invention also protects a computer-readable storage medium storing a computer program adapted to execute the multi-view pedestrian attribute recognition method when loaded by a computer.

[0104] Compared with the prior art, the beneficial effects of the present invention are:

[0105] The significant advancements of this invention are as follows:

[0106] This invention identifies attributes from three views: semantic relevance, regional relevance, and relevance between semantic regions. Semantic graph convolution is used to learn semantic relevance, and visual graph convolution is used to learn regional relevance. A synthetic graph is constructed by jointly using the semantic and visual graphs. A novel synthetic graph convolution is used to learn semantic and regional relevance, and embedded graph edges that fuse feature information from both graphs are used to predict attributes. Regarding the loss function, traditional methods use the same weighting strategy for all datasets, leading to overweighting or underweighting on certain datasets. This method uses different weighting methods for different datasets, differing from mainstream weighting approaches, effectively alleviating the attribute imbalance problem across different datasets.

[0107] The most prominent and essential feature of this invention is:

[0108] (1) The method of this invention adopts a multi-view approach to identify pedestrian attributes. It uses graph convolution to learn three types of views: semantic association, regional association, and semantic-regional association. The semantic association graph obtained by convolution of the semantic graph that learns semantic association and the visual association graph obtained by convolution of the visual graph that learns regional association are jointly constructed to form a composite graph. The information of the visual association graph and the semantic association graph are combined to obtain the inter-graph edges. The graph convolution of the composite graph is divided into intra-graph update and inter-graph update. Intra-graph update ensures the stable learning of regional and semantic relationships. Inter-graph update combines the inter-graph edges containing information from the two graphs with the information of the other graph. In this way, the final update can effectively combine the two relationships. Finally, the inter-graph edges are embedded to combine the information from the two graphs. Through multiple combinations of the two relationships, both regional and semantic relationships effectively guide the learning of each other.

[0109] (2) The method of the present invention uses semantic constraints and soft assignment mapping to obtain attribute-specific feature vectors and visual clustering features for high-level features respectively. It uses attribute-specific feature vectors to construct semantic graphs, uses conditional probabilities between attributes in the training set to construct adjacency matrices, performs graph convolution on the semantic graph to learn semantic relevance, uses visual clustering features to construct visual graphs, takes advantage of the proximity of regions corresponding to each attribute group, each cluster corresponds to the features of each attribute group, uses similarity between clusters and topological distance to construct adjacency matrices, performs graph convolution on the visual graph to learn region relevance, both are constrained by loss, and network aggregation learning.

[0110] (3) This invention addresses the issue of varying severity of attribute imbalance in public pedestrian attribute datasets by employing different weighting methods for the loss function. Compared to the two mainstream weighting methods, this approach more effectively alleviates the corresponding attribute imbalance across multiple datasets. During training, the semantic graph and visual graph are trained first to ensure they can effectively learn the two relationships before training the synthetic graph.

[0111] (4) Compared with CN115346237A: CN115346237A uses the Inception network to extract features and the SeNet attention mechanism to make the network pay more attention to the spatial location of pedestrian attributes. Attention is a behavioral and cognitive process that selectively focuses on a discrete aspect of information, ignoring other perceptible information. However, pedestrian images are affected by lighting and low resolution, making it difficult to locate pedestrian attributes. Moreover, it directly uses fully connected layers and activation functions for classification after attention, which makes it difficult to learn regional and semantic correlations. In contrast, this method can learn both correlations well through graph convolution and uses other attributes or regions to learn attributes that are difficult to learn.

[0112] (5) Compared with CN114663908A: CN114663908A is a pedestrian attribute recognition method based on multi-scale feature fusion. It uses VGG19 as the backbone network and inputs the features after the Conv4_3 and Conv5_3 convolutional layers into the feature fusion module. The fused features are then resized and classified to predict attributes. Multi-scale feature fusion can combine low-level semantic features and high-level semantic features. However, this method is difficult to explicitly learn regional correlations and cannot learn semantic correlations. This method uses an exponential weighting method for the loss function to address the imbalance of the dataset. However, the imbalance problem of different datasets is different. It is difficult to achieve good results on each dataset by using a uniform weighting method. This method uses different weighting methods for different datasets, and the weighting is different from the mainstream weighting method. It has achieved good results on each dataset. Attached Figure Description

[0113] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0114] Figure 1 This is a flowchart illustrating the pedestrian attribute recognition method of the present invention.

[0115] Figure 2 This is a schematic diagram of the network structure of the semantic association graph acquisition part in the method of the present invention.

[0116] Figure 3 This is a schematic diagram of the network structure of the visual association graph acquisition part in the method of the present invention.

[0117] Figure 4 This is a schematic diagram of the synthetic graph construction and synthetic graph convolution process in the method of the present invention. Detailed Implementation

[0118] Figure 1The illustrated embodiment shows that the process of the method of the present invention is as follows: pedestrian attribute recognition preprocessing and feature extraction → semantic graph construction of semantic relevance → visual graph construction of region relevance → synthetic graph construction and graph convolution → weighted loss function. The public dataset is grouped by attribute, with each group containing one type of attribute, such as age and gender as one group. ResNet-101 is used as the backbone network. For the i-th image I... i It first uses pedestrian images from a public dataset, processes them through the first three layers of ResNet-101, and then enters the fourth layer, which is divided into two branches: a semantic branch and a visual branch, to obtain semantic feature maps. and visual feature map The first three layers share weights, while the fourth layer is trained separately and the weights are not shared. After pooling, the feature vector is obtained. Predictions are made using a classifier, and semantic constraint loss is applied. constraint. A semantic graph is obtained after passing through multiple fully connected layers. Semantic graph convolution is then performed to learn semantic relevance, and semantic graph-constrained loss is used. constraint.

[0119] Multiple visual clustering features were obtained through a soft allocation strategy. Multiple visual clustering features Construct a visual graph, perform visual graph convolution, learn region correlations, and use the visual graph to constrain the loss. constraint.

[0120] Then, the semantic graph is convolved to obtain the semantic association graph. Visual association graph obtained by convolving with the visual graph Construct a synthetic graph, utilize graph convolution of the synthetic graph to learn the correlation between semantics and regions, and finally embed the inter-graph edges of the graph after graph convolution of the synthetic graph. The predicted probability of each attribute is obtained through a classifier, and the loss is constrained using a synthetic graph. Constraints. Using semantic constraint loss. Semantic graph constraint loss Visual graph constraint loss Synthetic graph constraint loss A joint supervision network, with a loss function of suit_polyloss, can effectively alleviate the property imbalance problem between different datasets.

[0121] Figure 2 The illustrated embodiment shows that semantic feature maps are extracted from pedestrian images using ResNet101. The feature vector is obtained by global average pooling (GAP). The attribute prediction vector is obtained after classification and then subjected to semantic constraint loss. constraint, This becomes the feature vector of semantic constraints. After passing through multiple fully connected layers, we obtain attribute-specific feature vectors. Each attribute-specific feature vector is a node in the semantic graph; all attribute-specific feature vectors... A semantic graph is constructed, and its adjacency matrix is ​​built based on prior knowledge from a public dataset. The semantic graph and its adjacency matrix are then input into a semantic graph convolutional GCN to learn semantic relevance and obtain a semantic association graph. Using semantic graphs to constrain loss constraint.

[0122] Figure 3 The illustrated embodiment shows that visual feature maps are extracted from pedestrian images using ResNet101. Multiple visual clustering features were obtained through a soft assignment strategy. Each visual clustering feature corresponds to a region feature for each attribute group; multiple visual clustering features A visual graph is constructed. An adjacency matrix is ​​obtained based on the similarity and topological distance between clusters. The visual graph and its adjacency matrix are then input into a visual graph convolutional GCN to learn region correlations—that is, the relationships between regions corresponding to attribute groups—and thus obtain a visual association graph. And constrain the loss using a visual graph. constraint.

[0123] Figure 4 The illustrated embodiment demonstrates that, in the process of constructing a synthetic graph, a semantic association graph is used. Visual association diagram Based on the nodes, the graph edges are obtained through fusion operations and fully connected layers. In the process of graph convolution, semantic association graph The updates are divided into intra-graph updates and inter-graph updates, with inter-graph updates guided by inter-graph edges. Semantic Relationship Graph In-graph update is performed after a semantic graph convolution;

[0124] Visual association diagram The updates are divided into intra-graph updates and inter-graph updates, visual association graphs In-graph update is performed after a visual graph convolution;

[0125] The inter-graph update process is: semantic association graph or visual association diagram Mapping matrices with semantic association graphs respectively or visual association graph mapping matrix Multiply, then multiply by the edges in the graph. Perform the fusion operation, given the semantic association graph weight matrix W. A,VAnd the weight matrix W of the visual association graph V,A The result after fusing with the inter-graph edges is then compared with the semantic association graph weight matrix W. A,V Or the weight matrix W of the visual association graph V,A Each performs a matrix multiplication operation to complete the inter-graph update of the visual association graph and the semantic association graph.

[0126] The inter-graph update results of the semantic association graph and the visual association graph are then fused with the intra-graph update results of the semantic association graph and the intra-graph update results of the visual association graph, respectively. Finally, the fused results are passed through a fully connected layer to obtain the visual embedding graph. and semantic embedding graph Visual Embedded Image Image edge and semantic embedding graph After a fusion operation and a fully connected layer, the updated edge embedding graph is obtained.

[0127] Example 1

[0128] The multi-view pedestrian attribute recognition method in this embodiment is a pedestrian attribute recognition method that uses graph convolution to learn the correlation between semantics and regions in two views and combines it with an optimized loss weighting. The specific steps are as follows:

[0129] The first step is to preprocess and extract features from the pedestrian images:

[0130] After resizing the pedestrian images to 256×128, the training images are enhanced using operations such as random horizontal flipping, hue and saturation adjustments, rotation, translation, cropping, erasing, and adding random Gaussian blur. The semantic feature map of the i-th image is then obtained using ResNet-101. and visual feature map The first three layers of ResNet-101 share weights, while the fourth layer is divided into two branches and trained separately without sharing weights; the downsampling and channel augmentation operations in the fourth layer of ResNet-101 are removed in the visual graph branch.

[0131] The second step is to learn semantic relevance using semantic graphs:

[0132] Based on the semantic feature map obtained in the first step above We obtain attribute-specific feature vectors by using semantic constraint loss constraints, construct a semantic graph using attribute-specific feature vectors, construct an adjacency matrix using the conditional probabilities of attributes in the training set, perform graph convolution, and then obtain the corresponding attribute probability for each node. We then use the loss function to constrain the training of the semantic graph.

[0133] Step 2.1, using semantic constraint loss to obtain attribute-specific feature vectors:

[0134] Semantic feature maps extracted from ResNet-101 The feature vector is obtained after global average pooling. Where c is The number of channels; The semantic classification prediction result is obtained by a semantic classifier consisting of a fully connected layer with an output dimension equal to the number of attributes, batch normalization, and an activation function. A semantic classifier is mathematically represented as:

[0135]

[0136] Among them W A,1 σ represents the parameters of the fully connected layer, σ is the sigmoid function, and BN represents the batch normalization layer. Semantic classification prediction is constrained using a semantic constraint loss function.

[0137] Will Different feature vectors with specific attributes are obtained by passing through fully connected layers with different output dimensions d. At this point, the number of fully connected layers is the same as the number of attributes, which can be mathematically represented as:

[0138]

[0139] It is the parameter of the j-th attribute in the fully connected layer, and the result is... d is the feature vector Length;

[0140] Step 2.2, Semantic Graph Convolution:

[0141] Use all attribute-specific feature vectors merge into As input to the semantic graph, semantic graph convolution is performed to mine the semantic relationships between pedestrian attributes. This part is an in-graph update. When faced with difficulties such as low resolution that make it difficult to identify some fine-grained attributes, the prediction of other attributes and the semantic relationships between attributes are used to predict these fine-grained attributes, which is expressed by the following formula:

[0142]

[0143] Where A A It is a learnable adjacency matrix. The output of semantic graph convolution is the semantic association graph. Let A be the adjacency matrix. A The matrix formed by the diagonals of the reciprocals and square roots of the matrix is ​​called the identity matrix, I. Adjacency matrix A AIt is obtained based on prior knowledge of the pedestrian attributes training set. The conditional probability P of one attribute occurring when another attribute occurs in the training set can be obtained by the following formula:

[0144] P ab =M ab / N b (4),

[0145] Where P ab M represents the probability that attribute a occurs when attribute b occurs. ab N represents the number of times attributes a and b appear simultaneously in the training set. b This represents the number of times attribute b appears. The frequency of an attribute appearing in the training set is used as the probability of that attribute. A A By P ab It is obtained by threshold normalization.

[0146] Semantic graph First, with the weight matrix W A Multiply, then with the normalized adjacent matrix Multiplication achieves graph convolution. W... A Set to size d×d, where d is the length of the feature vector, semantic association graph Size and same. The semantic graph prediction results are obtained through a semantic graph classifier consisting of global average pooling (GAP), batch normalization (BN), and a sigmoid activation function. The semantic graph classifier uses a semantic graph constraint to constrain the loss, and its formula is as follows:

[0147]

[0148] The third step is to learn the region relevance of the visual graph:

[0149] Based on the visual feature maps extracted from ResNet-101 Constructing a visual graph involves using topological relationships and similarity to build an adjacency matrix, followed by visual graph convolution. The specific steps are as follows:

[0150] Step 3.1, visual feature map Constructing a visual graph using a soft allocation strategy:

[0151] To learn the region relationships in an image, visual feature maps extracted from ResNet-101 are used. After being mapped onto various clusters using a soft assignment strategy, each cluster represents a set of attributes that correspond to the same visual clustering features; in the soft assignment strategy, Each local feature has a weight for each cluster, and the sum of the contributions of all local features to the same cluster is 1. The resulting visual clustering features are calculated by weighted summation of the transformed local features; in this way, the network can learn clustering on its own to cope with changes in human pose and camera viewpoint, and the resulting visual clustering features are... Multiple visual clustering features The visual graph is constructed, where m represents the number of nodes in the visual graph (different datasets have different numbers of nodes), and l represents the length of the feature vector of each node. The calculation formula is as follows:

[0152]

[0153] Among them, A VS W is the mapping matrix, representing the normalization performed at each position using softmax; vs Here is the weight matrix; a t,s ∈A VS The calculation formula is as follows:

[0154]

[0155] a t,s ∈A VS Representation of visual feature maps The s-th point Assign to visual image The confidence level of node t. For calculating the trainable weight matrix with assigned weights;

[0156] Step 3.2: Construct an adjacency matrix based on the topological distance and similarity between clusters and then perform graph convolution.

[0157] The visual graph updates based on the distance and similarity between regions corresponding to these attribute groups, thereby learning contextual relationships. When encountering appearance changes or partial occlusion of the body, it can identify the attributes corresponding to hard-to-identify regions through other regions. This part of the graph convolution is an in-graph update, and the formula is as follows:

[0158]

[0159] in The result of convolving a visual graph, i.e., a visual association graph, W V1 ∈R l×l and W V2 ∈R l×l A is a trainable weight matrix. V1 ∈R m×m and A V2 ∈R m×m Let A be an adjacency matrix. V1This is obtained from the normalized results of the similarity between clusters. First, the pairwise similarity between the two local regions, node k and node p, is calculated:

[0160]

[0161] in and These are transformations of visual features, respectively. Normalization yields A V1 The formula is as follows:

[0162]

[0163] Where m is the number of nodes in the visual graph.

[0164] A V1 This is obtained from the normalized result of the topological distance between clusters. The formula is as follows:

[0165]

[0166] Where d(·) represents the topological distance calculation operation, visual association graph Using a visual graph classifier for prediction, for Each node All of them are processed by a visual graph classifier to obtain visual graph node prediction results. The outputs of all nodes constitute the visual graph prediction. The visual graph classifier also includes a fully connected layer, a batch normalization layer, and a sigmoid function. The output dimension of the fully connected layer in the visual graph classifier is the number of attributes contained in the attribute group corresponding to the node. The formula for the visual graph classifier is as follows:

[0167]

[0168] in, These represent the parameters of the fully connected layer in a visual graph classifier.

[0169] The visual graph classifier is constrained using visual graph constraint loss so that each visual cluster feature can learn the region features of the corresponding attribute group.

[0170] Step 4: Synthesize graph convolution:

[0171] To better learn the visual semantic relationships between attributes, the semantic association map and the visual association map are combined into a composite map. Specifically:

[0172] Step 4.1: Construct a composite graph by combining the semantic association graph and the visual association graph:

[0173] All nodes in the semantic association graph are connected to all nodes in the visual association graph, forming inter-graph edges. The features of graph edges are generated by the two endpoints; graph edges The formula is as follows:

[0174]

[0175] in This is a visual association graph mapping matrix. For a fully connected layer, c(·) is the fusion operation, and the graph edges... K is the number of nodes in the semantic association graph, i.e., the number of attributes; m is the number of nodes in the visual association graph; d is the length of the node vector in the semantic association graph; and the number of edges between graphs. The number of nodes is the product of the number of nodes in the semantic graph and the number of nodes in the visual graph; composite graph Visual association diagram Semantic Relationship Graph And the edge between the graph composition;

[0176] Step 4.2, Construct the graph convolution strategy for the synthetic graph:

[0177] In the composite image In semantic association graphs and visual association graphs, updates are divided into intra-graph updates and inter-graph updates, with inter-graph updates guided by inter-graph edges. Intra-graph updates of semantic association graphs are performed using semantic graph convolution, while the other part is generated by combining inter-graph edges with the visual association graph. The visual association graph is mapped to d dimensions using a mapping matrix, then fused with inter-graph edges, and finally weighted with a weight matrix. This part aims to embed visual information into the semantic association graph, using region relationships to guide the learning of semantic relationships. The inter-graph update of the semantic association graph is expressed by the following formula:

[0178]

[0179] in The result is obtained from the inter-graph update of the semantic association graph. Let c(·) be the edge between graphs, and c(·) be the merge operation. W is the visual association graph mapping matrix, which maps the dimensions of the visual association graph to the same dimensions as the semantic association graph. V,A For the visual association graph weight matrix, the information obtained from the inter-graph edges and the visual association graph is to be allocated to each node of the semantic association graph. The sum of the contributions of all inter-graph edges connecting the semantic association graph node and the visual association graph node to the update of that node is 1.

[0180] The update formula for the semantic association graph is as follows:

[0181]

[0182] in This is the updated semantic embedding graph. For a fully connected layer with an input dimension of 2d and an output dimension of d, c(·) is the fusion operation, and W A Let be the weight matrix of the semantic graph convolution. The update of the visual association graph is similar to that of the semantic association graph, where the inter-graph update formula for the visual association graph is as follows:

[0183]

[0184] in The result is obtained from the inter-graph update of the visual association graph. Let c(·) be the edge between graphs, and c(·) be the merge operation. W is the semantic association graph mapping matrix. A,V It is the semantic association graph weight matrix;

[0185] The update formula for the visual association graph is as follows:

[0186]

[0187] in This is the updated visual embedding image. For a fully connected layer with input dimension 2l and output dimension l, c(·) is the fusion operation, and W V1 and W V2 This is the weight matrix for the visual graph convolution;

[0188] After the semantic and visual association graphs complete the inter-graph update operation, the inter-graph edges are updated, which consists of the inter-graph edges. Visual Embedded Image and semantic embedding graph The formula is obtained by passing through a fully connected layer with an input dimension of 3l and an output dimension of l, as shown below:

[0189]

[0190] in This is a fully connected layer with an input dimension of 3l and an output dimension of l, where c(·) is the fusion operation. The mapping matrix yields the embedded graph edges. and They are the same size.

[0191] Step 4.2, embedding inter-graph edge prediction attributes:

[0192] The inter-graph edges are embedded and then processed by the inter-graph edge embedding classifier to obtain the predicted inter-graph edges. The embedded graph edge classifier includes global max pooling, global average pooling, batch normalization layers, and activation functions. The formula for the embedded graph edge classifier is as follows:

[0193]

[0194] Where GMP stands for Global Max Pooling, GAP for Global Average Pooling, BN for Batch Normalization, and σ is the activation function. As the final prediction, and with the loss constrained by the synthetic graph. constraint.

[0195] Step 5, weighted loss function:

[0196] The loss function used is the adaptive binary clustering loss function based on the binary clustering loss function (polyloss). There are four losses used to guide the training of the entire network: one is the semantic constraint loss, two are the semantic graph constraint loss and the visual graph constraint loss, and one is the synthetic graph constraint loss. The binary clustering loss function consists of many polynomials, unifying the binary cross-entropy loss function and the binary focus loss function. A better loss weighting method is used based on the imbalance of the pedestrian attribute dataset. An adaptive binary clustering loss function is proposed on the basis of the binary clustering loss function and applied to these four losses (see formula (20)). Taking the loss of attribute features in the semantic graph as an example, it can be written as:

[0197]

[0198]

[0199] in Let be the binary cross-entropy loss function, ∈ be the hyperparameter, n be the number of samples, i and j represent the j-th attribute of the i-th image, and p be the hyperparameter. t The predicted value of an attribute for the target category can be expressed by the following formula:

[0200]

[0201] ρ ij This is a penalty coefficient used to mitigate the imbalanced data problem in pedestrian attribute recognition. Two strategies are used in the network. We use the positive sample rate of an attribute in a dataset to measure attribute imbalance; each attribute has a positive sample rate, where the formula for the positive sample rate of attribute j is as follows:

[0202] r j =N j / N (23),

[0203] Where N j This indicates the number of times attribute j appears in the dataset, and N represents the total number of samples in the dataset.

[0204] A dataset is considered severely imbalanced if the ratio of the maximum to the minimum positive sample rate for all attributes is greater than 100, and less imbalanced if it is less than 100. This ratio is as follows:

[0205] r = Max(r1, r2, ..., r j ) / Min(r1,r2,...,r j ) (twenty four),

[0206] In the three public datasets for pedestrian attribute recognition, the ratio for the PETA dataset is 48, while the ratios for the RAP and PA-100K datasets are 230 and 1880, respectively. The PETA dataset employs an exponential strategy, which uses relatively smooth attribute weights.

[0207]

[0208] Where r j This represents the positive sample ratio for attribute j. For the RAP and PA-100K datasets (which suffer from severe attribute imbalance and have few positive samples), a square root strategy was employed to emphasize attributes with rarer positive samples:

[0209]

[0210] The losses for semantic graphs, visual graphs, and synthetic graphs are calculated in a similar manner, respectively. as well as The total loss for training the network is as follows:

[0211]

[0212] To reduce the influence of easily negative samples, p = max(p - 0.15, 0) is set before calculating the loss. These losses are jointly and simultaneously optimized to fully learn the attributes and contextual relationships of pedestrian attribute recognition. To effectively learn semantic and visual relationships in the synthetic graph, the semantic and visual graphs fed into the synthetic graph need to have good semantic and visual features. Therefore, a maximum number of iterations is set, and only the visual association graph and semantic association graph acquisition parts are trained in the first 15 iterations. The prediction results of the visual association graph and semantic association graph acquisition parts are the average of the semantic graph prediction and the visual graph prediction. for: Used to evaluate the accuracy of the acquired part; after the first 15 iterations, the entire network is trained, and the prediction result of the entire network is the prediction result of the inter-edge embedding graph. Predicting results using embedded graph edges Evaluate the accuracy of the entire network.

[0213] This implementation example was tested on the PETA, PA100K, and RAP datasets. The PETA dataset consists of several small public datasets. This dataset comprises 19,000 images with resolutions ranging from 17×39 to 169×365 pixels. These 19,000 images include 8,705 individuals, each labeled with 61 binary attributes and 4 multi-class attributes. The images were randomly divided into 9,500 for training, 1,900 for validation, and 7,600 for testing. Thirty-five binary attributes were used for pedestrian attribute recognition experiments.

[0214] The RAP dataset comes from real-world indoor surveillance scenarios, using images from 26 cameras, containing 41,585 samples with resolutions ranging from 36×92 to 344×554. Specifically, 33,268 images were used for training and 33,268 for testing. Seventy-two fine-grained attributes (69 binary attributes and 3 multi-class attributes) were assigned to each image in the dataset. Three environmental and contextual factors—viewpoint, occlusion method, and body part—were explicitly annotated. Attribute annotation considered six parts (spatiotemporal information, whole-body attributes, accessories, pose and action, occlusion, and body part attributes). Fifty-one binary attributes were used for pedestrian attribute recognition experiments.

[0215] The PA-100K dataset is constructed from images captured by 598 real outdoor surveillance cameras. It includes 100,000 pedestrian images with resolutions ranging from 50×100 to 758×454, making it the largest dataset to date for pedestrian attribute recognition. The entire dataset is randomly divided into training, validation, and test sets in an 8:1:1 ratio. Each image in the dataset is labeled with 26 attributes, with labels of either 0 or 1, representing the presence or absence of the corresponding attribute.

[0216] Five criteria were used to evaluate the performance of networks on the PA-100K, PETA, and RAP datasets, including label-based standard mean precision (mA) and four instance-based standard precision (Accu), accuracy (Prec), recall, and F1.

[0217] The proposed synthetic graph method is compared with several classic methods, including HP-Net, VeSPA, VAA, RA, LG-Net, JLPLS-PAA, CoCNN, ALM, MT-CAS, DA-HAR, SEMC, MCFL, and SO-C-SAM. Tables 1, 2, and 3 list the accuracy comparisons of the proposed method with other methods on the PETA, PA100K, and RAP datasets, respectively, where ours represents the proposed method.

[0218] Table 1 Comparison results of the PETA dataset

[0219]

[0220] Table 2 Comparison results of the PA100K dataset

[0221]

[0222] Table 3 Comparison results of RAP datasets

[0223]

[0224] As can be seen from Tables 1, 2, and 3, the method of this invention achieves higher mA scores than other methods in all three databases, and its F1 score is comparable to other methods. The reason why the method of this invention performs better in label-based metrics than in instance-based metrics is that this application uses a loss function weighting method for constraint, which can better alleviate the imbalance of attribute distribution for different datasets.

[0225] HP-Net, VeSPA, VAA, RA, LG-Net, JLPLS-PAA, CoCNN, ALM, MT-CAS, DA-HAR, SEMC, MCFL, and SO-C-SAM are all well-known in the field.

[0226] The attribute groupings for the three public datasets are shown in Tables 4, 5, and 6.

[0227] Table 4. Attribute grouping of the PETA dataset

[0228]

[0229] Table 5. Attribute grouping of the RAP dataset

[0230]

[0231] Table 6. Attribute grouping of the PA100K dataset

[0232]

[0233]

[0234] This invention discloses a multi-view pedestrian attribute recognition method. After preprocessing and feature extraction of pedestrian images, a semantically constrained loss function is used to constrain attribute-specific feature vectors. A semantic graph convolution is then constructed to obtain a semantic association graph, learning semantic relevance. A soft-assignment strategy is applied to the visual feature map to obtain multiple clusters, constructing a visual graph. An adjacency matrix is ​​constructed based on the topological distance and similarity between clusters, and graph convolution is performed to obtain a visual association graph, learning region relevance. A composite graph is constructed by jointly using the semantic association graph and the visual association graph. Semantic and region relevances are constructed through graph convolution of the composite graph, and attributes are predicted using embedded graph edges. Finally, a weighted loss function is used for training. The composite graph constructed in this application can jointly learn region relations and semantic relations, enabling the two relations to better guide each other's learning. A segmented training method is used to allow the network to first learn the two relations well, and then the composite graph is used to mutually guide each other's learning. A better loss weighting method is used based on the imbalance of the pedestrian attribute dataset. An adaptive cluster loss is proposed based on the cluster loss, effectively solving the problem of varying degrees of imbalance in the pedestrian attribute dataset.

[0235] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A multi-view pedestrian attribute recognition method, the recognition method comprising the following contents: The attribute group is carried out on the public data set, and the pedestrian pictures of the public data set are divided into two branches after being processed by the first three layers of the ResNet-101 network and entering the fourth layer, and the two branches obtain semantic feature maps and visual feature maps respectively , The first three layers share weights, and the fourth layer is trained separately without sharing weights After pooling, a feature vector is obtained , The semantic classification prediction is carried out through a classifier, and a semantic constraint loss is used for constraint ; At the same time Through multiple fully connected layers to obtain attribute-specific feature vectors Each attribute-specific feature vector is a node of a semantic graph, and all attribute-specific feature vectors Form a semantic graph, and a semantic graph adjacency matrix is constructed according to prior knowledge of a public data set, the semantic graph and the semantic graph adjacency matrix are input into a semantic graph convolution GCN, the correlation of semantics is learned, and a semantic correlation graph is obtained , semantic correlation graph Through a classifier to perform semantic graph prediction, and using a semantic graph constraint loss Constraint; obtain multiple visual clustering features through a soft assignment strategy , each visual clustering feature corresponds to a region feature of each attribute group, and the multiple visual clustering features form a visual graph, obtain a visual graph adjacency matrix according to similarity and topological distance between clusters, input the visual graph and the visual graph adjacency matrix into a visual graph convolution GCN, learn correlation between regions corresponding to the attribute groups, and obtain a visual correlation graph , the visual correlation graph perform visual graph prediction through a classifier, and use a visual graph constraint loss constraint; A synthetic graph is constructed by combining a semantic correlation graph and a visual correlation graph, the synthetic graph is constructed by embedding the semantic correlation graph and the visual correlation graph through the inter-graph edges between the two graphs, guiding the learning of the other party, and constructing the semantic and regional correlation; All nodes of the semantic graph are connected to all nodes of the visual graph, forming inter-graph edges The inter-graph edges are characterized by their two end nodes; the composite graph is composed of the visual graph , the semantic graph and the inter-graph edges ​ In the construction of the synthetic graph, the semantic association graph is used The visual association graph The inter-graph edges are obtained through fusion operation and full connection layer The semantic association graph The update of the semantic association graph is divided into intra-graph update and inter-graph update, wherein the inter-graph update is guided by the inter-graph edge; the semantic association graph The intra-graph update is performed through a semantic graph convolution Visual association graph The update of the visual association graph is divided into intra-graph update and inter-graph update, the visual association graph undergoes an intra-graph update through a visual graph convolution; The inter-graph updating process is: the semantic association graph or the visual association graph is respectively multiplied by the semantic association graph mapping matrix or the visual association graph mapping matrix , and then is fused with the inter-graph edge , given the semantic association graph weight matrix and the visual association graph weight matrix , the result after the inter-graph edge is fused is respectively multiplied by the semantic association graph weight matrix or the visual association graph weight matrix , and the matrix multiplication operation is performed, so that the inter-graph updating of the visual association graph and the semantic association graph is completed. Inter-graph updating results of the semantic correlation graph and the visual correlation graph Then, the inter-graph edge is obtained after fusion operation, a fully connected layer and the updated inter-graph edge and the semantic embedding graph ; the visual embedding graph , the inter-graph edge and the semantic embedding graph After fusion operation and a fully connected layer, the updated inter-graph edge is obtained ; updated inter-embedding graph edges inter-embedding graph edge prediction by inter-embedding graph edge classifier and with synthetic graph constraint loss constrained The method is trained using a public dataset and used for pedestrian attribute recognition. 2.The multi-view pedestrian attribute recognition method of claim 1, wherein, The inter-graph edges are semantic correlation graphs With visual correlation graphs Obtained through a fusion operation, a fully connected layer. 3.The multi-view pedestrian attribute recognition method of claim 1, wherein, All constraint losses adopt suit_polyloss function, in which a penalty coefficient is introduced to alleviate the problem of unbalanced data in pedestrian attribute recognition ; there is a positive sample rate for each attribute in the data set, if the ratio of the maximum value and the minimum value of the positive sample rate of all attributes in the data set is greater than 100, the square root strategy is adopted, and if it is less than 100, the exponential strategy is adopted; The exponential strategy is: （1）， The square root strategy is: （2）， wherein is the positive sample ratio of attribute j; is the label value of the jth attribute of the ith image.

4. A multi-view pedestrian attribute recognition method, characterized in that, The specific steps of the method are: Firstly, the pedestrian image is preprocessed and feature extracted: After adjusting pedestrian pictures to 256x128, training images are augmented using random horizontal flipping, hue and saturation, rotation, translation, cropping, erasing and adding random Gaussian blur, and ResNet-101 is used to obtain the first i semantic feature map of the picture and visual feature map The first three layers of ResNet-101 share weights, and the fourth layer is divided into two branches and is trained separately without sharing weights; In the visual graph branch, the down-sampling in the fourth layer of ResNet-101 is removed and the channel increase operation in this part is removed; Secondly, the semantic graph learns the semantic correlation: The semantic feature map obtained according to the first step The attribute-specific feature vector of each attribute is obtained by semantic constraint loss constraint, the semantic graph is constructed using the attribute-specific feature vector, the adjacency matrix is constructed using the conditional probability of the attribute in the training set, the graph convolution is performed, then the corresponding attribute probability of each node is obtained, and the training of the semantic graph is constrained using a loss function. In step 2.1, the attribute-specific feature vector obtained is constrained by using a semantic constraint loss: semantic feature maps extracted from ResNet-101 feature vectors obtained by global average pooling wherein is the number of channels; semantic classification prediction results obtained by a semantic classifier composed of a fully connected layer with an output dimension of the number of attributes, batch normalization, and an activation function The semantic classifier is mathematically represented as: （3）， wherein denote the parameters of the fully connected layer, is a sigmoid function, and BN denotes a batch normalization layer. The semantic classifier is used to predict the semantic classification, and a semantic constraint loss function is used to constrain; will be described in detail below. different output dimensions for different attribute-specific feature vectors The number of fully connected layers is consistent with the number of attributes, which is mathematically represented as: （4）， is a parameter of the full connection layer for the th attribute, and the obtained , is the length of the attribute-specific feature vector ; In step 2.2, the semantic graph is convolved: Using all the attribute-specific feature vectors Fusing into As the input of the semantic graph, the semantic graph convolution is performed on the semantic graph to mine the semantic relationship between the pedestrian attributes, which is expressed by the following formula: （5）， wherein, is a learnable adjacency matrix, is the output of semantic graph convolution, i.e., a semantic correlation graph, is an adjacency matrix is a matrix composed of the inverse and square root of the diagonal, is an identity matrix; an adjacency matrix is obtained according to prior knowledge of a training set of pedestrian attributes; semantic map prediction results are obtained through a semantic map classifier composed of global average pooling GAP, batch normalization BN and a sigmoid activation function and are constrained by a semantic map constraint loss, and the formula of the semantic map classifier is as follows: （6）, Thirdly, the visual graph learns the regional correlation: According to the visual feature map extracted from ResNet-101 Construct a visual graph, construct an adjacency matrix using topological relations and similarity, and perform visual graph convolution. The specific steps are: Step 3.1, constructing visual feature map Constructing visual map using soft assignment strategy: To learn the region relationship in the image, the visual feature maps extracted from ResNet-101 Each cluster represents a group of attributes, which corresponds to the same visual cluster feature. In the soft assignment strategy, Each local feature has a weight for each cluster. The contribution of all local features to the same cluster is 1. The visual cluster feature is calculated by weighting and summing the transformed local features with the assigned weights. In this way, the network itself learns the cluster to cope with human pose and camera perspective changes, and the visual cluster feature is , multiple visual cluster features constitute a visual graph, where represents the number of nodes in the visual graph. Different datasets have different numbers of visual graph nodes, represents the length of the feature vector of each node in the visual graph. The calculation formula is as follows: （7）， wherein, is a mapping matrix, indicating normalization using softmax at each position; is a weight matrix; The calculation formula of is as follows: （8）， representing a visual feature map first point assigned to a visual map node confidence of the node, a trainable weight matrix for computing the assigned weight; In step 3.2, the adjacency matrix is constructed according to the topological distance and similarity between clusters to perform graph convolution: The visual graph is updated according to the distance and similarity between the regions corresponding to the attribute groups, thereby learning the context relationship, and when encountering appearance changes and partial body occlusions, the attributes of the regions that are difficult to identify are identified through other regions, the visual graph convolution is intra-graph update, and the formula is as follows: （9）， wherein is the result of the visual graph convolution, i.e. the visual affinity graph, and are trainable weight matrices; and is the adjacency matrix, is obtained from the normalized similarity between clusters, is obtained from the normalized topological distance between clusters; Visual association graph The prediction is made by a visual graph classifier, for Each node of the visual association graph is obtained by the visual graph classifier The output of all nodes of the visual association graph constitutes the visual graph prediction result The visual graph classifier also includes a full connection layer, a batch normalization layer and a sigmoid function, the output dimension of the full connection layer in the visual graph classifier is the number of attributes contained in the attribute group corresponding to the node, and a formula of the visual graph classifier is as follows: （10）， wherein, denote parameters of the fully connected layer of the visual map classifier; The visual graph classifier is constrained by using a visual graph constraint loss, so that each visual cluster feature can learn the regional features of the corresponding attribute group; Fourthly, the synthetic graph is graph-convolved: The semantic correlation graph and the visual correlation graph are constructed into a synthetic graph, which can better learn the visual semantic relationship between attributes, and specifically: In step 4.1, the synthetic graph is constructed by combining the semantic correlation graph and the visual correlation graph: All nodes of the semantic graph are connected to all nodes of the visual graph, forming inter-graph edges The characteristics of the inter-graph edges are generated from the two end nodes, the formula of the inter-graph edges is as follows: （11）， wherein is a visual correlation graph mapping matrix, is a fully connected layer, is a fusion operation, inter-graph edges , is a number of semantic correlation graph nodes, i.e., a number of attributes, is a number of visual correlation graph nodes, is a length of a semantic correlation graph node vector, inter-graph edges is a number of inter-graph edges, a number of semantic graph nodes multiplied by a number of visual graph nodes; a composed graph is composed of a visual correlation graph , a semantic correlation graph , and inter-graph edges ; In step 4.2, the synthetic graph convolution strategy is constructed: In the synthesis graph The semantic graph and the visual graph are divided into intra-graph update and inter-graph update, where the inter-graph update is guided by the inter-graph edge; the intra-graph update of the semantic graph is semantic graph convolution, and the other part is generated by the inter-graph edge combined with the visual graph, which is mapped to dimension by the mapping matrix, then fused with the inter-graph edge, and then calculated with the weight matrix to obtain the weighted sum, which is to embed the visual information into the semantic graph and guide the learning of semantic relationship by the region relationship. The inter-graph update of the semantic graph is represented by the following formula: （12）， wherein is the result of the inter-graph update of the semantic association graph, is the inter-graph edge, is the fusion operation, is the visual association graph mapping matrix, mapping the dimensions of the visual association graph to the same dimensions as the semantic association graph, is the visual association graph weight matrix, information obtained from the inter-graph edges and the visual association graph to be assigned to each node of the semantic association graph, being the sum of the contribution of all inter-graph edges connecting a semantic association graph node and the visual association graph nodes updating this node and being 1. The update formula of the semantic correlation graph is as follows: （13）， in This is the updated semantic embedding graph. For input dimension Output dimension is The fully connected layer For fusion operation, The weight matrix for semantic graph convolution. The inter-graph update formula of the visual correlation graph is as follows: （14）, wherein is the result of the inter-graph update for the visual association graph, is the inter-graph edge, is the fusion operation, is the semantic association graph mapping matrix, is the semantic association graph weight matrix; The update formula of the visual correlation graph is as follows: （15）， in This is the updated visual embedding image. For input dimension Output dimension is The fully connected layer For fusion operation, and This is the weight matrix for the visual graph convolution; After the semantic and visual association graphs complete the inter-graph update operation, the inter-graph edges are updated, which is updated by the inter-graph edges , visual embedding graph and semantic embedding graph through the fully connected layer with input dimension and output dimension , and the inter-graph edge update formula is as follows: （16）， wherein is a fully connected layer with input dimension and output dimension , is a fusion operation, is a visual graph mapping matrix, inter-graph edges resulting from inter-graph edge updates and identical size; In step 4.2, the inter-graph edge is embedded to predict the attribute: The inter-embedding graph edges pass through an inter-embedding graph edge classifier to obtain an inter-embedding graph edge prediction result The inter-embedding graph edge classifier comprises a global maximum pooling, a global average pooling, a batch normalization layer and an activation function, and a formula of the inter-embedding graph edge classifier is as follows: （17）， where GMP is global max pooling and GAP is global average pooling, is batch normalization, is an activation function, using as the final prediction, and with a synthetic map constraint loss constraint; Fifthly, a weighted loss function is used: The losses for guiding the whole network training have four, which are semantic constraint loss , semantic graph constraint loss , visual graph constraint loss and synthetic graph constraint loss , wherein only the first three losses are calculated in the first 15 iterations, and all losses are calculated after the first 15 iterations; all constraint losses adopt the suit_polyloss function; The total loss of the trained network is as follows: （18）， Train only the visual and semantic graph acquisition part in the first 15 iterations, the prediction result of the visual and semantic graph acquisition part is the average result of the semantic graph prediction and the visual graph prediction , to: , evaluate the accuracy of the acquisition part; train the entire network after the first 15 iterations, and the prediction result of the entire network is the inter-embedding graph edge prediction result , with the inter-embedding graph edge prediction result evaluate the accuracy of the entire network. 5.A computer readable storage medium, the storage medium storing a computer program, the computer program being adapted to execute the multi-view pedestrian attribute recognition method of any one of claims 1-4 when loaded by a computer.

Images