A pedestrian re-identification method based on relationship attention transformer

Abstract

The application discloses a pedestrian re-identification method based on a relationship attention transformer, and the relationship attention transformer model of the method mainly comprises a channel attention module and a near neighbor relationship attention module. The channel attention module extracts an effective dimension by utilizing the characteristics of transformer self-attention, so as to achieve the effect of dimension reduction, saves the computing power while guaranteeing the performance, and can preliminarily extract the correlation information between different samples; the near neighbor relationship attention module performs sparse processing on the attention weight of the non-related samples in the near neighbor samples, further strengthens the interaction of the related samples, so that the interference of outliers is reduced when pedestrian feature matching is performed, and the matching precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, in particular to the field of pedestrian re-identification, and more particularly to a pedestrian re-identification method based on relationship attention transformer. BACKGROUND

[0002] With the continuous deepening of urban digital intelligent reform, artificial intelligence technology is applied more and more widely, so the requirement for intelligent technology is also higher and higher. As one of the fields that are concerned, security is also in the forefront of innovation in intelligent technology. Pedestrian re-identification technology is one of the core technologies in the security field, and it has a wide range of uses, such as tracking criminals and finding missing persons, which can reduce the consumption of human, material and financial resources in the city while improving the efficiency of related work. However, pedestrian re-identification faces many challenges, including weather changes, light changes, pedestrian body occlusions, pedestrian clothing changes, and camera imaging differences. Although some breakthroughs have been made, there is still much room for improvement in practical application scenarios. SUMMARY

[0003] In order to overcome the shortcomings of the prior art, the present application provides a pedestrian re-identification method based on relationship attention transformer, which inputs the pedestrian features of a batch of samples, reduces the computational complexity, and deeply mines the relationship information between adjacent samples. The technical solution is as follows:

[0004] The present application provides a pedestrian re-identification method based on relationship attention transformer, which includes the following main steps:

[0005] Step 1, input the sample to the feature extractor, the feature extractor is an HRNet18 model, the model is connected with a global average pooling layer and a linear convolution layer at the end, to obtain a feature matrix , that is, a feature vector of dimensions is extracted from the sample , wherein ; ;

[0006] Step 2, input the feature matrix to the relationship attention transformer model, the feature matrix first enters the channel attention module, and the relationship weight of the corresponding sample is preliminarily obtained by calculating the similarity between two different feature vectors; the feature matrix is projected through 3 linear projections , , , obtain query, key and value matrix, respectively, denoted as ; from the feature matrix Randomly sample samples, denoted as , Through , Get ; the original mapped to dimensional space by the following formula:

[0007] ;

[0008] Step 3, take any sample pair , ), where , , calculate the similarity, so as to obtain the affinity matrix , the specific formula is as follows:

[0009] ;

[0010] The affinity matrix obtained in this way, the calculation complexity is reduced from to , ;

[0011] Step 4, input the affinity matrix into the relationship attention module, convert the similarity into attention weight, and then use threshold filtering to integrate the feature expression. The relationship attention module first takes the top values from each row of the affinity matrix according to the numerical size to calculate a top-k neighbor association matrix , the specific formula is as follows:

[0012] ;

[0013] The complete neighbor association matrix formula is as follows:

[0014] ;

[0015] Where is Hadamard product;

[0016] Step 5, do softmax operation on the neighbor association matrix , get neighbor relationship attention weight , the formula is as follows:

[0017] ;

[0018] The complete feature expression can be represented as:

[0019] ;

[0020] Step 6, train the transformer model to realize pedestrian re-identification.

[0021] Compared with the prior art, one of the above technical solutions has the following beneficial effects: the relationship attention transformer model mainly includes a channel attention module and a neighbor relationship attention module. The channel attention module extracts effective dimensions by using the characteristics of transformer self-attention to achieve dimension reduction, saves computing power while ensuring performance, and can also preliminarily extract the correlation information between different samples; the neighbor relationship attention module sparsifies the attention weight of the non-related samples in the neighbor samples, further strengthens the interaction of the related samples, thereby reducing the interference of outliers when performing pedestrian feature matching, and improving the matching accuracy. The advantages mainly include:

[0022] 1. The channel attention module reduces the calculation amount while ensuring the performance, and preliminarily extracts the relationship information between different samples, which is beneficial to the deployment of edge devices.

[0023] 2. The neighbor relationship attention module excavates the correlation information of the neighbor samples, reduces the interference of non-related samples, and improves the pedestrian feature matching accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 A relationship attention transformer model diagram is provided for the embodiments of the present disclosure. DETAILED DESCRIPTION

[0025] In order to clarify the technical solutions and working principles of the present application, the embodiments of the present disclosure will be further described in detail below in combination with the drawings. All the optional technical solutions described above can be combined to form optional embodiments of the present disclosure, which will not be described one by one here.

[0026] The terms "step 1", "step 2", "step 3", and the like in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those described herein.

[0027] The embodiments of the present disclosure provide a pedestrian re-identification method based on relationship attention transformer, which includes the following main steps:

[0028] Step 1, input samples Input into the feature extractor, the feature extractor is the HRNet18 model, the model end connects a global average pooling layer and a linear convolution layer, obtains a feature matrix , that is, the feature vector of dimension is extracted from the sample , where .

[0029] Step 2, input the feature matrix into the relationship attention transformer model, the process of the model is shown in Figure 1 . The feature matrix first enters the channel attention module, because of the self-attention characteristics of the transformer, it is easy to convert the dimension in the feature space, and because the input is the feature vector of multiple samples, the similarity between two different feature vectors can be calculated to obtain the relationship weight of the corresponding sample; The feature matrix is respectively projected by 3 linear projections , , , to obtain the query, key and value matrices, denoted as . Randomly sample samples from the feature matrix , denoted as , , , , ; map the original to dimensional space by the following formula:

[0030] ;

[0031] Step 3, take any sample pair , , where , , calculate the similarity between the two, so as to obtain the affinity matrix , the specific formula is as follows:

[0032] ;

[0033] The affinity matrix obtained in this way has a computational complexity of , , . Since the input is the feature vector of samples, the affinity matrix Each of them has contained the relationship information between two samples, but this information needs to be further refined and distinguished.

[0034] Step 4, the affinity matrix is input into the relationship attention module, which is different from directly doing softmax on to convert the similarity into attention weight, and then using threshold filtering to integrate the feature expression. The relationship attention module first calculates a top-k neighbor correlation matrix from the affinity matrix , taking the top values in each row according to the numerical value. The specific formula is as follows:

[0035] ;

[0036] The complete neighbor correlation matrix formula is as follows:

[0037] ;

[0038] Where is the Hadamard product. From the above formula, it can be seen that only when and are the top neighbors of each other, the value of the neighbor correlation matrix is 1, at which time the attention is concentrated in the neighbor sample features, and the attention to related samples is increased.

[0039] Step 5, do softmax operation on the neighbor correlation matrix to get the neighbor relationship attention weight , the formula is as follows:

[0040] ;

[0041] After the above operation, most of the attention values are set to 0, and the relationship information is limited to the related neighbor sample features, so that the relationship attention transformer model is more concentrated and robust in aggregating sample features. The complete feature expression can be expressed as:

[0042] ;

[0043] Step 6, train the transformer model, extract the effective features of the pedestrian images in the search set and the base library set through the relationship attention based transformer model, and calculate the cosine similarity between the features. Set the similarity threshold to judge whether the features match, so as to realize pedestrian re-identification.

[0044] Preferably, in step 6, considering that the relationship attention transformer model has a feature normalization process, a cosine distance is adopted to cooperate with a cross-entropy as a loss function to train the whole system.

[0045] The application is described above with reference to the drawings; obviously, the specific implementation of the application is not limited to the above-described manner, and various non-essential improvements to the method concept and technical solutions of the application, or direct application of the above-described concept and technical solutions of the application to other occasions without modification or equivalent replacement, are all within the protection scope of the application.

Claims

1. A pedestrian re-identification method based on relation attention transformer, characterized in that, The method comprises the following main steps: Step 1, Sample The input is fed into a feature extractor, which is an HRNet18 model. The model ends with a global average pooling layer and a linear convolutional layer to obtain a... Feature matrix That is, from the sample Extract 3D eigenvectors ,in ; Step 2, the feature matrix is input into the relational attention transformer model, the feature matrix first enters the channel attention module, and the relational weight of the corresponding sample is preliminarily obtained by calculating the similarity between two different feature vectors; the feature matrix is respectively projected by three linear projections , , to obtain the query, key and value matrices, respectively denoted as ; a plurality of samples are randomly sampled from the feature matrix , denoted as , , the similarity between the query and key matrices is calculated , to obtain ; the original is mapped to a dimensional space by the following formula: ； Step 3, take any sample pair (x, y) , ), where , , calculate the similarity of the two, so as to obtain the affinity matrix , the specific formula is as follows: ； The affinity matrix obtained in this way has a computational complexity reduced from O(n3) to O(n2) ; Step 4, input the affinity matrix into the relational attention module, instead of directly doing softmax on the affinity matrix, convert the similarity into attention weight, then use threshold filtering to integrate feature expression, the relational attention module first takes the top-k values from each row of the affinity matrix according to the numerical size to calculate a top-k neighbor association matrix , the specific formula is as follows: ​ ； The complete neighbor association matrix formula is shown as follows: ； wherein is a Hadamard product; Step 5, perform a near-neighbor correlation matrix Perform a softmax operation to obtain the nearest neighbor attention weights. The formula is as follows: ； The complete feature expression can be expressed as: ； Step 6, train the transformer model to realize pedestrian re-identification.

2. The relation attention transformer-based pedestrian re-identification method of claim 1, wherein, Step 6 uses cosine distance combined with cross entropy as the loss function to train the entire system.

3. The relation attention transformer-based pedestrian re-identification method of claim 2, wherein, Step 6 extracts effective features of pedestrian images in the search set and the base library set through the transformer model based on relationship attention, calculates the cosine similarity between the features, sets a similarity threshold to judge whether the features match, thereby realizing pedestrian re-identification.

Images