A crowded pedestrian re-identification method based on relational graph attention network

Abstract

The present invention belongs to the field of artificial intelligence and computer vision technology, and specifically relates to a method for re-identifying crowded pedestrians based on a graph attention network. The method of the present invention first uses a posture feature extraction module to extract appearance features, uses OpenPose to obtain a set of key points of a person, and uses multiple linear layers to convert it into a high-dimensional vector, and then uses a graph attention network to obtain shape features in the image, thereby excluding information that interferes with the person from the shape features, and then uses the proposed shape-guided feature enhancement module to mine appearance features based on shape features through a two-stage cross-attention process, thereby obtaining the discriminative features of the key target person and re-identifying the person's identity. The method of the present invention solves the problem of re-identifying crowded pedestrians and improves the recognition accuracy of the model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and specifically relates to a crowded pedestrian re-identification method based on a relational graph attention network. Background Art

[0002] With the rapid development of social economy, public safety has become increasingly important. Nowadays, surveillance equipment has spread to every corner of daily life, but the application of surveillance information is still limited to a relatively basic level. In an emergency, manually reviewing surveillance footage to identify and retrieve specific people is a time-consuming and labor-intensive task, especially in complex shooting environments, and the accuracy of human eye resolution is relatively low.

[0003] Therefore, the industry began to actively explore the person re-identification (Re-ID) technology, which has become an important part of the intelligent security field. Person re-identification (Re-ID) is a technology that aims to identify specific pedestrians from a large number of pedestrian images captured by different cameras to meet various search needs. Due to its wide range of practical applications, such as criminal investigation, danger warning, unmanned supermarket management, lost person rescue and many other fields, pedestrian re-identification has become a hot topic in the field of scientific research and has received widespread attention. However, despite the continuous progress of pedestrian re-identification technology, the difficulty of this task is relatively low because it is assumed that the target person always appears alone in a scene, and the person is significantly different from the environment. This also leads to the current pedestrian re-identification methods are often only applicable to person recognition in scenes with a small number of people. However, in reality, it is often necessary to find pedestrians in crowded scenes, such as shopping malls, intersections and schools, so this task is out of touch with real application scenarios.

[0004] For example, in the paper "DiP: Learning Discriminative Implicit Parts for Person Re-Identification", the authors proposed a method to learn discriminative implicit parts (Dip), which decouples implicit body parts from explicit body parts and can learn to extract features that help distinguish identities. In addition, the authors introduced a novel implicit position that provides a geometric interpretation for each Dip. Finally, the feature representation of Dip is further improved by introducing additional Dip weights to handle invisible or occluded situations. This model solves the problem of conventional pedestrian re-identification well and brings certain performance improvements. However, in subsequent embodiments, it was found that its effect was not obvious because it was not suitable for crowded pedestrian re-identification.

[0005] In actual application scenarios, users usually need to identify crowded scenes, such as shopping malls, intersections or schools. In these cases, a large number of people overlap closely. If the existing model is used directly for identification, it is impossible to accurately identify people in crowded scenes due to its lack of ability to eliminate strong similarity interference. This also shows that although the model continues to improve in the field of scientific research, it still performs poorly in practical applications. Therefore, in response to this problem, the present invention abandons the assumption that "pedestrians appear alone" and proposes a new task, namely Crowded Person Re-id. And contributes a dataset named Market-1501-Crowded, which is processed from the commonly used dataset Market-1501 dataset. It contains a total of 32,688 pictures. There are a total of 1,501 people, of which 751 pedestrians are labeled for the training set and 750 pedestrians are labeled for the test set. Some pictures of this dataset are shown as follows. Figure 1 shown.

[0006] In order to solve this task, the present invention proposes a crowded pedestrian re-identification method based on relational graph attention network. Summary of the invention

[0007] In view of the technical problems in the above-mentioned prior art, the present invention proposes a crowded pedestrian re-identification method based on a relational graph attention network.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions:

[0009] A crowded pedestrian re-identification method based on a relational graph attention network comprises the following steps: step 1: preprocessing P*K image data; step 2: inputting the preprocessed P*K visible light images obtained in step 1 into a feature extraction network; step 3: using a ResNet50 network, mining information-rich image features from the shallow features of the image input in step 2; step 4: for the image input in step 1, using an OpenPose model, obtaining a key point set for each image; step 5: using multiple linear layers, converting the key point set into a high-dimensional vector; step 6: using a relational graph attention network, converting the high-dimensional vector into a significant posture feature; step 7: inputting the image features of step 3 and the posture features of step 6 into a shape-guided feature enhancement module to obtain the final image features, and calculating the identity loss and WRT loss; step 8: using the obtained identity loss and WRT loss, calculating the total loss, and training the image features into a network with good performance; step 9: if the specified number of training rounds is reached, then proceed to step 10, otherwise continue to complete the training and return to step 1; step 10: end.

[0010] Further, as a preferred technical solution of the present invention, in the step 1, the images are all from the crowded pedestrian re-identification dataset Market-1501-Crowded; in Market-1501-Crowded, various situations of crowd crowding are fully considered. For the original Market-1501 dataset, images of a part of the crowd are randomly selected, and in each image, a fine-grained image of a person in other images is randomly selected and placed in the picture, and randomly reduced and placed behind the person, or randomly enlarged and placed in front of the person, thereby simulating the complex relationships in crowded scenes.

[0011] Further, as a preferred technical solution of the present invention, the steps of making the crowded pedestrian re-identification dataset Market-1501-Crowded are as follows: Step 1-1: Use the deeplabv3_resnet101 model to identify all the pictures of the original dataset, and only retain the character part, and set the other parts to 0; Step 1-2: Transfer the pictures in step 1-1 whose character part occupies 55% of the picture to another folder; Step 1-3: Manually delete the pictures with poor recognition effect in step 1-2; Step 1-4: Take the pictures in step 1-3 as the benchmark, and randomly merge the characters in other pictures; Step 1-5: Manually delete the pictures with poor merging effect in step 1-4; Step 1-6: Replace the original dataset pictures with the synthesized pictures with better effect.

[0012] Further, as a preferred technical solution of the present invention, in order to simulate the diversity of crowded people, in the to-be-synthesized pictures, 50% of the newly added people are in front of the original people, and the other 50% of the newly added people are behind the original people; the person in the to-be-synthesized picture is recorded as p1, the complete picture is recorded as b1, and the picture of the newly added person is recorded as p2;

[0013] When the newly added character image is before the original character, the synthesis steps are as follows:

[0014] Step 1-4-1: randomly enlarge p2 by 10% to 30%, and place the midpoint of p2 10% to 30% below the midpoint of b1, with a deviation of 40% to 70% to the left and right, so as to achieve the effect of larger near and smaller far without completely covering p1;

[0015] When the newly added character image is behind the original character, the synthesis steps are as follows:

[0016] Step 1-4-2: Ignore step 1-4-1, randomly reduce p2 by 10% to 30%, and place the midpoint of p2 10% to 30% above the midpoint of b1, with a deviation of 30% to 60% to the left and right, so as to achieve the effect of larger objects near and smaller objects far away and not being completely covered by p1;

[0017] Step 1-4-3: Overlay p1 on the image output in step 1-4-2 to achieve the effect that p1 is before p2;

[0018] Since p1 and p2 only retain the image of the person, the area around the person is black because its value is set to 0. In order not to dye the entire image black, it is necessary to ignore the value of p1 and p2 that is 0 when overlaying. Since the model deeplabv3_resnet101 will also recognize some dark pixels as humans, it is necessary to ignore the pixels in p1 and p2 whose RGB value sum is less than 50 when overlaying. In fact, since the model deeplabv3_resnet101 cannot accurately identify the boundaries of humans, it often contains a lot of the environment around the person. Therefore, when overlaying, it is necessary to ignore the pixel in p1 and p2 that satisfies the pixel or the sum of the RGB values ​​of the two pixels to its left and right is less than 50. Since the midpoint needs to be offset, b1 needs to be placed on an empty drawing board with a length and width that are three times the size of b1, and b1 is placed in the middle. Finally, only the area where b1 is located on the drawing board is taken.

[0019] Further, as a preferred technical solution of the present invention, the step of extracting preliminary features of the image in step 2 of the training process is as follows:

[0020] Step 2-1: Input the visible light image into the feature extraction network to extract its preliminary features;

[0021] Step 2-2: Output features.

[0022] In step 2, the present invention uses the first layer of the residual network to extract features and fully perceive the shallow information of the image.

[0023] Further, as a preferred technical solution of the present invention, the step of extracting image features in step 3 of the training process is as follows:

[0024] Step 3-1: Input the features extracted in step 2 into the network composed of the last four convolutional layers of ResNet50 to obtain information-rich image features;

[0025] Step 3-2: Perform MAX-pooling on the features in step 3-1 and proceed to step 3-3;

[0026] Step 3-3: Output the pooled feature F of the image.

[0027] In step 3, the present invention uses a residual network to extract features. Among them, the ResNet50 network is a typical network in the residual network, which includes 50 two-dimensional convolution operations. The present invention uses the ResNet50 network to increase the network depth and improve the feature expression ability.

[0028] Further, as a preferred technical solution of the present invention, the OpenPose model is used in step 4. For a given image I, the model will output m nodes corresponding to the image. And think of it as a set of body key points of the person in each picture.

[0029] Further, as a preferred technical solution of the present invention, in step 5, in order to use the node set output in step 4 in the deep neural network, it is necessary to convert it into a high-dimensional feature vector The present invention uses a re-refinement network composed of several fully connected layers to convert these nodes into high-dimensional feature vectors.

[0030] Further, as a preferred technical solution of the present invention, in step 6, in order to represent the body shape, the present invention needs to capture the local and global relationships between nodes. To achieve this, the present invention uses a relational graph attention network (R-GAT), which is different from GCN in that it incorporates the attention mechanism into the graph operation to consider the importance across nodes, and compared with GAT, the relational graph attention network also amplifies the different levels of relationships between bones, which is crucial for capturing discriminative shapes when certain body parts are invisible due to occlusion or extreme viewing angles; the input of the relational graph attention network is a graph consisting of the high-dimensional feature vector j' obtained in step 5 and k relationship representations, the high-dimensional feature vector j' has a total of m node information, and the k relationships are understood as k edges; use Represents the input eigenvector matrix; W (r) Represents the relationship matrix, and the final feature matrix is ​​represented by H (r) , and its calculation formula is as follows:

[0031] H (r) =J'×W (r) (1)

[0032] and W (r) ∈R j'×j' are all learnable parameters; for H (r) Each node The adjacent index set is recorded as It is updated after the graph attention network based on the weighted sum of its neighboring nodes and the influence of all associated edges. Its calculation formula is as follows:

[0033]

[0034] in, is the attention score of the i-th and j-th nodes, σ is the Sigmoid activation function, and SM represents the Softmax activation function; finally, the graph is aggregated by calculating the average value of the node representation to obtain the shape embedding s, which is calculated as shown in formula (3):

[0035]

[0036] Further, as a preferred technical solution of the present invention, the shape-guided feature enhancement module in step 7 is operated as follows: in the first stage, the shape feature s output by step 6 is used as a query, and the image pooling feature F output by step 3 is used as a key and value, and its calculation formula is as shown in formula (4):

[0037] Q=W q (s),K=W k (F),V=W v (F) (4)

[0038] Among them, W q , W k With W v are the attention matrices of query, key and value respectively; the correlation score between Q and K is used to search for features directly related to the shape in V, and then perform feature fusion with the shape features through regularization and batch normalization. The calculation formula is as follows:

[0039] f fused =W n2 (BN(Norm(QK T )V))+s (5)

[0040] Among them, f fused represents the appearance features mixed with shape features, Norm represents regularization, BN represents batch normalization, and W n2 Represents the coefficient matrix of the deep neural network; considering f fused It contains shape discrimination information and appearance discrimination information related to the shape, and effectively serves as a query in the second-stage attention mechanism, which helps to obtain appearance features directly and indirectly related to the body shape; as the query feature of the second-stage attention, f fused It plays a key role in determining whether identity features directly and indirectly related to shape can be effectively extracted from appearance features; in order to ensure that the query f fused In order to improve the identifiability, the present invention also adopts cross entropy loss and WRT loss to jointly constrain GeM(f fused), where GeM represents the GeM pooling layer, and its calculation formula is as shown in equations (6) and (7):

[0041]

[0042]

[0043] in, is the result of the cross entropy loss in the first stage, batchnum is the number of samples in a batch, FC 1 is a fully connected layer, whose output will be used as the predicted probability, q i Is the image really labeled as i? Specifically, it is 1, not 0; is the result of the first WRT loss. For image i, j is a positive example, i.e., a person with the same identity, k is a negative example, and d i,j is the Euclidean distance between image i and image j;

[0044] In the second stage, the output feature f of the attention mechanism in the first stage is fused As a query, to emphasize the appearance features directly and indirectly related to the body shape, the image pooling feature F output in step 3 is still used as the key and value, and the same cross-attention calculation and feature fusion operation as in the first stage are performed to obtain a more deeply fused feature f related to the key pedestrian. final , and also use cross entropy loss and WRT loss to jointly constrain GeM(f final ), and its calculation formula is as follows (8) and (9):

[0045]

[0046]

[0047] The total network loss can be expressed as shown in formula (10), which will be used as the final loss function to constrain the model training process;

[0048]

[0049] Among them, λ is a hyperparameter and is set separately in the experiment.

[0050] The method for re-identifying crowded pedestrians based on a relational graph attention network described in the present invention has the following technical effects compared with the prior art by using the above technical solution: The present invention proposes a method based on a relational graph attention network for the problem of pedestrian re-identification in crowded scenes, so that the Rank-1 accuracy rate reaches 68.58%, reaching the current leading level. The method first uses OpenPose to obtain a set of key points of the character, and then combines the relational graph attention network to extract the shape features in the picture, thereby eliminating information that interferes with the character. Then, using the shape-guided feature enhancement module, in a two-stage cross-attention process, the appearance features based on the shape features are mined, thereby obtaining discriminative target person features and realizing the re-identification of the person's identity. Finally, the features are combined with the WRT loss function and the joint id loss function for network training, achieving a recognition accuracy rate of 68.58%. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 These are some pedestrian images in the Market-1501-Crowed dataset;

[0052] Figure 2 It is a schematic diagram of the network framework of crowded pedestrian re-identification based on the relationship graph attention network of the present invention;

[0053] Figure 3 This is a flow chart of the training phase of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

[0055] The present invention proposes a crowded pedestrian re-identification method based on a relational graph attention network to solve the new task of crowded pedestrian re-identification. The network framework diagram of the crowded pedestrian re-identification based on the relational graph attention network of the present invention is as follows: Figure 2 shown.

[0056] The training flow chart of the network proposed by the present invention is as follows Figure 3 As shown. This training process uses mini-batch training. In each batch, P pedestrians are randomly selected, and K images are randomly selected for each of these pedestrians. There are a total of P*K pedestrian images. Next, we will take an image as an example to introduce the training process, as follows:

[0057] Step 1: Preprocess P*K image data and proceed to step 2.

[0058] Step 2: Input the preprocessed P*K visible light images obtained in step 1 into the feature extraction network and proceed to step 3;

[0059] Step 3: Use the ResNet50 network to mine the image features rich in information from the shallow features of the image input in step 2, and proceed to step 4;

[0060] Step 4: For the image input in step 1, use the OpenPose model to obtain the key point set of each image and proceed to step 5;

[0061] Step 5: Use multiple linear layers to transform the key point set into a high-dimensional vector and proceed to step 6;

[0062] Step 6: Use the graph attention network to transform the high-dimensional vector into a significant posture feature and proceed to step 7;

[0063] Step 7: Input the image features of step 3 and the posture features of step 6 into the shape-guided feature enhancement module to obtain the final image features, calculate the identity loss and WRT loss, and proceed to step 8;

[0064] Step 8: Use the identity loss and WRT loss obtained in step 8 to calculate the total loss, train the image features to produce a network with good performance, and proceed to step 9;

[0065] Step 9: If the specified number of training rounds is reached, proceed to step 10, otherwise continue training and return to step 1;

[0066] Step 10: End.

[0067] The images in step 1 are all from the Market-1501-Crowded dataset for crowded pedestrian re-identification proposed in the present invention. Visible light images are composed of three primary colors: red, green, and blue, so the image data contains three channels, each channel corresponding to a primary color. In Market-1501-Crowded, various situations of crowding are fully considered. For the original Market-1501 dataset, images of a part of the crowd are randomly selected, and in each image, a fine-grained image of a person in other images is randomly selected and placed in the picture, and randomly reduced and placed behind the person, or randomly enlarged and placed in front of the person, thereby simulating the complex relationships in crowded scenes.

[0068] The steps of preparing the Market-1501-Crowded data set in step 1 of the training process of the present invention are as follows:

[0069] Step 1-1: Use the deeplabv3_resnet101 model to identify all the pictures in the original dataset, and only keep the characters, and set the other parts to 0;

[0070] Step 1-2: Transfer the pictures in step 1-1 where the characters occupy 55% of the pictures to another folder;

[0071] Step 1-3: Manually delete the images with poor recognition effect in stage 2;

[0072] Step 1-4: Take the pictures in step 1-3 and randomly merge the characters in other pictures with each picture as the benchmark;

[0073] Step 1-5: Manually delete the images with poor merging effect in stage 4;

[0074] Step 1-6: Use the synthesized pictures with better effects to replace the original data set pictures.

[0075] In order to ensure the final synthesized image has a good effect in step 1-2, only the images where the person part occupies 55% of the image are retained, because such people often have a good initial recognition effect, and the initial recognition effect will greatly affect the authenticity of the final image. At the same time, the images where the person part occupies 55% of the image account for 70% of all images, which is also to reflect the randomness of the image modification, thereby enhancing the final robustness of the model, so that the model can finally recognize single-person images and multi-person images.

[0076] In order to simulate the diversity of crowded people in steps 1-4, 50% of the newly added characters are in front of the original characters in the synthesized pictures, and the other 50% of the newly added characters are behind the original characters. The present invention records the characters in the synthesized pictures as p1, the complete pictures as b1, and the newly added characters as p2.

[0077] In order to ensure the authenticity of the synthesis, if the newly added character image is before the original character, the synthesis steps of the present invention are as follows:

[0078] Step 1-4-1: p2 is randomly enlarged by 10% to 30%, and the midpoint of p2 is placed 10 to 30% below the midpoint of b1, with a deviation of 40 to 70% to the left and right, so as to achieve the effect of larger near and smaller far away without completely covering p1.

[0079] If the newly added character image is after the original character, the synthesis steps of the present invention are as follows:

[0080] Step 1-4-2: Ignore step 1-4-1, randomly reduce p2 by 10% to 30%, and place the midpoint of p2 10 to 30% above the midpoint of b1, with a deviation of 30 to 60% to the left and right, to achieve the effect of larger objects near and smaller objects far away, and not being completely covered by p1.

[0081] Step 1-4-3: Overlay p1 on the image output in step 1-4-2 to achieve the effect that p1 is before p2.

[0082] It is worth mentioning that since p1 and p2 only retain the image of the person, the area around the person is black because its value is set to 0. In order not to dye the entire image black, it is necessary to ignore the value of p1 and p2 that is 0 when overlaying. In fact, since the model deeplabv3_resnet101 will also recognize some dark pixels as humans, it is necessary to ignore the pixels in p1 and p2 whose RGB value sum is less than 50 when overlaying. In fact, since the model deeplabv3_resnet101 cannot accurately identify the boundaries of humans, it often contains a lot of the environment around the person. Therefore, when overlaying, it is necessary to ignore the pixel in p1 and p2 that satisfies the pixel or the sum of the RGB values ​​of the two pixels to its left and right is less than 50. Since the midpoint needs to be offset, b1 needs to be placed on an empty drawing board with a length and width that are three times the size of b1, and b1 is placed in the middle. Finally, only the area where b1 is located on the drawing board is taken.

[0083] The steps of extracting preliminary features of the image in step 2 of the training process of the present invention are as follows:

[0084] Step 2-1: Input the visible light image into the feature extraction network to extract its preliminary features;

[0085] Step 2-2: Output features.

[0086] In step 2, the present invention uses the first layer of the residual network to extract features and fully perceive the shallow information of the image.

[0087] The steps of extracting image features in step 3 of the training process of the present invention are as follows:

[0088] Step 3-1: Input the features extracted in step 2 into the network composed of the last four convolutional layers of ResNet50 to obtain information-rich image features;

[0089] Step 3-2: Perform MAX-pooling on the features in step 3-1 and proceed to step 3-3;

[0090] Step 3-3: Output the pooled feature F of the image.

[0091] In step 3, the present invention uses a residual network to extract features. Among them, the ResNet50 network is a typical network in the residual network, which includes 50 two-dimensional convolution operations. The present invention uses the ResNet50 network to increase the network depth and improve the feature expression ability.

[0092] In step 4, the present invention uses a relational graph attention network to extract body shape information from the skeleton-based posture, and its overall structure is as follows: Figure 2 As shown. Considering that there is a lot of physical information of people around in crowded pedestrian re-identification, if too much attention is paid to semantic information, the model will fall into the interference of valid information and invalid information. In order to make the model effectively focus on the identity characteristics of the target person, it is necessary to first make the model focus on the overall information of the target person. Therefore, separating valid people from invalid people naturally becomes the primary method of identifying people. Specifically, using the OpenPose model, for a given image I, the model will output m nodes corresponding to the image And think of it as a set of body key points of the person in each picture.

[0093] In step 5, in order to utilize the node set output in step 4 in the deep neural network, the present invention needs to convert it into a high-dimensional feature vector Specifically, the present invention uses a re-refinement network composed of several fully connected layers to convert these nodes into high-dimensional feature vectors.

[0094] In step 6, in order to represent the body shape, the present invention needs to capture the local and global relations between nodes. To achieve this, the present invention uses a relational graph attention network (R-GAT), which differs from GCN in that it incorporates the attention mechanism into the graph operation to consider the importance across nodes, and compared to GAT, the relational graph attention network also amplifies the different levels of relations between skeletons, which is crucial for capturing discriminative shapes when some body parts are invisible due to occlusion or extreme viewing angles.

[0095] The input of the relationship graph attention network is a graph consisting of the high-dimensional feature vector j' obtained in step 5 and k relationship representations. The high-dimensional feature vector j' has m node information, and the k relationships can be understood as k edges. Represents the input eigenvector matrix. (r) represents the relationship matrix, then the final feature matrix can be expressed as H (r) , and its calculation formula is as follows:

[0096] H (r) =J'×W (r) (1)

[0097] and W (r) ∈R j'×j' are all learnable parameters. (r) Each node The adjacent index set is recorded as It is updated after the graph attention network based on the weighted sum of its neighboring nodes and the influence of all associated edges. Its calculation formula is as follows:

[0098]

[0099] in, is the attention score of the i-th and j-th nodes, σ is the Sigmoid activation function, and SM represents the Softmax activation function. Finally, the graph is aggregated by calculating the average value of the node representation to obtain the shape embedding s, which is calculated as shown in formula (3):

[0100]

[0101] The present invention proposes a shape-guided feature enhancement module in step 7. In addition to shape features, the appearance of pedestrians also includes important identity clues such as clothing style and hairstyle, which are closely related to body shape. When the shape information is combined with the appearance information, the information of people around can be effectively excluded.

[0102] Therefore, the present invention aims to mine the appearance clues mentioned above and suppress features that are not related to identity, such as background and people around. Since the appearance features directly associated with shape features are limited, it is necessary to first mine the appearance features directly related to the shape, and then use these appearance features to further explore the appearance features indirectly related to body shape. Therefore, the present invention designs a shape-guided feature enhancement module, which includes a two-stage attention mechanism to systematically extract appearance features related to body shape, both directly and indirectly extracting appearance features, so as to obtain more comprehensive image information. The structure of this module is as follows: Figure 2 shown.

[0103] The module operates as follows: In the first stage, the shape feature s output by step 6 is used as the query, and the image pooling feature F output by step 3-3 is used as the key and value. The calculation formula is as follows:

[0104] Q=W q (s),K=W k (F),V=W v (F) (4)

[0105] Among them, W q , Wk With W v are the attention matrices of query, key, and value, respectively. The correlation score between Q and K is used to search for features directly related to the shape in V, and then perform feature fusion with the shape features through regularization and batch normalization. Its calculation formula is as follows:

[0106] f fused =W n2 (BN(Norm(QK T )V))+s (5)

[0107] Among them, f fused represents the appearance features mixed with shape features, Norm represents regularization, BN represents batch normalization, and W n2 represents the coefficient matrix of the deep neural network. Considering f fused It contains shape information and appearance information related to the shape, which can be used as a query in the second-stage attention mechanism to help obtain appearance features directly and indirectly related to the body shape. As the query feature of the second-stage attention, f fused It plays a key role in determining whether identity features directly and indirectly related to shape can be effectively extracted from appearance features. In order to ensure that the query f fused In order to improve the identifiability, the present invention also adopts cross entropy loss and WRT loss to jointly constrain GeM(f fused ), where GeM represents the GeM pooling layer. Its calculation formula is as shown in equations (6) and (7):

[0108]

[0109]

[0110] in, is the result of the cross entropy loss in the first stage, batchnum is the number of samples in a batch, FC 1 is a fully connected layer, whose output will be used as the predicted probability, q i Is the real label of the image i? Specifically, it is 1, not 0. is the result of the first WRT loss. For image i, j is a positive example, i.e., a person with the same identity, k is a negative example, and d i,j is the Euclidean distance between image i and image j.

[0111] In the second stage, the present invention converts the output feature f of the first stage attention mechanism into fusedAs a query, to emphasize the appearance features directly and indirectly related to the body shape, the image pooling feature F output in step 3 is still used as the key and value, and the same cross-attention calculation and feature fusion operation as in the first stage are performed to obtain a more deeply fused feature f related to the key pedestrian. final , and also use cross entropy loss and WRT loss to jointly constrain GeM(f final ). Its calculation formula is as follows (8) and (9):

[0112]

[0113]

[0114] The total network loss can be expressed as shown in formula (10), which will be used as the final loss function to constrain the model training process.

[0115]

[0116] Among them, λ is a hyperparameter and is set separately in the experiment.

[0117] The test process of the present invention is:

[0118] Step a: Input the query set and gallery set, and proceed to step b;

[0119] Step b: Using the model obtained during the training process, perform feature extraction on all pedestrian images in the query set and gallery set input in step a, and proceed to step c;

[0120] Step c: Calculate the similarity between the query set features and the gallery set features, and proceed to step d;

[0121] Step d: According to the similarity, the matching result corresponding to each pedestrian image in the query set is obtained, and then the process goes to step e;

[0122] Step e: End.

[0123] The query set in step a of the test process represents a set of pedestrian images to be queried, and the gallery set represents a set of pedestrian images to be matched by the query set.

[0124] Embodiment 1:

[0125] The existing Market-1501-Crowed dataset needs to be used to accurately identify people. To solve this problem, the implementation steps of the present invention are as follows:

[0126] First, follow the scheme below as the model setup.

[0127] In this embodiment, mini-batch gradient descent is used to update the model parameters of the present invention. The ResNet50 network is selected as the basic skeleton, and the parameters of ResNet50 are pre-trained on ImageNet. In the first 10 epochs, a warm-up learning rate strategy is adopted, and then the learning rate is adjusted to 0.01 and 0.001 at the 20th and 50th epochs, respectively, and a total of 80 rounds of training are performed. In the test phase, pedestrian features processed by batch normalization (BN) layer are used for retrieval. In this experiment, the deep learning framework pytorch1.8 is used, and an NVIDIA3090 graphics card is used to accelerate the training process.

[0128] Then, follow the steps below to train the model.

[0129] Step 1: Preprocess P*K image data and proceed to step 2;

[0130] Step 2: Input the preprocessed P*K visible light images obtained in step 1 into the feature extraction network and proceed to step 3;

[0131] Step 3: Use the ResNet50 network to mine the image features rich in information from the shallow features of the image input in step 2, and proceed to step 4;

[0132] Step 4: For the image input in step 1, use the OpenPose model to obtain the key point set of each image and proceed to step 5;

[0133] Step 5: Use multiple linear layers to transform the key point set into a high-dimensional vector and proceed to step 6;

[0134] Step 6: Use the graph attention network to transform the high-dimensional vector into a significant posture feature and proceed to step 7;

[0135] Step 7: Input the image features of step 3 and the posture features of step 6 into the shape-guided feature enhancement module to obtain the final image features, calculate the identity loss and WRT loss, and proceed to step 8;

[0136] Step 8: Use the identity loss and WRT loss obtained in step 8 to calculate the total loss, train the image features to produce a network with good performance, and proceed to step 9;

[0137] Step 9: If the specified number of training rounds is reached, proceed to step 10, otherwise continue training and return to step 1;

[0138] Step 10: End.

[0139] This example uses the Market-1501-Crowded dataset to complete the crowded pedestrian re-identification task. The performance of other methods is reproduced and the comparison results are shown in Table 1.

[0140] Table 1 Performance comparison of the proposed method and other methods on the Market-1501-Crowded dataset

[0141]

[0142] In experiments on the Market-1501-Crowded dataset, the DiP method showed the best results except for the model of the present invention. The authors of this method proposed a method to learn discriminative implicit parts (Dips), which are decoupled from explicit body parts and can learn to extract features that help distinguish identities. In addition, the authors proposed a novel implicit position that provides a geometric interpretation for each Dip. Finally, additional Dip weights were introduced to handle invisible or occluded situations and further improve the feature representation of Dip. This model solves the problem of conventional pedestrian re-identification well and brings certain improvements, but its effect is not obvious in crowded pedestrian re-identification.

[0143] Compared with this model, the Rank-1 and mAP of the proposed model are improved by 11.32% and 13.27% respectively, which shows that the design of the proposed method solves the task well and proves the effectiveness of the proposed model.

[0144] Embodiment 2:

[0145] In this embodiment, the application of the present invention in a specific scenario will be introduced. In crowded public places, such as shopping malls, intersections or schools, when the public security department needs to track the whereabouts of certain people, they can use a large amount of image data in the monitoring system to locate their positions. Combined with the address and time information in the image, the public security department can analyze the activity trajectories of these people.

[0146] Firstly, pedestrian detection technology is used to extract images containing pedestrians from the original videos of regional surveillance cameras, and they are named according to the camera identification and shooting time.

[0147] These images are then fed into the model of the present invention for processing. The model identifies the occurrence of the target individual in other images and determines all of its related images.

[0148] Then, by parsing the image name, the movement path of the target individual can be understood.

[0149] Finally, based on this movement path information, the target individual can be accurately located.

[0150] This model has the ability to identify people in crowded places with just one image. Even if the target individual is mixed in the crowd, it can be quickly located. The public security department can use surrounding surveillance cameras to automatically track the location and activities of specific people, which helps the police analyze their movement trajectory more quickly and take action to arrest specific people.

[0151] The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims

1. A crowded pedestrian re-identification method based on relational graph attention network, characterized in that: The following steps are involved: Step 1: Preprocess P*K image data; Step 2: Input the preprocessed P*K visible light images obtained in step 1 into the feature extraction network; Step 3: Use the ResNet50 network to mine the shallow features of the image input in step 2 into image features rich in deep information; Step 4: For the image input in step 1, use the OpenPose model to obtain the key point set of each image; Step 5: Use multiple linear layers to transform the key point set into a high-dimensional vector; Step 6: Use the graph attention network to transform the high-dimensional vector into significant posture features; Step 7: Input the image features of step 3 and the posture features of step 6 into the shape-guided feature enhancement module to obtain the final image features, and calculate the identity loss and WRT loss; Step 8: Use the obtained identity loss and WRT loss to calculate the total loss and train a network with good performance based on the image features; Step 9: If the specified number of training rounds is reached, proceed to step 10, otherwise continue training and return to step 1; Step 10: End; In step 1, the images are all from the crowded pedestrian re-identification dataset Market-1501-Crowded; In Market-1501-Crowded, various situations of crowds are fully considered. For the original Market-1501 dataset, a part of the crowd images are randomly selected. In each image, a fine-grained image of a person in another image is randomly selected and placed in the image. The image is randomly reduced and placed behind the person, or randomly enlarged and placed in front of the person, so as to simulate the complex relationships in crowded scenes. The steps to create the crowded pedestrian re-identification dataset Market-1501-Crowded are as follows: Step 1-1: Use the deeplabv3_resnet101 model to identify all the pictures in the original dataset, and only keep the characters, and set the other parts to 0; Step 1-2: Transfer the pictures in step 1-1 where the characters occupy 55% of the pictures to another folder; Step 1-3: Manually delete the images with poor recognition results in step 1-2; Step 1-4: Take the pictures in step 1-3 and randomly merge the characters in other pictures with each picture as the benchmark; Step 1-5: Manually delete the images with poor merging effect in step 1-4; Step 1-6: Use the synthesized pictures with better effects to replace the original data set pictures; In order to simulate the diversity of crowded people, in the synthesized pictures, 50% of the newly added people are in front of the original people, and the other 50% are behind the original people; the person in the synthesized picture is recorded as p1, the complete picture is recorded as b1, and the picture of the newly added person is recorded as p2; When the newly added character image is before the original character, the synthesis steps are as follows: Step 1-4-1: randomly enlarge p2 by 10% to 30%, and place the midpoint of p2 10% to 30% below the midpoint of b1, with a deviation of 40% to 70% to the left and right, so as to achieve the effect of larger near and smaller far without completely covering p1; When the newly added character image is behind the original character, the synthesis steps are as follows: Step 1-4-2: Ignore step 1-4-1, randomly reduce p2 by 10% to 30%, and place the midpoint of p2 10% to 30% above the midpoint of b1, with a deviation of 30% to 60% to the left and right, so as to achieve the effect of larger objects near and smaller objects far away and not being completely covered by p1; Step 1-4-3: Overlay p1 on the image output in step 1-4-2 to achieve the effect that p1 is before p2; Since p1 and p2 only retain the character image, the area around the character is black because its value is set to 0. In order not to dye the entire image black, it is necessary to ignore the value of p1 and p2 being 0 when overwriting; In step 4, the OpenPose model is used. For a given image I, the model will output m nodes corresponding to the image. And regard it as a set of body key points of the person in each picture; In step 5, in order to use the node set output in step 4 in the deep neural network, it is necessary to convert it into a high-dimensional feature vector In step 6, the input of the relationship graph attention network is a graph consisting of the high-dimensional feature vector j' obtained in step 5 and k relationship representations. The high-dimensional feature vector j' has m node information in total, and the k relationships are understood as k edges. Represents the input eigenvector matrix; W (r) Represents the relationship matrix, and the final feature matrix is ​​represented by H (r) , and its calculation formula is as follows: H (r) =J'×W (r) (1) and W (r) ∈R j'×j' are all learnable parameters; for H (r) Each node The adjacent index set is recorded as It is updated after the graph attention network based on the weighted sum of its neighboring nodes and the influence of all associated edges. Its calculation formula is as follows: in, is the attention score of the i-th and j-th nodes, σ is the Sigmoid activation function, and SM represents the Softmax activation function; finally, the graph is aggregated by calculating the average value of the node representation to obtain the shape embedding s, which is calculated as shown in formula (3): The shape-guided feature enhancement module in step 7 is operated as follows: in the first stage, the shape feature s outputted from step 6 is used as the query, and the image pooling feature F outputted from step 3 is used as the key and value, and the calculation formula is as shown in formula (4): Q=W q (s),K=W k (F),V=W v (F)(4) Among them, W q , W k With W v are the attention matrices of query, key and value respectively; the correlation score between Q and K is used to search for features directly related to the shape in V, and then perform feature fusion with the shape features through regularization and batch normalization. The calculation formula is as follows: f fused =W n2 (BN(Norm(QK T )V))+s(5) Among them, f fused represents the appearance features mixed with shape features, Norm represents regularization, BN represents batch normalization, and W n2 Represents the coefficient matrix of the deep neural network; considering f fused It contains shape discrimination information and appearance discrimination information related to the shape, and effectively serves as a query in the second-stage attention mechanism, which helps to obtain appearance features directly and indirectly related to the body shape; as the query feature of the second-stage attention, f fused It plays a key role in determining whether identity features directly and indirectly related to shape can be effectively extracted from appearance features; in order to ensure that the query f fused In order to improve the identifiability, the present invention also adopts cross entropy loss and weighted regularized triple loss (WRT) loss to jointly constrain GeM (f fused ), where GeM represents the GeM pooling layer, and its calculation formula is as shown in equations (6) and (7): in, is the result of the cross entropy loss in the first stage, batchnum is the number of samples in a batch, FC 1 is a fully connected layer, whose output will be used as the predicted probability, q i Is the image really labeled as i? Specifically, it is 1, not 0; is the result of the first WRT loss. For image i, j is a positive example, i.e., a person with the same identity, k is a negative example, and d i,j is the Euclidean distance between image i and image j; In the second stage, the output feature f of the attention mechanism in the first stage is fused As a query, to emphasize the appearance features directly and indirectly related to the body shape, the image pooling feature F output in step 3 is still used as the key and value, and the same cross-attention calculation and feature fusion operation as in the first stage are performed to obtain a more deeply fused feature f related to the key pedestrian. final , and also use cross entropy loss and WRT loss to jointly constrain GeM(f final ), and its calculation formula is as follows (8) and (9): The total network loss can be expressed as shown in formula (10), which will be used as the final loss function to constrain the model training process; Among them, λ is a hyperparameter and is set separately in the experiment.

Images