A pedestrian re-identification network based on high-altitude visual field

Abstract

The present invention discloses a high-altitude visual field pedestrian re-identification network, characterized by a multi-branch network including global branches and local branches; the global branch constructs a feature pyramid based on the outputs of the four stages {R2, R3, R4, R5} of ResNet50 to obtain feature semantic information in each stage of ResNet50; the local branch evenly divides the feature vector output by Conv4_1 of the R4 stage into five parts, and adopts different combination strategies to construct feature extraction branches for these five parts to learn local semantic features of different coarse and fine granularity. The present invention uses cross entropy and triplet loss functions to jointly optimize the multi-branch network, combining the advantages of both, while ensuring the network feature extraction effect, reducing the intra-class spacing and improving the inter-class spacing.

Description

Technical Field

[0001] The present invention relates to a pedestrian re-identification technology, and in particular to a pedestrian re-identification network under a high-altitude visual range. Background Art

[0002] The goal of person re-identification (PIR) is to use an algorithm to find multiple images of the same person within a candidate image set, given a target image (e.g., a pedestrian). Re-identification technology has significant application value in intelligent security, particularly by leveraging the high altitude capabilities of drones to establish visual monitoring and tracking platforms. Drones are already widely used in smart cities and high-altitude surveillance, but research on PIR from high altitudes is rare. However, leveraging the high altitude capabilities of drones to establish visual monitoring and tracking platforms is a technology urgently needed in the market.

[0003] Person re-identification at high altitudes has primarily been studied using datasets collected by drones. However, the unpredictable nature of drones leads to variations in pose and camera perspective, making it difficult to extract invariant features from pedestrian images. In recent years, research has focused on extracting local features with strong representational power to complement global features.

[0004] A SDALF method has been proposed that uses the symmetry of pedestrian images to obtain angle-invariant features.

[0005] There are methods that propose local features and fusion of global features. Specifically, a pedestrian picture is normalized into a body part map according to different parts of the person through a spatial transformation network, and then combined with the whole body Figure 1 It is used as input data and input into the shared convolutional neural network.

[0006] Some methods handle the posture changes of pedestrians by applying human body part models and graph structures to establish spatial correspondences.

[0007] The Bi-Cov operator is proposed based on Gabor filter and covariance operator to deal with the interference caused by illumination and background changes.

[0008] The partial least squares method is used to assign weights to each feature.

[0009] Currently, mainstream pedestrian re-identification research focuses on fixed-camera, high-resolution, and standard-sized datasets (such as Market1501, DukeMTMC-reID, and MSMT17). However, the PRAI-1581 image dataset collected by drones under high-altitude viewing conditions naturally has problems such as peculiar perspectives, low resolution, and inconsistent sizes. Mainstream network models have achieved good recognition results on high-resolution and standard-sized datasets on the ground, but lack adaptability to global and local image features on the PRAI-1581 dataset captured by drones. The recognition effect is far lower than that of datasets such as Market1501, and cannot meet the requirements of high-altitude viewing scenarios.

[0010] At the same time, the loss functions widely used in image classification at this stage include cross entropy loss, contrast loss, triple loss, etc., but the current re-identification network still uses a single loss function, which cannot well combine the advantages of different loss functions. Summary of the Invention

[0011] The main purpose of this invention is to provide a pedestrian re-identification network under high-altitude viewing conditions. To address the problems of unusual viewing angles, low-resolution targets, and non-uniform sizes in high-altitude images, a multi-branch network model is proposed to adapt to pedestrian re-identification under high-altitude viewing conditions. This model uses a ResNet50 backbone network to establish a multi-branch network. The global branch is used to extract information expressed by global features, while the local branch is used to extract local feature information of varying granularity. Furthermore, the cross-entropy and triplet loss functions are used to jointly optimize the multi-branch network, combining the advantages of both to ensure effective network feature extraction while reducing intra-class spacing and increasing inter-class spacing. This, in turn, improves model recognition performance.

[0012] The technical solution adopted by the present invention is: a high-altitude visual field pedestrian re-identification network, characterized by a multi-branch network including a global branch and a local branch; the global branch constructs a feature pyramid based on the output of the four stages {R2, R3, R4, R5} of ResNet50, for obtaining feature semantic information in each stage of ResNet50;

[0013] The local branch evenly divides the feature vector output by Conv4_1 in the R4 stage into five parts, and adopts different combination strategies for these five parts to construct feature extraction branches for learning local semantic features with different coarse and fine granularity.

[0014] Furthermore, the global features of the global branch are constructed by the residual block outputs {C2, C3, C4, C5} of the last four stages {R2, R3, R4, R5} of ResNet50. C1 is not included because the image only passes through one convolution layer in the R1 stage, the number of convolutions is small, and it does not contain much strong feature information. The last four stages contain multiple convolution layers, and the feature information with strong expression has been extracted.

[0015] The features extracted from the deep layer are merged with the feature map of the previous layer through upsampling to construct a feature pyramid.

[0016] Furthermore, the local branch performs global maximum pooling on conv4_1 of the R4 stage of ResNet50 and then evenly divides it horizontally into five basic feature blocks. These five feature blocks use different strategies to obtain branches branch1, branch2 and branch3.

[0017] Among them, branch 1 directly contains these five feature blocks, and branch 1 tends to extract the best local feature information;

[0018] In branch 2, every two adjacent blocks are merged to obtain four larger blocks. Due to the problem of data concentration in the image, a body part may be in different blocks. Merging adjacent features can solve the problem of part alignment.

[0019] In branch 3, merge every three adjacent blocks into three larger blocks;

[0020] Finally, a total of 12 feature blocks are obtained from branch1, branch2 and branch3 as local features.

[0021] Furthermore, the features extracted from the multi-branch network are subjected to triplet loss and cross-entropy loss calculations. The triplet construction method follows the hard batch triplet method proposed by Alexander et al., that is, each image in the current batch has a unique triplet. The triplet loss is calculated for all feature blocks separately, and the average is taken as the triplet loss value of the current network.

[0022] All feature blocks extracted from the global branch network and the local branch network are then processed by the fully connected layer to calculate the cross entropy loss. It is necessary to calculate the average value of all feature blocks separately as the cross entropy loss value of the network;

[0023] To calculate the cross entropy loss, we must first obtain the probability that the current image belongs to each category. The calculation method is shown in formula (1):

[0024]

[0025] Where: i represents the category label; W i and bi represent the weight and bias of the feature on the i-th output in the fully connected layer; x represents the feature value; p i Indicates the probability that the feature of the current image belongs to the i-th category;

[0026] After calculating the probability of belonging to each category, the cross entropy loss is calculated. The function expression is shown in formula (2):

[0027]

[0028] Where: i represents the label of the sample image; y i Represents the label of the i-th sample image; N represents the total number of samples;

[0029] The hard batch triplet loss function expression is shown in formula (3):

[0030]

[0031] Where: P represents the number of categories of pedestrians in the current batch; K represents the number of images of each pedestrian; xa represents the selected anchor point image; xp represents the selected positive sample image; xn represents the selected negative sample image; α represents the threshold value of the distance between positive and negative sample pairs; is the Euclidean distance between the positive sample and the image in the feature space,

[0032] f(·) is the embedding function representing the feature extraction network; is the Euclidean distance between the negative sample and the image in the feature space,

[0033]

[0034] In summary, the average value of the hard batch triplet loss and the cross entropy loss is calculated for the extracted feature blocks, and the sum is finally taken as the total network loss. The calculation process is shown in formula (4):

[0035]

[0036] Among them, f i Represents the i-block feature; num represents the number of feature blocks extracted by the network.

[0037] Furthermore, in the testing phase, all features obtained in the global and local branches are resized and concatenated together as the final features of the image. The Euclidean distance is used as the distance measurement function, and the concatenated feature vector is measured with the features of the images in the image library to obtain a distance order table.

[0038] Advantages of the present invention:

[0039] The present invention uses a multi-branch network, with different branches focusing on different features for extraction;

[0040] The construction of the global branch network relies on the feature information in each layer of the network to create a pyramid-shaped convolutional feature hierarchy, obtaining strong semantic information at all scales from top to bottom;

[0041] The local branch network is constructed in the R4 stage. The reason why it is not output in the last layer is to avoid being affected by global information, so that the network can independently learn coarse-grained and fine-grained local semantic features.

[0042] The network uses two loss functions, cross entropy and triplet, to combine their advantages, making the optimized network have stronger feature extraction capabilities in high-altitude visual scenes.

[0043] The obtained multi-branch network is highly practical and can be used not only for pedestrian re-identification in high-altitude scenes, but also for pedestrian re-identification on ground-based datasets such as Market1501 and DukeMTMC-reID.

[0044] To address the challenges of unusual perspectives, low-resolution objects, and non-uniform object sizes in high-altitude imagery, this paper proposes a multi-branch network model for person re-identification in high-altitude imagery. This model uses a ResNet50 backbone network to establish a multi-branch network. The global branch extracts information expressed by global features, while the local branch extracts local features of varying granularity. Furthermore, the multi-branch network is optimized using a cross-entropy and triplet loss function, combining their strengths to ensure effective feature extraction while reducing intra-class distances and increasing inter-class distances. This, in turn, improves model recognition performance.

[0045] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 This is a multi-branch network model diagram based on ResNet50 of the present invention;

[0048] Figure 2 It is a global branch network structure diagram of the present invention;

[0049] Figure 3 It is a local branch network structure diagram of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0051] like Figure 1 As shown in the figure, based on the last four stages of ResNet50, the present invention proposes a multi-branch network that combines global and local multi-scale features. The multi-branch network consists of two parts: a global branch and a local branch. The global branch constructs a feature pyramid based on the outputs of the four stages {R2, R3, R4, R5} of ResNet50, with the aim of obtaining the feature semantic information in each stage of ResNet50. The local branch evenly divides the feature vector output by Conv4_1 in the R4 stage into five parts, and adopts different combination strategies for these five parts to construct feature extraction branches, with the aim of learning local semantic features of different coarse and fine granularity.

[0052] The detailed structure of the global branch is as follows Figure 2 As shown in the figure, the global features are constructed from the residual block outputs {C2, C3, C4, C5} of the last four stages of ResNet50 {R2, R3, R4, R5}. C1 is not included because the image only passes through a single convolutional layer in stage R1, with a small number of convolutions, and does not contain much strong feature information. The last four stages involve multiple convolutional layers, and strong feature information has already been extracted. The features extracted from the deep layer are merged with the feature map of the previous layer through upsampling to construct a feature pyramid. Taking the feature fusion of C5 in the R5 stage and C4 in the R4 stage as an example, {C5, C4} needs to be reduced to 256 dimensions through a 1×1 convolution, resulting in {C5*, C4*}. After upsampling C5*, it is merged through element-by-element addition to obtain f4**. To eliminate the overlap caused by the upsampling process, 3×3 convolution is used to fuse the elements again, generating the feature map f4*. Finally, global max pooling is performed to convert it into a 256×1×1 feature block f4, thus completing the feature fusion of C5 and C4. Except for f5, which is directly obtained through global max pooling of C5*, the features of the remaining stages are fused in the same way, resulting in {f5, f4, f3, f2}. These four feature blocks are then globally mean pooled and resized to form a single feature vector. After that, the triplet and cross-entropy loss are calculated.

[0053] The detailed structure of local branches is as follows Figure 3As shown in the figure, conv4_1 (the output of the first block in R4) of the R4 stage of ResNet50 is globally max-pooled and then evenly divided horizontally into 5 basic feature blocks. These 5 feature blocks use different strategies to obtain branches branch1, branch2, and branch3. Among them, branch1 directly contains these 5 feature blocks, and branch1 tends to extract the best local feature information. In branch2, every two adjacent blocks are merged to obtain 4 larger blocks. Due to the problem of data concentration in the image, a body part may be in different blocks. Merging adjacent features can solve the problem of part alignment. In branch3, every three adjacent blocks are merged into three larger blocks. Finally, a total of 12 feature blocks are obtained from branch1, branch2, and branch3 as local features.

[0054] Features extracted from the multi-branch network require triplet loss and cross-entropy loss calculations. Triplet construction follows the hard batch triplet method proposed by Alexander et al., meaning each image in the batch has a unique triplet. Triplet loss is calculated for all feature blocks individually, and the average is taken as the triplet loss for the current network. All feature blocks extracted from the global and local branch networks are then processed through the fully connected layer to calculate cross-entropy loss. Similarly, the average of all feature blocks is calculated individually as the cross-entropy loss for the network.

[0055] To calculate the cross entropy loss, we must first obtain the probability that the current image belongs to each category (pedestrian ID). The calculation method is shown in formula (1).

[0056]

[0057] Where: i represents the category label; W i and bi represent the weight and bias of the feature on the i-th output in the fully connected layer; x represents the feature value; p i Indicates the probability that the feature of the current image belongs to the i-th category.

[0058] After calculating the probability of belonging to each category, the cross entropy loss is calculated. The function expression is shown in formula (2).

[0059]

[0060] Where: i represents the label of the sample image; y i Represents the label of the i-th sample image; N represents the total number of samples.

[0061] The expression of the hard batch triplet loss function is shown in formula (3).

[0062]

[0063] Where: P represents the number of categories of pedestrians in the current batch; K represents the number of images of each pedestrian;

[0064] xa represents the selected anchor point image; xp represents the selected positive sample image; xn represents the selected negative sample image; α represents the threshold value of the distance between positive and negative sample pairs; is the Euclidean distance between the positive sample and the image in the feature space,

[0065] f(·) is the embedding function representing the feature extraction network; is the Euclidean distance between negative samples and images in the feature space,

[0066]

[0067] In summary, the average value of the hard batch triplet loss and the cross entropy loss is calculated for the extracted feature blocks, and the sum is finally taken as the total network loss. The calculation process is shown in formula (4):

[0068]

[0069] Among them, f i Represents the i-block feature; num represents the number of feature blocks extracted by the network.

[0070] In order to obtain stronger discrimination ability in the testing phase, all features obtained in the global and local branches are resized and concatenated together as the final features of the image. The Euclidean distance is used as the distance measurement function, and the similarity between the concatenated feature vector and the features of the images in the image library is measured to obtain a distance sequence table.

[0071] The present invention is based on an improved multi-branch network algorithm flow framework of ResNet50, including the design of global branch networks and local branch networks; and the combined use of cross entropy and triplet loss functions in multi-branch networks.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A high-altitude visual area pedestrian re-identification network, characterized by: A multi-branch network including a global branch and a local branch; the global branch constructs a feature pyramid based on the output of the four stages {R2, R3, R4, R5} of ResNet50 to obtain feature semantic information in each stage of ResNet50; The local branch evenly divides the feature vector output by Conv4_1 in the R4 stage into five parts, and adopts different combination strategies to construct feature extraction branches for these five parts, so as to learn local semantic features with different coarse and fine granularity; The global features of the global branch are constructed by the residual block outputs {C2, C3, C4, C5} of the last four stages {R2, R3, R4, R5} of ResNet50. C1 is not included because the image only passes through one convolution layer in the R1 stage, the number of convolutions is small, and it does not contain much strong feature information. The last four stages contain multiple convolution layers, and the feature information with strong expression has been extracted; The features extracted from the deep layer are merged with the feature map of the previous layer through upsampling to construct a feature pyramid; The local branch performs global maximum pooling on conv4_1 of the R4 stage of ResNet50 and then evenly divides it horizontally into 5 basic feature blocks. These 5 feature blocks use different strategies to obtain branches branch1, branch2 and branch3; Among them, branch 1 directly contains these five feature blocks, and branch 1 tends to extract the best local feature information; In branch 2, every two adjacent blocks are merged to obtain four larger blocks. Due to the problem of data concentration in the image, a body part may be in different blocks. Merging adjacent features can solve the problem of part alignment. In branch 3, merge every three adjacent blocks into three larger blocks; Finally, a total of 12 feature blocks are obtained from branch1, branch2 and branch3 as local features.

2. The high-altitude visual area pedestrian re-identification network according to claim 1 is characterized in that: The features extracted from the multi-branch network are subjected to triplet loss and cross entropy loss calculations. The triplet construction method follows the hard batch triplet method proposed by Alexander et al., that is, each image in the current batch has a unique triplet. The triplet loss is calculated separately for all feature blocks, and the average is taken as the triplet loss value of the current network. All feature blocks extracted from the global branch network and the local branch network are then processed by the fully connected layer to calculate the cross entropy loss. It is necessary to calculate the average value of all feature blocks separately as the cross entropy loss value of the network; To calculate the cross entropy loss, we must first obtain the probability that the current image belongs to each category. The calculation method is shown in formula (1): （1） Where: Represents the category label; and Represents the features in the fully connected layer at weights and biases on the outputs; represents the eigenvalue; Indicates that the features of the current image belong to Probability of the class; After calculating the probability of belonging to each category, the cross entropy loss is calculated. The function expression is shown in formula (2): （2） Where: The label representing the sample image; Representative Labels of sample images; represents the total number of samples; The hard batch triplet loss function expression is shown in formula (3): （3） Where: Indicates the number of categories of pedestrians in the current batch; Indicates the number of images each pedestrian has; Indicates the selected anchor point image; Represents the selected positive sample image; Represents the selected negative sample image; Indicates the threshold of the distance between positive and negative sample pairs; is the Euclidean distance between the positive sample and the image in the feature space, Represent the feature extraction network for the embedding function; is the Euclidean distance between negative samples and images in the feature space, ； In summary, the average value of the hard batch triplet loss and the cross entropy loss is calculated for the extracted feature blocks, and the sum is finally taken as the total network loss. The calculation process is shown in formula (4): （4） in, express Block features; Indicates the number of feature blocks extracted by the network.

3. The high-altitude visual area pedestrian re-identification network according to claim 1 is characterized in that: In the testing phase, all features obtained from the global and local branches are resized and concatenated together as the final features of the image. The Euclidean distance is used as the distance measurement function. The concatenated feature vector is similar to the features of the image in the image library to obtain a distance sequence table.

Images