A person re-identification method based on multi-layer fusion and alignment partitioning

Abstract

The present invention discloses a pedestrian re-identification method based on multi-layer fusion and alignment division, including the following steps: constructing a pedestrian re-identification network model and training it; using the multi-layer fusion module in the network model to fuse the feature maps of different levels with the feature map of the last layer to obtain the multi-layer fusion features that finally contain shallow feature information; using the alignment division module in the network model to extract the center position of the pedestrian, and then expanding the center position to both sides to obtain the local features of the pedestrian's precise segmentation of the local area; connecting the multi-layer fusion features, the local features and the global features according to the channel dimension to obtain the final pedestrian discrimination features to complete pedestrian re-identification. Beneficial effects of the present invention: the proposed fusion module can fuse the information carried by the feature maps of different levels, and on this basis extract the multi-layer fusion features and add them to the final discrimination features for auxiliary identification, thereby effectively improving the performance of re-identification.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-identification, and in particular to a pedestrian re-identification method based on multi-layer fusion and alignment division. Background Art

[0002] In recent years, deep learning methods have been widely used in pedestrian re-identification tasks. The mainstream method is to extract features containing high-level semantic information through deep networks for identification. However, simply using the high-level semantic information (objects or parts) contained in deep features often ignores the information carried by shallow network features (color, texture, etc.), which is not conducive to identification. For example, some pedestrians with bright and prominent clothing can be identified even at low image resolutions, and shallow information is very effective in identifying such images. For images with less obvious features, it is necessary to use deep networks to extract high-level semantics (certain prominent images of the body). Most of the current deep learning-based methods only use deep features and ignore the practical value of shallow features. Summary of the invention

[0003] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: to propose a pedestrian re-identification method based on multi-layer fusion and alignment division, so that deep features can carry shallow information and the divided local features can be well aligned.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a pedestrian re-identification method based on multi-layer fusion and alignment division, comprising the following steps: constructing and training a pedestrian re-identification network model; using the multi-layer fusion module in the network model to fuse feature maps of different levels with the feature map of the last layer to obtain a multi-layer fusion feature that finally contains shallow feature information; using the alignment division module in the network model to extract the center position of the pedestrian, and then expanding the center position to both sides to obtain local features of the local area of ​​the pedestrian for accurate segmentation; connecting the multi-layer fusion features, the local features and the global features according to the channel dimension to obtain the final pedestrian discrimination features to complete pedestrian re-identification.

[0007] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and alignment partitioning described in the present invention, wherein: the multi-layer fusion module includes an improved Non-local module; the Non-local module gives a feature map F∈R C×H×W , where C is the number of channels of the feature map, H×W represents the size of the feature map; F is divided into two paths and the spatial feature correlation matrix X∈R is calculated after convolution. N×N , where N = H × W, and the EmbeddedGaussian function is used to calculate X:

[0008]

[0009] where x i With x j Represents the eigenvalues ​​and x at spatial positions i and j respectively. ij Represents the correlation between spatial positions i and j; multiply the obtained correlation matrix X by F, and the obtained feature map F′ is

[0010] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and alignment partitioning described in the present invention, the multi-layer fusion module includes a fusion step, given two feature maps at different levels and Where C1 and C2 are the number of channels of the two feature maps, H1×W1 and H2×W2 represent the size of the feature maps. The shallower feature map is passed through a 1×1 convolution layer to obtain Make its channel number C′1=C2; then calculate the spatial feature correlation matrix of two different feature maps by using the model of the Non-local module Where N1 = H1 × W1, N2 = H2 × W2; the following is obtained:

[0011]

[0012] where y i represents the eigenvalue at spatial position i of feature map F2, y j represents the eigenvalue at the spatial position j of the feature map F′1, y ij Represents the correlation between the corresponding i and j positions in the two feature maps; perform matrix multiplication of Y and F′1: Each pixel position in the obtained feature map F′2 is equivalent to carrying the information of each position through its correlation with the feature map F1, thus realizing the fusion of two levels of features.

[0013] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and alignment partitioning described in the present invention, the alignment partitioning module includes: using STN as the basic network architecture and generating a three-dimensional tensor; sending the three-dimensional tensor to STN to learn the center position of the pedestrian in the image, and generating a conversion matrix of affine transformation:

[0014]

[0015] Fixed t x , and directly fix the scale parameter (s according to the size of the area to be divided x ,s y );Use learning to get t y value and expand to both sides of the Y axis, that is, for t y Add or subtract the length s of the side of the divided area in the Y-axis direction y , get the Y-axis center of other segmented areas; set the limiting conditions:

[0016]

[0017] The limiting condition means that when the sampling area exceeds the boundary of the feature map, a region size is sampled inward starting from the upper boundary or the lower boundary as an edge.

[0018] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and aligned division described in the present invention, wherein: the network model of pedestrian re-identification includes, the network model infrastructure adopts Resnet-50; the network is divided into three branches at res_layer4, which is divided into a global feature branch and two local feature branches divided into 2 and 3 blocks respectively; multi-layer fusion is used on the global feature branch, and the feature maps at res_layer1, res_layer2, and res_layer3 are respectively fused with the feature map of the deepest layer res_layer4 of the network through a multi-layer fusion module; the three feature maps are added element by element to obtain a multi-layer fused feature map.

[0019] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and alignment division described in the present invention, wherein: the local feature branch includes, when two local feature branches respectively obtain feature maps through their respective res_layer4; the feature maps are sent to the STN subnetwork for learning, and the feature maps of the two branches are respectively divided into 2 and 3 local feature maps using the alignment division module; GMP is used for each obtained local feature map, and the obtained local features are obtained through the corresponding dimensionality reduction layer to obtain each 256-dimensional local feature; the multi-layer fusion features, global features and local features are connected according to the channel dimension to obtain the final pedestrian discrimination features.

[0020] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and alignment partitioning described in the present invention, the training of the network model includes: using cross entropy loss and hard sample sampling triple loss to train the model; using cross entropy loss for all features to optimize the classification task, connecting local features from the same feature map to obtain local connection features; using triple loss for multi-layer fusion features, global features and local connection features to perform metric learning, and the final loss function used is: in represents the sum of the cross entropy losses of all features, Representing the sum of triplet losses, β1 and β2 are balance parameters, which are set to β1=2 and β2=1 respectively in the experiment.

[0021] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and alignment division described in the present invention, wherein: the STN includes two convolutional layers and a pooling layer; t is obtained through a fully connected layer y , the STN learns t y Combined with other fixed parameters, the corresponding affine transformation matrix is ​​obtained, which can sample the central area of ​​the pedestrian in the original feature map, that is, the first local area divided.

[0022] As a preferred solution of the pedestrian re-identification method based on multi-layer fusion and aligned partitioning described in the present invention, the multi-layer fusion feature map includes using GAP to process the multi-layer fusion feature map and the feature map of the res_layer4 layer respectively; the processed features are then passed through a dimensionality reduction layer including 1×1conv, batchnormalisation and ReLU activation to obtain 256-dimensional multi-layer fusion features and global features.

[0023] The beneficial effects of the present invention are as follows: the proposed multi-layer fusion module can fuse the information carried by feature maps of different levels, and on this basis extract the multi-layer fusion features and add them to the final discriminant features for auxiliary recognition, thereby effectively improving the performance of re-recognition; the alignment partitioning module optimizes the existing brute force partitioning method without introducing additional supervision, and accurately aligns some local areas that are difficult to align due to camera perspective problems. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0025] Figure 1 A schematic diagram of local features obtained by violent division according to the first embodiment of the present invention;

[0026] Figure 2 A schematic diagram of local features obtained by alignment and division according to the first embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of the Non-local model structure described in the first embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of the fusion model structure of the multi-layer fusion module described in the first embodiment of the present invention;

[0029] Figure 5 Schematic diagram of the overall structure of the pedestrian re-identification network model according to the first embodiment of the present invention;

[0030] Figure 6 A schematic diagram of an original picture for visualization of alignment and division according to the second embodiment of the present invention;

[0031] Figure 7 A schematic diagram of a group of visualization results of the local area alignment division described in the second embodiment of the present invention;

[0032] Figure 8 This is another set of result schematic diagrams for visualizing the local area alignment division described in the second embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0034] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0035] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0036] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0037] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0038] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0039] Example 1

[0040] Pedestrian re-identification is in great demand in the field of intelligent security. It aims to associate the same pedestrians at different times and locations. The general approach is to give a pedestrian image to be retrieved, extract the features of the query image and the images in the gallery through a trained model, sort the images in the gallery according to the similarity of feature embedding, and perform pedestrian image retrieval. In recent years, the task of pedestrian re-identification has made great progress. However, in an open outdoor environment, pedestrian images will have large differences due to interference such as posture, occlusion, clothing, background clutter, and camera perspective. Therefore, pedestrian re-identification is still a very challenging task. There are generally two major directions to solve the problem of pedestrian re-identification, called representation learning and metric learning. Representation learning includes traditional manual description features and currently popular deep learning features. And with the vigorous development of deep learning, deep features extracted by deep neural networks have replaced manual features to a certain extent for human retrieval.

[0041] There are two problems in the deep learning-based person re-identification task: first, only deep features containing high-level semantic information (objects or parts) are used for recognition, while shallow information (color or texture) is ignored; second, the existing partition-based method for extracting multi-granular features makes it difficult to align the obtained local features.

[0042] In order to extract rich pedestrian image representations, many existing pedestrian re-identification methods aggregate global and local features to improve performance. First, strong structural information, such as prior knowledge or human posture, is used to extract relevant local features. Second, horizontal local feature strips of the same size are obtained by brute force division of global features. However, these methods have certain limitations. Although the first type of method can obtain the corresponding local feature components, it introduces additional supervision to the local feature extraction, making network training more complicated. The second type of method directly performs brute force segmentation on the global features, which will lead to alignment problems of local features and produce alignment errors between the body and the background, such as Figure 1 , which reduces the performance of the trained network.

[0043] Based on the above problems, this embodiment proposes a method of multi-layer fusion and aligned partitioning to perform pedestrian re-identification. The proposed multi-layer fusion method uses a non-local model. Different from the previous method using a non-local model, which operates on the same feature map to obtain the long-distance dependencies of the corresponding pixels, the method in the present invention fuses feature maps of different levels with the feature map of the last layer so that the final deep features contain information about shallow features. At the same time, the Spatial Transformer Network (STN) is used to extract the center position of the pedestrian, and then the center position is expanded on both sides along the Y axis to obtain the accurate segmentation of the pedestrian. Local area, such as Figure 2 It should be noted that STN is performed in an end-to-end learning manner without introducing additional supervision, and the segmentation method proposed in this embodiment effectively solves the alignment problem.

[0044] A method of multi-layer fusion and aligned partitioning is proposed to solve the problem. The proposed fusion module fuses the deep and shallow feature maps so that the deep features can carry shallow information; the aligned partitioning improves the existing partitioning method by using the constructed SpatialTransformerNetwork (STN) subnet, so that the divided local features are well aligned.

[0045] Specifically, the pedestrian re-identification method based on multi-layer fusion and alignment division includes the following steps:

[0046] S1: Build and train a person re-identification network model.

[0047] The network model for person re-identification in this step includes: the network model infrastructure adopts Resnet-50; the network is divided into three branches at res_layer4, which is divided into a global feature branch and two local feature branches divided into 2 and 3 blocks respectively; multi-layer fusion is used on the global feature branch. In order to extract multi-granularity features, the feature maps at res_layer1, res_layer2, and res_layer3 are fused with the feature map of the deepest layer res_layer4 of the network through the multi-layer fusion module; the three feature maps are added element by element to obtain a multi-layer fused feature map.

[0048] Furthermore, the local feature branch includes:

[0049] After the two local feature branches obtain feature maps through their respective res_layer4, the feature maps are sent to the STN subnetwork for learning, and the feature maps of the two branches are divided into 2 and 3 local feature maps respectively using the alignment partitioning module; GMP is used for each local feature map obtained (different from the use of GAP on the global feature map, the use of GMP on the local feature map is more conducive to mining the most discriminative local features), and the obtained local features are obtained through the corresponding dimensionality reduction layer to obtain each 256-dimensional local feature; the multi-layer fusion features, global features and local features are connected according to the channel dimension to obtain the final pedestrian discrimination features.

[0050] It should be noted that the training of the network model in this embodiment includes:

[0051] The model is trained using cross entropy loss and triplet loss for hard sample sampling. The cross entropy loss is used for all features to optimize the classification task, and local features from the same feature map are connected to obtain local connection features.

[0052] The triplet loss is used for multi-layer fusion features, global features, and local connection features for metric learning (which helps to aggregate the final features of pedestrians with the same identity during the test phase). The final loss function used is: in represents the sum of the cross entropy losses of all features, Representing the sum of triplet losses, β1 and β2 are balance parameters, which are set to β1=2 and β2=1 respectively in the experiment.

[0053] STN consists of two convolutional layers and a pooling layer, and a fully connected layer is used to obtain t y , STN learns t yCombined with other fixed parameters, the corresponding affine transformation matrix is ​​obtained, which can sample the central area of ​​the pedestrian in the original feature map, that is, the first local area of ​​the division. The multi-layer fusion feature map includes using GAP to process the multi-layer fusion feature map and the feature map of the res_layer4 layer respectively; the processed results are then passed through the dimensionality reduction layer including 1×1conv, batchnormalisation and ReLU activation to obtain 256-dimensional multi-layer fusion features and global features.

[0054] S2: Use the multi-layer fusion module in the network model to fuse the feature maps of different levels with the feature maps of the last layer to obtain the final multi-layer fusion features containing shallow feature information.

[0055] Specifically, the non-local module applied to two-dimensional images takes a weighted sum of the eigenvalue of a certain position on the feature map and the values ​​of all other positions. The corresponding weight is determined by the correlation between the eigenvalues. In this way, no matter how far apart the two positions are in space, as long as they have a certain correlation, they can promote each other. See, Figure 3 's hint.

[0056] The multi-layer fusion module in this step includes:

[0057] Adopt the improved Non-local module; the Non-local module gives the feature map F∈R C×H×W , where C is the number of channels of the feature map, H×W represents the size of the feature map; F is divided into two paths and the spatial feature correlation matrix X∈R is calculated after convolution. N×N , where N = H × W, and the EmbeddedGaussian function is used to calculate X:

[0058]

[0059] where x i With x j Represents the eigenvalues ​​and x at spatial positions i and j respectively. ij Represents the correlation between spatial positions i and j; multiply the obtained correlation matrix X by F, and the obtained feature map F′ is

[0060] Through the discussion of the non-local module applied to two-dimensional images, it can be found that this module is only used to establish the correlation of pixel positions in a certain layer of feature maps, effectively establishing the long-range dependence of pixels in a feature map with other pixels in itself. Taking this into account, in this step, by improving the non-local module, the connection between a feature map and pixels in another feature map is established, rather than with itself. For reference, Figure 4Further, the multi-layer fusion module includes a fusion step,

[0061] Given two feature maps at different levels and Where C1 and C2 are the number of channels of the two feature maps, H1×W1 and H2×W2 represent the size of the feature maps. The shallower feature map is passed through a 1×1 convolution layer to obtain Make its channel number C′1=C2;

[0062] Then, by using the model of the Non-local module, the spatial feature correlation matrix of two different feature maps is calculated Where N1 = H1 × W1, N2 = H2 × W2;

[0063] The following is obtained:

[0064]

[0065] where y i represents the eigenvalue at spatial position i of feature map F2, y j represents the eigenvalue at the spatial position j of the feature map F′1, y ij Represents the correlation between the corresponding i and j positions in the two feature maps.

[0066] Perform matrix multiplication of Y and F′1: Each pixel position in the obtained feature map F′2 is equivalent to carrying the information of each position through its correlation with the feature map F1, thus realizing the fusion of two levels of features.

[0067] It should also be noted that in the existing deep learning-based person re-identification methods, most of them directly use the final layer features containing high-level semantic information extracted by the deep network for pedestrian retrieval, thereby ignoring the image information contained in the shallow features. In order to solve this problem, this step effectively fuses the feature information contained in each network layer by improving the non-local module, so as to extract an auxiliary feature that can carry multi-level information to assist in recognition.

[0068] By using the proposed method, multiple different shallow feature maps are fused with the final deep feature, and then element-by-element addition is performed to obtain the final feature that integrates multiple layers of feature information. For the specific structure, refer to Figure 5 's hint.

[0069] S3: Use the alignment and segmentation module in the network model to extract the center position of the pedestrian, and then expand the center position to both sides to obtain the local features of the local area of ​​the pedestrian for accurate segmentation.

[0070] The introduction of PCB has made the partitioning method play a very important role in pedestrian re-identification, and some improvements based on it have also greatly improved the accuracy of pedestrian re-identification. However, these methods directly use brute force partitioning from top to bottom, without aligning the local features obtained by the partitioning well. The alignment partitioning proposed in this step is actually an alignment partitioning module based on Spatial Transformer Network (STN). Since STN is a lightweight subnet and can be inserted into any position of the network, it does not bring unnecessary burden to network training by introducing additional supervision.

[0071] Specifically, the alignment and division module in this step includes:

[0072] STN is used as the basic network architecture and a three-dimensional tensor is generated. The three-dimensional tensor is sent to STN to learn the center position of the pedestrian in the image and generate the transformation matrix of the affine transformation:

[0073]

[0074] Since the segmentation method is performed in the Y-axis direction, the problem of misalignment of the local area after segmentation also occurs in the Y-axis, and the purpose of using STN in the method is to learn the center position of the pedestrian. x , and directly fix the scale parameter (s according to the size of the area to be divided x ,s y ); In fact, STN only learns in the proposed structure to find the center area of ​​the pedestrian on the Y axis for alignment and division. The structure of STN is shown in Table 1.

[0075] Table 1: STN subnetwork configuration (s: stride, p: padding, input size: 2048×24×8).

[0076]

[0077] Using learning to get t y value and expand to both sides of the Y axis, that is, for t y Add or subtract the length s of the side of the divided area in the Y-axis direction y , get the Y-axis center of other segmented areas;

[0078] Note that y When expanding, the sampling may exceed the upper and lower boundaries of the feature map. According to the STN position parameter range of [-1,1], set the limiting conditions:

[0079]

[0080] The restriction means that when the sampling area exceeds the boundary of the feature map, the upper boundary or the lower boundary is used as an edge to start sampling an area size inward. Although this may cause area overlap, it has almost no negative impact on the algorithm, because even if the overlapping area is finally optimized twice in different sampling areas, it is based on alignment and the optimization obtained is similar, so overlap is allowed.

[0081] S4: Connect the multi-layer fusion features, local features and global features according to the channel dimension to obtain the final pedestrian discriminant features to complete pedestrian re-identification.

[0082] Therefore, this embodiment improves some problems of existing methods for pedestrian re-identification tasks. The proposed fusion module can fuse the information carried by feature maps at different levels, and on this basis extract multi-layer fusion features and add them to the final discriminant features for auxiliary recognition, thereby effectively improving the performance of re-identification. Without introducing additional supervision, the alignment partitioning module can better optimize the existing brute force partitioning method and accurately align some local areas that are difficult to align due to camera perspective problems.

[0083] Example 2

[0084] To verify the actual effect of the present invention, experiments were conducted on the three most commonly used datasets for person re-identification tasks: Marker-1501, DukeMTMC-reID, and CUHK03. The first successful match probability (rank-1) and mean average precision (mAP) were used to evaluate the experimental results. Since the rank-1 standard will reach saturation when the matching reaches a certain level, the discrimination is not high, so the evaluation of mAP is more focused.

[0085] It should be noted that Marker-1501 includes 1501 pedestrians of different identities captured by 6 cameras. The dataset generates 32668 pictures of individual pedestrians through the DPM detector. They are divided into non-overlapping training / test sets. The training set contains 12936 pictures of 751 pedestrians of different identities, and the test set contains 3368 query pictures and 19732 gallery pictures from 750 pedestrians of different identities. The detection boxes of the query pictures are drawn manually to ensure the accuracy of the test results.

[0086] DukeMTMC-reID is a pedestrian re-identification subset of the DukeMTMC dataset, which is collected using 8 cameras and includes 36,411 images of 1,812 pedestrian identities. There are 1,404 pedestrians in the pictures that appear under more than 2 cameras. Through random sampling, these 1,404 pedestrian pictures are divided into training set and test set, which include 702 pedestrian pictures respectively. The remaining 408 pedestrians only appear under 1 camera, and their pictures are also added to the gallery of the test set as interference. The training set includes 16,522 pictures, the gallery contains 17,661 pictures, and the query set contains 2,228 pictures.

[0087] The CUHK03 dataset includes 14,097 images of 1,467 pedestrians, each of which is captured by two different cameras. The images of 767 identities are used for training, and the images of the other 700 identities are used for testing. The dataset provides two annotation methods: manual annotation and detector automatic annotation. The present invention conducts experiments under both annotation datasets.

[0088] Experimental parameter setting: This example uses the partitioning method, so the experiment follows the PCB setting. During the training phase, the input image size is adjusted to 384×128, and then random horizontal flipping, standardization, and random erasing are used for data enhancement. During the test, the image is adjusted to 384×128 and only standardized. The network infrastructure uses ResNet-50 pre-trained on the ImageNet dataset. The experimental batch size is set to 32. 8 identities are randomly sampled from the training set, and 4 images are sampled from each identity. The network is trained using the Adam optimizer, and the initial learning rate is set to 3×10 -4 , a total of 200 cycles of training were performed. When the training reached 140 and 180 cycles, the learning rate dropped to 3×10 -5 With 3×10 -6 The margin of the triplet loss is set to 1.2. During the test evaluation, following MGN, the features of the original image and the features of the horizontally flipped image are averaged to obtain the final features.

[0089] The results of the method proposed in this embodiment are compared with recent methods on three data sets. Specifically, the related comparison methods are divided into two categories according to whether the partitioning method is used: methods using partitioning (the first category) and other deep methods (the second category). In order to ensure the fairness of the comparison, no post-processing methods such as reordering are used in the comparison results.

[0090] The performance comparisons of the Market-1501 and DukeMTMC-reID datasets are shown in Tables 2 and 3 respectively.

[0091] Table 2: Performance comparison of experimental results on Market-1501 dataset (* indicates additional supervision)%.

[0092]

[0093] Table 3: Performance comparison of experimental results on DukeMTMC-reID dataset (* indicates additional supervision)%.

[0094]

[0095] It can be seen that MGN is still in an advantageous position among the partition-based methods on the two datasets. The method proposed in this invention is only slightly lower than MGN in rank-1 on Market-1501, but is better than MGN in both mAP and DukeMTMC-reID's mAP and rank-1 indicators. In particular, the mAP indicator is significantly improved compared to MGN on both datasets. For other deep learning methods, it can be seen from the table that this method has significant advantages. Compared with Auto-ReID, mAP and rank-1 are improved by 4.56% and 0.77% respectively on the evaluation indicators of the DekMTMC-reID dataset. Compared with the excellent DSA-reID on Market-1501, the result is only slightly lower in rank-1, but considering that DSA-reID introduces additional supervision, this method is still competitive.

[0096] The performance comparison of CUHK03 is shown in Table 4.

[0097] Table 4: Performance comparison of experimental results on CUHK03 dataset (* indicates additional supervision)%.

[0098]

[0099] It can be seen that existing methods such as CASN (PCB) and Auto-ReID have achieved excellent results on the two annotation sets of CUHK03. EANet also has good performance on the detection annotation set. Under these two annotation sets, the results of this method are better than these methods. Compared with Auto-ReID, the mAP / rank-1 on the manual annotation set is improved by 3.65% / 0.96%, and the mAP / rank-1 on the detection annotation set is improved by 4.03% / 2.06%. It is worth noting that MGN performs well on Market-1501 and DukeMTMC-reID, but performs poorly on CUHK03. In contrast, the method of the present invention has good performance on all three datasets.

[0100] In order to verify the effect of the method proposed in the present invention.

[0101] This embodiment also conducts ablation experiments on the Market-1501 and DukeMTMC-reID datasets, and the results are shown in Tables 5 and 6. All experiments have the same settings except for adding or removing corresponding modules.

[0102] Table 5: Aligned partition ablation experiment (BL1: Baseline1; BL2: Baseline2; LCF: Local Connection Feature; AP: Aligned Partition) %.

[0103]

[0104]

[0105] Table 6: Multi-layer fusion ablation experiment (GF: global feature; MF: multi-layer fusion feature) %.

[0106]

[0107] Network architecture analysis: The present invention proposes an alignment partitioning method to optimize the brute force partitioning method. Since PCB and MGN are both typical partitioning methods, the local feature branches of the experimental network architecture are selected in the form of these two architectures for comparison. The experiment compares two network architectures based on ResNet-50: Baseline1 contains a global branch and a local branch, and the local branch is divided into 6 blocks in the form of PCB; Baseline2 contains a global branch and two local branches, and the local branches are divided into 2 and 3 blocks in the form of MGN. From the comparison of the corresponding experiments in the two groups of experiments based on these two architectures in Table 5, it can be seen that the model architecture of using MGN divided into 2 and 3 branches can obtain better results. It can be seen that simply extracting global and local features without a continuous span of the receptive field is relatively unfavorable for recognition, which also verifies that the local features extracted by the continuous multi-granularity partitioning structure are more advantageous for identifying pedestrians.

[0108] Improvement of performance by local connection features: For person re-identification, the corresponding pedestrian needs to be determined based on the feature similarity between the query image and the gallery image. All features are connected in the network to obtain the final discriminant feature. However, the similarity of each local feature is not consistent, and the connection will cause a certain deviation in the recognition of the feature. Since each local feature is divided from the corresponding global feature, if the triple loss is used to optimize each local feature, not only will the optimization complexity be greatly increased, but also the optimization will not achieve the desired effect due to the local alignment problem. In order to solve this problem, the local features obtained from the same global feature are connected to obtain the local connection feature, and the triple loss is used for the local connection feature to reduce the metric distance of the discriminant feature formed by the connection of the local features. According to the comparison of the first group of experiments in the two network structures in Table 5 with their own second group of experiments, it can be seen that the metric learning optimization of the local connection feature can make the final discriminant feature have better performance.

[0109] Improvement of performance by alignment partitioning: The use of local connection features is to perform corresponding metric learning optimization on the local features contained in the final recognition features. The use of local connection features does not have a relevant impact on the use of alignment partitioning. In Table 5, an experiment on alignment partitioning is conducted based on the use of local connection features. By comparing the second and third groups of Baseline1, and the second and third groups of Baseline2, it can be seen that alignment partitioning has brought a huge improvement in the experimental results. The mAP of the two datasets in Baseline1 has increased by 2.88% / 1.25%, and the mAP of Baseline2 has increased by 0.86% / 0.75%, respectively. Figures 6 to 8 As shown in the figure, in the Market-1501 dataset, the alignment division results of the two local branches of Baseline2 are visualized. Compared with the brute force division that will cause the problem of alignment between body parts and background, it can be seen from the figure that each local area has the corresponding parts of the pedestrians for corresponding alignment. It can also be seen that the alignment division has brought excellent improvements to the division method.

[0110] Improvement of performance by multi-layer fusion: In order to ensure the fairness and accuracy of the comparison of multi-layer fusion experiments, the third group of experiments of the Baseline2 architecture in Table 5 was used as the basis for the experiments on the multi-layer fusion module. As shown in Table 6, due to the addition of the multi-layer fusion module to obtain multi-layer fusion features as auxiliary identification, the length of the pedestrian discrimination features finally connected increased. In order to ensure the consistency of feature comparison, a global feature was added to the first group of basic experiments as the second group of experiments, and compared with the experimental group (the third group) with the addition of the multi-layer fusion module. From the second group of experiments in Table 6, it can be seen that the addition of a global feature only slightly improves the basic experimental group, which shows that the existence of a global feature has met the requirements of the multi-granularity characteristics of the feature. However, from the third group, it can be seen that the results are still significantly improved compared with the first group, which proves the effectiveness of the multi-layer fusion module.

[0111] Based on the above verification, the present invention improves some problems of existing methods for pedestrian re-identification tasks. The proposed fusion module can fuse the information carried by feature maps at different levels, and on this basis extract multi-layer fusion features to add to the final discriminant features for auxiliary identification, thereby effectively improving the performance of re-identification. The alignment partitioning module can optimize the existing brute force partitioning method without introducing additional supervision, and accurately align some local areas that are difficult to align due to camera perspective problems. From the experimental results of the present invention, it can be seen that the proposed method has achieved excellent results on existing general data sets, which also proves the effectiveness of the method. Of course, the model becomes relatively complex due to the multiple paths divided. The next improvement plan focuses on streamlining the model and trying to introduce the multi-granularity of features in different ways, so as to better improve the accuracy while making the model as lightweight as possible.

[0112] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0113] In addition, the operations of the processes described in the present invention may be performed in any suitable order unless otherwise indicated by the present invention or otherwise clearly contradicted by the context. The processes described in the present invention (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0114] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described in the present invention includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself. The computer program can be applied to input data to perform the functions described in the present invention, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on a display.

[0115] As used in this application, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in operation. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program, and / or a computer. As an example, an application running on a computing device and the computing device can both be components. One or more components can exist in a process and / or thread in execution, and a component can be located in a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures thereon. These components can communicate in a local and / or remote process manner, such as according to a signal having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems in a signal manner through a network such as the Internet).

[0116] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A pedestrian re-identification method based on multi-layer fusion and alignment partitioning, characterized by: The following steps are included: Build and train a person re-identification network model; The multi-layer fusion module in the network model is used to fuse the feature maps of different layers with the feature map of the last layer. The multi-layer fusion module adopts an improved Non-local module. The improved Non-local module is given a feature map, and the spatial feature correlation matrix is ​​calculated using an Embedded Gaussian function. The spatial feature correlation matrix is ​​feature-fused with the feature map to obtain a multi-layer fusion feature containing shallow feature information. The center position of the pedestrian is extracted by using the alignment and segmentation module in the network model. The alignment and segmentation module adopts STN as the basic network architecture, determines the center position of the pedestrian by learning to generate the transformation matrix of the affine transformation, and expands the center position to both sides to obtain the local features of the local area of ​​the pedestrian for accurate segmentation; The multi-layer fusion features, the local features and the global features are connected according to the channel dimension to obtain the final pedestrian discrimination features to complete pedestrian re-identification.

2. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning according to claim 1, characterized in that: The multi-layer fusion module includes: Adopt improved Non-local module; The Non-local module is given a feature map F∈R C×H×W , where C is the number of channels of the feature map, H×W represents the size of the feature map; Divide F into two paths and calculate the spatial feature correlation matrix X∈R through convolution. N×N , where N = H × W, and the EmbeddedGaussian function is used to calculate X: where x i With x j Represents the eigenvalues ​​and x at spatial positions i and j respectively. ij represents the correlation between spatial positions i and j; Multiply the obtained correlation matrix X by F, and the obtained feature map F′ is 3. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning according to claim 1 or 2, characterized in that: The multi-layer fusion module includes a fusion step, Given two feature maps at different levels and Where C1 and C2 are the number of channels of the two feature maps, respectively, and H1×W1 and H2×W2 represent the size of the feature maps; The shallower feature map is obtained by passing it through a 1×1 convolution layer Make its channel number C′1=C2; Then, by using the model of the Non-local module, the spatial feature correlation matrix of two different feature maps is calculated Where N1 = H1 × W1, N2 = H2 × W2; The following is obtained: where y i represents the eigenvalue at spatial position i of feature map F2, y j represents the eigenvalue at the spatial position j of the feature map F1′, y ij Represents the correlation between the corresponding i and j positions in the two feature maps; Perform matrix multiplication of Y and F1′: Each pixel position in the obtained feature map F′2 is equivalent to carrying the information of each position through its correlation with the feature map F1, thus realizing the fusion of two levels of features.

4. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning as claimed in claim 3, characterized in that: The alignment and division module comprises: Use STN as the basic network architecture and generate a three-dimensional tensor; The three-dimensional tensor is fed into STN to learn the center position of the pedestrian in the image and generate the transformation matrix of the affine transformation: Fixed t x , and directly fix the scale parameter (s according to the size of the area to be divided x ,s y ); Using learning to get t y value and expand to both sides of the Y axis, that is, for t y Add or subtract the length s of the side of the divided area in the Y-axis direction y , get the Y-axis center of other segmented areas; According to the STN position parameter range of [-1,1], set the limiting conditions: The limiting condition means that when the sampling area exceeds the boundary of the feature map, a region size is sampled inward starting from the upper boundary or the lower boundary as an edge.

5. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning as claimed in claim 4, characterized in that: The network model for pedestrian re-identification includes: The network model infrastructure adopts Resnet-50; The network is divided into three branches at res_layer4, one global feature branch and two local feature branches divided into 2 and 3 blocks respectively; Using multi-layer fusion on the global feature branch, the feature maps at res_layer1, res_layer2, and res_layer3 are fused with the feature map of the deepest layer res_layer4 of the network through a multi-layer fusion module; The three feature maps are added element by element to obtain a multi-layer fused feature map.

6. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning as claimed in claim 5, characterized in that: The local feature branch includes: After the two local feature branches pass through their respective res_layer4 to obtain feature maps; The feature map is sent to the STN sub-network for learning, and the feature maps of the two branches are divided into 2 and 3 local feature maps respectively by using the alignment partitioning module; Use GMP for each local feature map obtained, and pass it through the corresponding dimensionality reduction layer to obtain each 256-dimensional local feature; The multi-layer fusion features, global features and local features are connected according to the channel dimension to obtain the final pedestrian discrimination features.

7. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning according to claim 5 or 6, characterized in that: The training of the network model includes: The model is trained using cross entropy loss and hard sample sampling triplet loss; Use cross entropy loss for all features to optimize the classification task, and connect local features from the same feature map to obtain local connection features; The metric learning is performed using triplet loss for multi-layer fusion features, global features, and local connection features. The final loss function used is: L = β1L softmax +β2L triplet Where L softmax represents the sum of the cross entropy losses of all features, L triplet Representing the sum of triplet losses, β1 and β2 are balance parameters, which are set to β1=2 and β2=1 respectively in the experiment.

8. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning as claimed in claim 7, characterized in that: The STN includes two convolutional layers and a pooling layer; Through a fully connected layer, we get t y , the STN learns t y Combined with other fixed parameters, the corresponding affine transformation matrix is ​​obtained, which can sample the central area of ​​the pedestrian in the original feature map, that is, the first local area divided.

9. The pedestrian re-identification method based on multi-layer fusion and alignment partitioning as claimed in claim 8, characterized in that: The multi-layer fusion feature map includes: Use GAP to process the multi-layer fusion feature map and the feature map of the res_layer4 layer respectively; The processed features are then passed through dimensionality reduction layers including 1×1conv, batchnormalisation and ReLU activation to obtain 256-dimensional multi-layer fusion features and global features.

Images