Pedestrian re-identification method and device based on complementary feature dynamic fusion network model

By adopting complementary features dynamic fusion network model in pedestrian re-identification technology, combining Vit and NFC modules, the global and local features of pedestrians are extracted, and the problem of low recognition accuracy in the existing technology is solved, and high in complex environments is achieved Accurate pedestrian identification.

CN117746462BActive Publication Date: 2024-11-01SHENZHEN POLYTECHNIC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311762029.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-19
Publication Date
2024-11-01
Estimated Expiration
2043-12-19

AI Technical Summary

Technical Problem

The existing pedestrian re-identification technology is limited by posture marking and powerful posture estimation networks, which ignores the global information and implicit relationship of the image, resulting in low recognition accuracy, especially in the case of light changes, object occlusion and noisy background. Effective identification.

Method used

Using a dynamic fusion network model based on complementary features, combined with the visual converter (Vit) network module and neighborhood feature constraint (NFC) module, the main branch and auxiliary branch are extracted through multi-level features, and the pedestrian global and local features are extracted, and feature fusion is carried out to Improve identification accuracy.

Benefits of technology

Through the fusion of rich details and accurate features, the accuracy of pedestrian recognition is improved, especially in complex environments, which can more effectively capture pedestrian characteristics, improving the accuracy of recognition results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117746462B_ABST
    Figure CN117746462B_ABST
Patent Text Reader

Abstract

The present invention discloses a pedestrian re-identification method and device based on a complementary feature dynamic fusion network model, wherein the complementary feature dynamic fusion network model includes an HFE main branch and an auxiliary branch, wherein the HFE main branch includes a Vit network module and an NFC module, wherein the method includes obtaining a result vector of a target pedestrian from an image to be identified based on the Vit network module, and extracting a pedestrian global feature vector from the result vector; obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch; and determining a pedestrian re-identification result based on the pedestrian global feature vector and the pedestrian local feature vector. In this way, feature fusion is performed based on the Vit network module and the auxiliary branch module, thereby obtaining a more detailed and accurate local feature vector, and then re-identification is performed based on the global feature vector and the local feature vector, thereby improving the accuracy of the pedestrian re-identification result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a pedestrian re-identification method and device based on a complementary feature dynamic fusion network model. Background Art

[0002] Person Re-identification (ReID) is different from face recognition, iris recognition, fingerprint recognition and other identification methods with unique identification. Pedestrian Re-identification mainly relies on the appearance characteristics of pedestrians, which are closely related to appearance characteristics such as clothing and posture. It has a broader prospect in applications such as investigation and security and pedestrian behavior analysis. Since pedestrians can enter and exit freely within the range of the camera without being constrained by any conditions. Therefore, the captured pedestrian images are seriously affected by environmental factors such as shooting perspective differences, lighting changes, object occlusion, and noisy background, resulting in great differences in the appearance characteristics (resolution and posture, etc.) of the same pedestrian in different images. Even due to object occlusion, the complete pedestrian image cannot be captured. In addition, different pedestrians have many visual similarities in clothing color, style, etc., which adds more difficulties to identification.

[0003] There are many person re-identification technologies, but they often rely on sufficient pose-labeled images and powerful pose estimation networks, and are therefore easily limited by additional auxiliary models. Some person re-identification technologies also require attention through convolution operations, ignoring the global information and implicit relationships of the image, thus limiting the exploration of correlations between model learning features, making the accuracy of person re-identification not high enough. Summary of the invention

[0004] The present invention provides a pedestrian re-identification method and device based on a complementary feature dynamic fusion network model, aiming to improve the accuracy of pedestrian re-identification.

[0005] To achieve the above object, the present invention provides a pedestrian re-identification method based on a complementary feature dynamic fusion network model, characterized in that the complementary feature dynamic fusion network model includes a multi-level feature extraction (Hierarchical Feature Extraction, HFE) main branch and an auxiliary branch, wherein the HFE main branch includes a vision transformer (Vision Transformer, Vit) network module and a neighborhood feature constraint (Neighborhood Feature Constraint, NFC) module;

[0006] The pedestrian re-identification method based on the complementary feature dynamic fusion network model includes:

[0007] Obtaining a result vector of a target pedestrian from the image to be identified based on the Vit network module, and extracting a global feature vector of the pedestrian from the result vector;

[0008] Obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch;

[0009] A pedestrian re-identification result is determined based on the pedestrian global feature vector and the pedestrian local feature vector.

[0010] Optionally, the step of obtaining a result vector of a target pedestrian from the image to be identified based on the Vit network module, and extracting a global feature vector of the pedestrian from the result vector comprises:

[0011] Based on the Vit network module in the HFE main branch, a result vector of a pedestrian to be identified is obtained based on the image to be identified and basic information;

[0012] A pedestrian global feature vector is extracted from the result vector in stages based on a plurality of conversion blocks.

[0013] Optionally, the Vit network module in the HFE main branch obtains a result vector of a pedestrian to be identified based on the image to be identified and basic information, including:

[0014] Based on the convolution operation of the Vit network module, the image to be recognized is divided into a plurality of overlapping image blocks, and the image blocks are converted into vector sequences;

[0015] The position information code and the camera information code are added to the vector sequence in an element-by-element manner to obtain the result vector.

[0016] Optionally, before obtaining the splicing vector through the NFC module and extracting the pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch, the method further includes:

[0017] The three-dimensional features of the image to be identified are extracted based on the auxiliary branch, and the two-dimensional features corresponding to the three-dimensional features are integrated with the inheritance class vector obtained from the main branch to obtain the supplementary two-dimensional features.

[0018] Optionally, the auxiliary branch includes a convolutional neural network (CNN) module, a feature reshaping (DSF) module and a shared dynamic aggregation (SDA) module;

[0019] The extracting of the three-dimensional features of the image to be identified based on the auxiliary branch, and integrating the two-dimensional features corresponding to the three-dimensional features with the inheritance class vector obtained from the main branch to obtain the supplementary two-dimensional features includes:

[0020] Extracting three-dimensional features of the input image based on the bottleneck residual block of the CNN module;

[0021] Based on the DSF module, the dimensionality of the three-dimensional features is reduced to obtain two-dimensional hierarchical features;

[0022] The two-dimensional hierarchical features are concatenated to the inherited class vector inherited from the main branch to obtain a supplementary two-dimensional feature.

[0023] Optionally, the HFE main branch includes a Vit network module and an NFC module;

[0024] Obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch includes:

[0025] Extracting a target class vector from the target features output by the penultimate conversion block in the Vit network module based on the NFC module;

[0026] Splitting the vector sequence obtained by the Vit network module into a plurality of sub-vector sequences;

[0027] According to the target class vector, the sub-vector sequence is respectively concatenated to obtain a concatenated vector;

[0028] A pedestrian local feature vector is extracted from the concatenated vector and the complementary two-dimensional features input from the auxiliary branch in stages based on a shared transformation block.

[0029] Optionally, extracting the pedestrian local feature vector from the concatenated vector and the supplementary two-dimensional features input by the auxiliary branch in stages based on the shared conversion block comprises:

[0030] In the shallow stage, the supplementary two-dimensional features are interacted with the splicing vector in a one-to-one correspondence, and the interaction result is saved as a result vector, wherein the supplementary two-dimensional features include pedestrian texture information and edge texture information;

[0031] In the deep stage, the supplementary two-dimensional features are aggregated with the concatenated vector in a manner of aggregating adjacent convolutional layers to obtain a result vector, wherein the supplementary two-dimensional features include discriminative features.

[0032] Optionally, determining a pedestrian re-identification result based on the pedestrian global feature vector and the pedestrian local feature vector includes:

[0033] Concatenate the pedestrian global feature vector and the pedestrian local feature vector to obtain a pedestrian re-identification feature vector;

[0034] The pedestrian re-identification result is determined based on the similarity between the pedestrian re-identification feature vector and the target pedestrian feature vector.

[0035] Optionally, before obtaining a result vector of a target pedestrian from the image to be identified based on the Vit network module and extracting a pedestrian global feature vector from the result vector, the method further includes:

[0036] During the training phase, the complementary feature dynamic fusion network model is supervised based on the classification loss and the triplet loss to obtain a total loss function for optimizing the complementary feature dynamic fusion network model based on the total loss function.

[0037] In addition, the present invention also provides a pedestrian re-identification device based on a complementary feature dynamic fusion network model, comprising:

[0038] A global feature extraction module, used to obtain a result vector of a target pedestrian from the image to be identified based on the Vit network module, and extract a global feature vector of the pedestrian from the result vector;

[0039] A local feature extraction module, used to obtain a splicing vector through the NFC module, and extract a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch;

[0040] The recognition module is used to determine the pedestrian re-recognition result based on the pedestrian global feature vector and the pedestrian local feature vector.

[0041] Compared with the prior art, the present invention proposes a pedestrian re-identification method and device based on a complementary feature dynamic fusion network model. The complementary feature dynamic fusion network model includes a multi-level feature extraction main branch and an auxiliary branch. The HFE main branch includes a visual converter network module and a neighborhood feature constraint module. The method includes obtaining a result vector of a target pedestrian from an image to be identified based on a Vit network module, and extracting a pedestrian global feature vector from the result vector; obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the complementary two-dimensional features input by the auxiliary branch; and determining a pedestrian re-identification result based on the pedestrian global feature vector and the pedestrian local feature vector. In this way, feature fusion is performed based on the Vit network module and the auxiliary branch module, thereby obtaining a more detailed and accurate local feature vector, and then re-identification is performed based on the global feature vector and the local feature vector, thereby improving the accuracy of the pedestrian re-identification result. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1Schematic diagram of the hardware structure of a pedestrian re-identification device based on a complementary feature dynamic fusion network model according to various embodiments of the present invention;

[0043] Figure 2 It is a flowchart of a first embodiment of a pedestrian re-identification method based on a complementary feature dynamic fusion network model of the present invention;

[0044] Figure 3 It is a structural diagram of a complementary feature dynamic fusion network model involved in the pedestrian re-identification method based on a complementary feature dynamic fusion network model of the present invention;

[0045] Figure 4 It is a first detailed flow chart of the first embodiment of the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention;

[0046] Figure 5 It is another flowchart of the first embodiment of the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention;

[0047] Figure 6 It is a schematic diagram of a feature reshaping module involved in a pedestrian re-identification method based on a complementary feature dynamic fusion network model of the present invention;

[0048] Figure 7 It is a flowchart diagram of a second refinement of the first embodiment of the pedestrian re-identification method based on a complementary feature dynamic fusion network model of the present invention;

[0049] Figure 8 It is a pedestrian re-identification result image involved in the first embodiment of the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention;

[0050] Fig. 9 It is a functional module diagram of the first embodiment of the pedestrian re-identification device based on the complementary feature dynamic fusion network model of the present invention.

[0051] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0053] The pedestrian re-identification device based on the complementary feature dynamic fusion network model mainly involved in the embodiments of the present invention refers to a network connection device that can achieve network connection. The pedestrian re-identification device based on the complementary feature dynamic fusion network model can be a server, a cloud platform, etc.

[0054] Reference Figure 1 , Figure 1It is a schematic diagram of the hardware structure of the pedestrian re-identification device based on the complementary feature dynamic fusion network model involved in each embodiment of the present invention. In the embodiment of the present invention, the pedestrian re-identification device based on the complementary feature dynamic fusion network model may include a processor 1001 (such as a central processing unit, CPU), a communication bus 1002, an input port 1003, an output port 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the input port 1003 is used for data input; the output port 1004 is used for data output, and the memory 1005 can be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 can optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art can understand that Figure 1 The hardware structure shown in the figure does not constitute a limitation of the present invention, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0055] Continue to refer to Figure 1 , Figure 1 The memory 1005 as a readable storage medium may include an operating system, a network communication module, an application module, and a pedestrian re-identification program based on a complementary feature dynamic fusion network model. Figure 1 In the example, the network communication module is mainly used to connect to the server and perform data communication with the server; and the processor 1001 is used to call the pedestrian re-identification program based on the complementary feature dynamic fusion network model stored in the memory 1005, and perform the following operations:

[0056] Obtaining a result vector of a target pedestrian from the image to be identified based on the Vit network module, and extracting a global feature vector of the pedestrian from the result vector;

[0057] Obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch;

[0058] A pedestrian re-identification result is determined based on the pedestrian global feature vector and the pedestrian local feature vector.

[0059] like Figure 2 As shown, the first embodiment of the present invention proposes a pedestrian re-identification method based on a complementary feature dynamic fusion network model, the method comprising:

[0060] Step S101, obtaining a result vector of a target pedestrian from an image to be identified based on the Vit network module, and extracting a global feature vector of the pedestrian from the result vector;

[0061] In order to improve the accuracy of pedestrian recognition, a dynamic fusion network model based on complementary features is proposed by utilizing the representation modeling capabilities of Convolutional Neural Networks (CNN) and Vision Transformer (ViT). The dynamic fusion network model based on complementary features includes a main branch and an auxiliary branch of Hierarchical Feature Extraction (HFE). The HFE main branch includes a Vision Transformer (Vit) network module and a Neighborhood Feature Constraint (NFC) module; the auxiliary branch includes a Convolutional Neural Network (CNN) module, a Down Sampling and Flatten (DSF) module and a Shared Dynamic Aggregation (SDA) module. CNN convolutional neural network has representation learning capabilities and can perform translation-invariant classification of input images according to its hierarchical structure. Vit can model global information and has excellent performance. See the dynamic fusion network model based on complementary features. Figure 3 , Figure 3 This is a structural diagram of a complementary feature dynamic fusion network model involved in the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention. The network model consists of two branches, and each branch extracts different feature preferences. The auxiliary branch provides feature complementarity for the main branch. In order to ensure the training effect of the complementary feature dynamic fusion network, a neighborhood feature constraint module (NFC) is additionally introduced in the HFE main branch to allow the network to focus on extracting local key features.

[0062] See also Figure 4 , Figure 4 : is a first detailed flow chart of the first embodiment of the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention, such as Figure 4 As shown, the step S101 includes:

[0063] Step S1011, obtaining a result vector of a pedestrian to be identified based on the image to be identified and basic information based on the Vit network module in the HFE main branch;

[0064] Based on the convolution operation of the Vit network module, the image to be recognized is divided into a plurality of overlapping image blocks, and the image blocks are converted into vector sequences;

[0065] The position information code and the camera information code are added to the vector sequence in an element-by-element manner to obtain the result vector.

[0066] In the Vit network of the HFE main branch, for a given image to be recognized Wherein, H, W, and C represent the height, width, and number of channels of the image to be identified, respectively. The image to be identified x is cut into N overlapping image blocks of size P×P by using a convolution operation, and the image blocks are converted into a vector sequence for subsequent feature correlation calculation based on the vector sequence. In this embodiment, the vector sequence is represented as:

[0067]

[0068] Where N represents the length of the vector sequence, S represents the step size of the convolution operation, S<P; (P 2 ×C) represents the dimension of the vector sequence.

[0069] Then use a trainable linear mapping to transform (P 2 ×C)-dimensional vector sequence is mapped to d-dimensional, and the value of d in this embodiment is 2.

[0070] After obtaining the d-dimensional vector sequence, the learnable one-dimensional position information code is added to the vector sequence by element addition according to the standard Vision Transformer, and the camera information code is added in the same way to obtain the result vector. At this time, the result vector can be expressed as

[0071] Step S1012: extracting pedestrian global feature vectors from the result vector in stages based on multiple conversion blocks.

[0072] After obtaining the result vector, the 12 Transformer Blocks in the Vit network are used to extract the global features of pedestrians in stages. Figure 3 , in the shallow SDA of the Vit network 1-1×9 , and deep SDA 2-1×2 、SDA 3-1×1 The CLS (classification) of the Vit main branch encodes the statistical characteristics of the set, which serves as the overall representation of the pedestrian identity.

[0073] Step S102, obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch;

[0074] This embodiment requires obtaining supplementary two-dimensional features based on auxiliary branches in advance, see Figure 5 , Figure 5 is another flow chart of the first embodiment of the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention, such as Figure 5 As shown, before step S102, the following steps are also included:

[0075] Step S1002: extracting the three-dimensional features of the image to be identified based on the auxiliary branch, and integrating the two-dimensional features corresponding to the three-dimensional features with the inheritance class vector obtained from the main branch to obtain the supplementary two-dimensional features.

[0076] This embodiment extracts the three-dimensional features of the input image based on the bottleneck residual block of the CNN module;

[0077] Based on the DSF (feature reshaping) module, the three-dimensional features are reduced in dimension to obtain two-dimensional hierarchical features;

[0078] The two-dimensional hierarchical features are concatenated to the inherited class vector inherited from the main branch to obtain a supplementary two-dimensional feature.

[0079] The auxiliary branch includes a CNN convolutional neural network (CNN) module, a feature reshaping (Down Sampling and Flatten, DSF) module and a dynamic fusion (Shared Dynamic Aggregation, SDA) module. The auxiliary branch is composed of a CNN network stacked with bottleneck residual modules composed of small convolutions. In this embodiment, the features of different convolutional layers are represented as H i , h, w and c represent the length, width and number of channels of the feature map, respectively. In order to further improve the ability of the Vit network in the HFE main branch to select complementary features on the auxiliary branch and activate highly discriminative regional features, this embodiment proposes a shared dynamic aggregation module (Shared Dynamic Aggregation, SDA) composed of attention modules to provide HFE with detailed information of different scales.

[0080] In the specific implementation process, before the dynamic fusion of heterogeneous features, it is necessary to first solve the heterogeneous features (i.e., H in the auxiliary branch). i The difference in feature dimension between the result vector Z in the main branch and the feature captured in the auxiliary branch It is three-dimensional, and the length h, width w and number of channels c are different in different stages. The features in HFE are two-dimensional. And it always remains unchanged, so the hierarchical features of different stages of the auxiliary branch must go through the feature reshaping module (Down Sampling and Flatten, DSF down sampling and flattening) before entering any SDA module, converting the current three-dimensional hierarchical features H i Converted into a fixed two-dimensional feature f i See also Figure 6 , Figure 6 : is a schematic diagram of a feature reconstruction module involved in the pedestrian re-identification method based on a complementary feature dynamic fusion network model of the present invention. The feature reconstruction module is as follows: Figure 6 What is shown includes the convolution layer (normalized afterwards), the average pooling layer Avgpool, the flattening Flatten and the transpose Transpose operations.

[0081] Two-dimensional feature f i First, inherit the class vector in the main branch, and define the inherited class vector as the inherited class vector (ICLS), and concatenate the inherited class vector to each two-dimensional feature f i To obtain complementary two-dimensional features, the fusion of CNN network and Vit network is realized, and the fusion of local features and global features is realized.

[0082] After the supplementary two-dimensional features are obtained, a pedestrian local feature vector is extracted based on the concatenated vector and the supplementary two-dimensional features input by the auxiliary branch. Figure 7 , Figure 7 is a flowchart of the second refinement of the first embodiment of the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention; Figure 7 As shown, the step S102 includes:

[0083] Step S1021, obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch includes:

[0084] Step S1022, extracting a target class vector from the target features output by the penultimate conversion block in the Vit network module based on the NFC module;

[0085] Step S1023, dividing the vector sequence obtained by the Vit network module into a plurality of sub-vector sequences;

[0086] Step S1024, concatenating the target class vector with the sub-vector sequence to obtain a concatenated vector;

[0087] Step S1025 , extracting pedestrian local feature vectors from the concatenated vector and the supplementary two-dimensional features input from the auxiliary branch in stages based on the shared conversion block.

[0088] In order to extract local fine-grained features of pedestrians, this embodiment adds an NFC module in the HFE main branch. This embodiment extracts a target class vector from the target features output by the penultimate conversion block in the Vit network module based on the NFC module.

[0089] The vector sequence is divided into S groups along the length direction of the vector sequence to obtain S sub-vector sequences, corresponding to S local neighborhood features, where S refers to the proportion of the neighborhood feature area to the penultimate hidden layer feature of the Vit network in the HFE main branch. The sub-vector sequence of this embodiment is expressed as:

[0090]

[0091] The sub-vector sequence representing the local neighborhood features is concatenated with the target class vector to obtain a concatenated vector, which is expressed as:

[0092]

[0093] In this way, local neighborhood features including target class vector features are obtained.

[0094] The concatenated vector and the complementary two-dimensional feature input of the auxiliary branch are input into a shared conversion block, and the correlation of features in each local area is calculated separately. S pedestrian local feature vectors are extracted in stages based on the shared conversion block:

[0095] Specifically, at the shallow stage, the supplementary two-dimensional features are interacted with the splicing vector in a one-to-one correspondence, and the interaction result is saved as a result vector, wherein the supplementary two-dimensional features include pedestrian texture information and edge texture information;

[0096] In the deep stage, the supplementary two-dimensional features are aggregated with the concatenated vector in a manner of aggregating adjacent convolutional layers to obtain a result vector, wherein the supplementary two-dimensional features include discriminative features.

[0097] This embodiment learns features that are dependent and complementary to the class vector CLS in the SDA module. The SDA module consists of a multi-head attention mechanism, a multi-layer perceptron, layer normalization, and residual splicing.

[0098] At different stages of the Vit network of the main branch of HFE, the demand for complementary features is different. In the shallow layer of HFE, sufficient detail clues are very helpful in building high-quality body part relationships. The pedestrian texture information and edge texture information captured by the shallow layer of the auxiliary branch are relatively complete, and the pedestrian semantic information is clear. Therefore, for the first 9 TransformerBlocks of HFE, the complementary hierarchical features interact with the features output by the Transformer Block in HFE in a one-to-one correspondence, such as Figure 3 SDA 1-1 As shown. In the deep layer of HFE, the diverse discriminative features need to be concentrated on the rough parts of the pedestrian's body. At this time, the features extracted by the deep layer of the auxiliary branch are abstract, and different channels extract discriminative features of different modes of pedestrians. In the deep layer of the Vit network, the auxiliary branch features can activate the discriminative features in HFE and accelerate the convergence speed. This is achieved by aggregating the features of adjacent convolutional layers, as shown in Figure 3 SDA 2-1 and SDA 3-1 shown.

[0099] Step S103: determining a pedestrian re-identification result based on the pedestrian global feature vector and the pedestrian local feature vector.

[0100] Specifically, the pedestrian global feature vector and the pedestrian local feature vector are concatenated to obtain a pedestrian re-identification feature vector; and a pedestrian re-identification result is determined based on the similarity between the pedestrian re-identification feature vector and a target pedestrian feature vector.

[0101] In this embodiment, the obtained pedestrian global feature vector and pedestrian local feature vector are concatenated to obtain the final pedestrian re-identification feature vector. Each image to be identified is compared with the target pedestrian image for similarity to obtain a recognition distance matrix, and the image to be identified with the smallest distance to the target pedestrian image is regarded as the most likely candidate pedestrian. Usually, the Euclidean distance between two samples in the embedding space is calculated as the similarity, because the Euclidean distance is the simplest and easiest to understand.

[0102] In this embodiment, x is defined p and x g They represent the target pedestrian image (probe) and the image to be identified in the candidate set gallery, respectively. and They respectively represent the pedestrian re-identification feature vector of the target pedestrian image and the pedestrian re-identification feature vector of the image to be identified. and The Euclidean distance between pg , then:

[0103]

[0104] Re-identification based on the above method can obtain accurate recognition results. Figure 8 , Figure 8 is a pedestrian re-identification result image involved in the first embodiment of the pedestrian re-identification method based on the complementary feature dynamic fusion network model of the present invention; based on Figure 8 It can be seen from the recognition results that the number of erroneous images retrieved by the baseline network based on Vision Transformer (ViT-BoT) is greater than that of the model proposed in this invention. In addition, among the images retrieved by this application, the accuracy of the first few images (especially rank-5) is higher than that of the baseline network based on Vision Transformer (ViT-BoT). The average accuracy rate reaches 66.6%, which shows the effectiveness and accuracy of the pedestrian re-identification based on the complementary feature dynamic fusion network model proposed in this application.

[0105] The pedestrian re-identification task based on deep learning is mainly divided into two stages. The first stage is the training of the network, and the second stage is to use the trained model to extract the features of the target pedestrian and the pedestrian images in the candidate set for similarity measurement. In the training stage, this embodiment only sets a classifier for the HFE main branch based on the complementary feature dynamic fusion network model. The loss calculated by the main branch classifier is optimized for the entire network.

[0106] Specifically, before step S101, the complementary feature dynamic fusion network model is supervised based on the classification loss and the triplet loss in the training phase to obtain a total loss function for optimizing the complementary feature dynamic fusion network model based on the total loss function.

[0107] In the training stage, the classification (Identity Loss, ID) loss and triplet loss are used to supervise the dynamic fusion network model based on complementary features. The total loss function of the model is expressed as L, then:

[0108]

[0109] Where α is the balance factor, usually set to 1; L ID represents the classification loss, L tri represents the triplet loss, L tri (CLS) represents the triplet loss of CLS, represents the classification loss of local neighborhood features, The triplet loss representing the local neighborhood features, in the inference stage, concatenates the features in the HFE branch and the NFC sub-branch as the final representation of the pedestrian. According to the similarity matrix of the final representation, a list of candidates matching the pedestrian to be retrieved is obtained.

[0110] This embodiment uses the above scheme to obtain the result vector of the target pedestrian from the image to be identified based on the Vit network module, and extract the pedestrian's global feature vector from the result vector; obtain the splicing vector through the NFC module, and extract the pedestrian's local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch; determine the pedestrian re-identification result based on the pedestrian's global feature vector and the pedestrian's local feature vector. In this way, feature fusion is performed based on the Vit network module and the auxiliary branch module, thereby obtaining a more detailed and accurate local feature vector, and then re-identification is performed based on the global feature vector and the local feature vector, thereby improving the accuracy of the pedestrian re-identification result.

[0111] In addition, the present invention also proposes a pedestrian re-identification device based on a complementary feature dynamic fusion network model, referring to Fig. 9 , Fig. 9 is a functional module diagram of the first embodiment of the pedestrian re-identification device based on the complementary feature dynamic fusion network model of the present invention, such as Fig. 9 As shown, the pedestrian re-identification device based on the complementary feature dynamic fusion network model includes:

[0112] A global feature extraction module 10, used for obtaining a result vector of a target pedestrian from the image to be identified based on the Vit network module, and extracting a pedestrian global feature vector from the result vector;

[0113] A local feature extraction module 20, configured to obtain a splicing vector through the NFC module, and extract a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch;

[0114] The recognition module 30 is used to determine a pedestrian re-recognition result based on the pedestrian global feature vector and the pedestrian local feature vector.

[0115] Furthermore, the global feature extraction module 10 includes:

[0116] A result vector obtaining unit, configured to obtain a result vector of a pedestrian to be identified based on the image to be identified and basic information based on the Vit network module in the HFE main branch;

[0117] The global feature extraction unit is used to extract the pedestrian global feature vector from the result vector in stages based on multiple conversion blocks.

[0118] Furthermore, the result vector obtaining unit includes:

[0119] A vector sequence acquisition module, used for dividing the image to be identified into a plurality of overlapping image blocks based on the convolution operation of the Vit network module, and converting the image blocks into a vector sequence;

[0120] An adding unit is used to add the position information code and the camera information code to the vector sequence in an element-by-element addition manner to obtain the result vector.

[0121] Furthermore, the local feature extraction module 20 further includes:

[0122] The integration module is used to extract the three-dimensional features of the image to be identified based on the auxiliary branch, and integrate the two-dimensional features corresponding to the three-dimensional features with the inheritance class vector obtained from the main branch to obtain the supplementary two-dimensional features.

[0123] Furthermore, the integration module includes:

[0124] A three-dimensional feature extraction unit, used to extract three-dimensional features of an input image based on the bottleneck residual block of the CNN module;

[0125] A two-dimensional hierarchical feature obtaining unit, used for reducing the dimension of the three-dimensional feature based on the DSF module to obtain a two-dimensional hierarchical feature;

[0126] A concatenation unit, used for concatenating the two-dimensional hierarchical features to the inheritance class vector inherited from the main branch to obtain a supplementary two-dimensional feature;

[0127] Furthermore, the local feature extraction module 20 includes:

[0128] A target class vector extraction unit, configured to extract a target class vector from a target feature output by a penultimate conversion block in the Vit network module based on the NFC module;

[0129] A segmentation unit, used to segment the vector sequence obtained by the Vit network module into a plurality of sub-vector sequences;

[0130] A splicing vector obtaining unit, used for splicing the target class vector with the sub-vector sequence to obtain a splicing vector;

[0131] The extraction unit is used to extract the pedestrian local feature vector from the concatenated vector and the supplementary two-dimensional features input by the auxiliary branch in stages based on the shared conversion block.

[0132] Furthermore, the extraction unit comprises:

[0133] An interaction unit, used for interacting the supplementary two-dimensional features with the splicing vector in a one-to-one correspondence at a shallow stage, and saving the interaction result as a result vector, wherein the supplementary two-dimensional features include pedestrian texture information and edge texture information;

[0134] An aggregation unit is used to aggregate the supplementary two-dimensional features with the concatenated vector in a manner of aggregating adjacent convolutional layers in a deep stage to obtain a result vector, wherein the supplementary two-dimensional features include discriminative features.

[0135] Furthermore, the identification module 30 includes:

[0136] A feature concatenation unit, used for concatenating the pedestrian global feature vector and the pedestrian local feature vector to obtain a pedestrian re-identification feature vector;

[0137] The determination unit is used to determine the pedestrian re-identification result based on the similarity between the pedestrian re-identification feature vector and the target pedestrian feature vector.

[0138] Furthermore, the global feature extraction module 10 also includes:

[0139] The loss module is used to supervise the complementary feature dynamic fusion network model based on the classification loss and the triplet loss during the training phase to obtain a total loss function for optimizing the complementary feature dynamic fusion network model based on the total loss function.

[0140] In addition, the present invention also proposes a computer-readable storage medium, on which a pedestrian re-identification program based on a dynamic fusion network model of complementary features is stored. When the pedestrian re-identification program based on a dynamic fusion network model of complementary features is executed by a processor, the steps of the pedestrian re-identification method based on a dynamic fusion network model of complementary features are implemented as described above, which will not be repeated here.

[0141] Compared with the prior art, the present application proposes a pedestrian re-identification method and device based on a complementary feature dynamic fusion network model. The complementary feature dynamic fusion network model includes a multi-level feature extraction main branch and an auxiliary branch. The HFE main branch includes a visual converter network module and a neighborhood feature constraint module. The method includes obtaining a result vector of a target pedestrian from an image to be identified based on a Vit network module, and extracting a pedestrian global feature vector from the result vector; obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the complementary two-dimensional features input by the auxiliary branch; and determining a pedestrian re-identification result based on the pedestrian global feature vector and the pedestrian local feature vector. In this way, feature fusion is performed based on the Vit network module and the auxiliary branch module, thereby obtaining a more detailed and accurate local feature vector, and then re-identification is performed based on the global feature vector and the local feature vector, thereby improving the accuracy of the pedestrian re-identification result.

[0142] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A pedestrian re-identification method based on a complementary feature dynamic fusion network model, characterized in that: The complementary feature dynamic fusion network model includes a multi-level feature extraction HFE main branch and an auxiliary branch, wherein the HFE main branch includes a visual converter Vit network module and a neighborhood feature constraint NFC module; the auxiliary branch includes a convolutional neural network CNN module, a feature reshaping DSF module and a dynamic fusion SDA module; the pedestrian re-identification method based on the complementary feature dynamic fusion network model includes: Step 1, obtaining a result vector of a target pedestrian from the image to be identified based on the Vit network module, and extracting a global feature vector of the pedestrian from the result vector; The step 1 specifically includes: dividing the image to be identified into a number of overlapping image blocks based on the convolution operation of the Vit network module, and converting the image blocks into a vector sequence; adding the position information code and the camera information code to the vector sequence in an element-by-element manner to obtain the result vector; extracting the pedestrian global feature vector from the result vector in stages based on the 12 conversion blocks in the Vit network; Step 2, extracting the three-dimensional features of the image to be identified based on the auxiliary branch, and integrating the two-dimensional features corresponding to the three-dimensional features with the inheritance class vector obtained from the main branch to obtain a supplementary two-dimensional feature; The step 2 specifically comprises: extracting three-dimensional features of the input image based on the bottleneck residual block of the CNN module; reducing the dimension of the three-dimensional features based on the DSF module to obtain two-dimensional hierarchical features; splicing the two-dimensional hierarchical features to the inherited class vector inherited from the main branch to obtain supplementary two-dimensional features; Step 3, obtaining a splicing vector through the NFC module, and extracting a pedestrian local feature vector based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch; The step 3 is specifically as follows: based on the NFC module, a target class vector is extracted from the target features output by the penultimate conversion block in the Vit network module; the vector sequence obtained by the Vit network module is divided into a plurality of sub-vector sequences; the target class vector is spliced ​​with the sub-vector sequence to obtain a spliced ​​vector; at a shallow stage, the supplementary two-dimensional features are interacted with the spliced ​​vector in a one-to-one correspondence, and the interaction result is saved as a result vector, wherein the supplementary two-dimensional features include pedestrian texture information and edge texture information; at a deep stage, the supplementary two-dimensional features are aggregated with the spliced ​​vector in a manner of aggregating adjacent convolutional layers to obtain a result vector, wherein the supplementary two-dimensional features include discriminative features; Step 4, determining a pedestrian re-identification result based on the pedestrian global feature vector and the pedestrian local feature vector; The step 4 specifically comprises: concatenating the pedestrian global feature vector and the pedestrian local feature vector to obtain a pedestrian re-identification feature vector; and determining a pedestrian re-identification result based on the similarity between the pedestrian re-identification feature vector and the target pedestrian feature vector.

2. The method according to claim 1, characterized in that Before obtaining the result vector of the target pedestrian from the image to be identified based on the Vit network module and extracting the pedestrian global feature vector from the result vector, the method further includes: During the training phase, the complementary feature dynamic fusion network model is supervised based on the classification loss and the triplet loss to obtain a total loss function for optimizing the complementary feature dynamic fusion network model based on the total loss function.

3. A pedestrian re-identification device based on a complementary feature dynamic fusion network model, characterized in that: The complementary feature dynamic fusion network model includes a multi-level feature extraction HFE main branch and an auxiliary branch, wherein the HFE main branch includes a visual converter Vit network module and a neighborhood feature constraint NFC module; the auxiliary branch includes a convolutional neural network CNN module, a feature reshaping DSF module and a dynamic fusion SDA module; the device includes: A global feature extraction module, used to obtain a result vector of a target pedestrian from the image to be identified based on the Vit network module, and extract a global feature vector of the pedestrian from the result vector; The global feature extraction module includes: a result vector obtaining unit, which is used to obtain the result vector of the pedestrian to be identified based on the image to be identified and the basic information of the Vit network module in the HFE main branch; a global feature extraction unit, which is used to extract the pedestrian global feature vector from the result vector in stages based on multiple conversion blocks; the result vector obtaining unit includes: a vector sequence obtaining module, which is used to divide the image to be identified into a plurality of overlapping image blocks based on the convolution operation of the Vit network module, and convert the image blocks into a vector sequence; an adding unit, which is used to add the position information code and the camera information code to the vector sequence in an element-by-element addition manner to obtain the result vector; An integration module, used for extracting the three-dimensional features of the image to be identified based on the auxiliary branch, and integrating the two-dimensional features corresponding to the three-dimensional features with the inheritance class vector obtained from the main branch to obtain a supplementary two-dimensional feature; The integration module includes: a three-dimensional feature extraction unit, used to extract three-dimensional features of the input image based on the bottleneck residual block of the CNN module; a two-dimensional hierarchical feature acquisition unit, used to reduce the dimension of the three-dimensional features based on the DSF module to obtain two-dimensional hierarchical features; a splicing unit, used to splice the two-dimensional hierarchical features to the inherited class vector inherited from the main branch to obtain a supplementary two-dimensional feature; A local feature extraction module, used for obtaining a splicing vector through the NFC module, and extracting a local feature vector of a pedestrian based on the splicing vector and the supplementary two-dimensional features input by the auxiliary branch; The local feature extraction module includes: a target class vector extraction unit, which is used to extract a target class vector from the target features output by the penultimate conversion block in the Vit network module based on the NFC module; a segmentation unit, which is used to segment the vector sequence obtained by the Vit network module into a plurality of sub-vector sequences; a splicing vector acquisition unit, which is used to obtain a splicing vector by splicing the sub-vector sequence according to the target class vector; an extraction unit, which is used to extract the pedestrian local feature vector from the splicing vector and the supplementary two-dimensional features input by the auxiliary branch in stages based on the shared conversion block; the extraction unit includes: an interaction unit, which is used to interact the supplementary two-dimensional features with the splicing vector in a one-to-one correspondence at a shallow stage, and save the interaction result as a result vector, wherein the supplementary two-dimensional features include pedestrian texture information and edge texture information; an aggregation unit, which is used to aggregate the supplementary two-dimensional features with the splicing vector in a manner of aggregating adjacent convolutional layers at a deep stage to obtain a result vector, wherein the supplementary two-dimensional features include discriminative features; A recognition module, used for determining a pedestrian re-recognition result based on the pedestrian global feature vector and the pedestrian local feature vector; The recognition module includes: a feature splicing unit, used to splice the pedestrian global feature vector and the pedestrian local feature vector to obtain a pedestrian re-identification feature vector; a determination unit, used to determine the pedestrian re-identification result based on the similarity between the pedestrian re-identification feature vector and the target pedestrian feature vector.