Pedestrian re-identification method, system, medium and device based on cross-modal constraint

Abstract

The application discloses a pedestrian re-identification method and system based on cross-modal constraints, wherein the pedestrian re-identification method is trained by constructing a global feature extraction network for extracting global deep features of a depth image and an RGB image, a local feature extraction network for extracting local features from the deep features and a cross-modal transition network for extracting transition state features, and by fusing global loss and local loss to train a pedestrian re-identification system based on cross-modal constraints; and the system is used for pedestrian re-identification. The method can effectively utilize rich heterogeneous features in cross-modal image samples without prior knowledge of one-to-one corresponding cross-modal sample pairs, effectively extract more discriminative features in the depth image, and further improve the recognition effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a pedestrian re-identification method based on cross-modal constraints. BACKGROUND

[0002] Each pixel value in a depth image (D) describes the distance between the point and the shooting device in the scene. Such a modal image is not easily affected by light changes. When the light condition is poor at night or the like, a visible light (RGB) image cannot be obtained, and this type of image can be obtained as a substitute for the RGB image. After the depth image and the RGB image of two modalities are obtained, the RGB-D cross-modal pedestrian re-identification for realizing cross-modal pedestrian matching between the two images emerges as the times require. Given a pedestrian to be searched in the depth image, the RGB-D cross-modal pedestrian re-identification can retrieve the pedestrian from an RGB pedestrian candidate library. Compared with the pedestrian re-identification based on the RGB image, the RGB-D cross-modal pedestrian re-identification can be performed in the case of poor light condition such as night. Therefore, the task is more in line with the demand of the actual monitoring scene.

[0003] In the prior art, prior knowledge of a one-to-one cross-modal sample pair is usually required; and the rich heterogeneous features in the cross-modal images cannot be effectively utilized, and more discriminative features in the depth image cannot be effectively extracted. Therefore, there is still a lot of room for improvement in the recognition effect. SUMMARY

[0004] The application aims at the problems in the prior art, and provides a pedestrian re-identification method and system based on cross-modal constraints, which can effectively improve the recognition effect.

[0005] Technical scheme: The application discloses a pedestrian re-identification method based on cross-modal constraints, which comprises a training stage and a recognition stage.

[0006] S1, a global feature extraction network 100 is constructed, the global feature extraction network comprises a depth image global branch 110, an RGB image global branch 120 and a global loss calculation module 130; the depth image global branch 110 is used for extracting depth image global deep features D d , comprising a cascaded depth image shallow feature extraction module 111 and a depth image deep feature extraction module 112; the RGB image global branch 120 is used for extracting RGB image global deep features D r , comprising a cascaded RGB image shallow feature extraction module 121 and an RGB image deep feature extraction module 122; and the global loss calculation module 130 is used for calculating a global loss according to the depth image global deep features D d and the RGB image global deep features D rCalculate the global softmax loss L GS and global Triplet loss L GT ;

[0007] S2. Construct a local feature extraction network 200, which includes a depth map adaptive pooling module 210, an RGB image adaptive pooling module 220, and a local loss calculation module 230; the depth map adaptive pooling module 210 is used to extract local features T of the depth map based on the global deep features of the depth map. d The RGB image adaptive pooling module 220 is used to perform global deep feature D of the RGB image. r Extracting local features T from RGB image r The local loss calculation module 230 is used to calculate the local loss based on the local features T of the depth map. d and RGB image local features T r Calculate the local softmax loss L LS and local triplet loss L LT ;

[0008] S3. Construct a cross-modal transition network 300, which includes a global graph model 310 and a graph softmax loss calculation module 320; the global graph model is used to obtain the global deep features D of the depth map. d and RGB image global deep feature D r Transitional state characteristics G d,r The graph softmax loss calculation module is used to calculate the transition state feature G. d,r Calculate the graph softmax loss L GR ;

[0009] S4. The global feature extraction network 100, the local feature extraction network 200 and the cross-modal transition network 300 are iteratively trained using the training set. During the training process, each batch of samples includes images of M pedestrians, where each pedestrian includes N depth maps and N RGB images.

[0010] The objective of the iterative training is to minimize the first loss function:

[0011]

[0012] The superscript t represents the number of epochs in the iterative training. L represents the global softmax loss from iteration 1 to t-1. GS and global Triplet loss L GT The average value;

[0013] The identification phase includes:

[0014] S5. Input the RGB images in the candidate set into the global branch 120 of the RGB image sequentially to obtain the corresponding global deep feature D of the RGB image. r,a and RGB image local features T r,a D r,a and T r,a After concatenation, candidate image features F are formed. a =[D r,a ,T r,a ], a = 1, 2, ..., Num, where Num is the number of RGB images in the candidate set; the depth image to be identified is input into the global branch 110 of the depth map to obtain the corresponding global deep feature D′ of the depth map. d and local features T′ of the depth map d , D′ d and T′ d After concatenation, the features of the image to be identified are F′=[D′ d ,T′ d ]; Calculate F′ and F a The Euclidean distance between the two images is used to select the candidate image corresponding to the minimum Euclidean distance as the recognition result of the image to be recognized.

[0015] Further, the input to the shallow feature extraction module 111 of the depth map is the depth map, which is then processed by the point distribution map extraction module to obtain a point distribution map; the point distribution map is processed by the point attention model to obtain a point attention map; the point attention map is processed by the point feature extraction module to obtain point features; the point distribution map and the point features are concatenated and then input into a multilayer perceptron to obtain the global shallow feature S of the depth map. d .

[0016] Furthermore, the global graph model 310 consists of R graph convolutional layers, each of which includes 2N nodes, where N is the number of depth maps of a pedestrian in a batch of samples during the training process.

[0017] The i-th node in layer r and the j-th node The edges between them form an adjacency matrix.

[0018]

[0019] in For calculation and The similarity between nodes; Conv1() is a convolutional layer with a 1×1 kernel, used to encode the similarity between two nodes; i,j=1,2,…,2N,r=1,2,…,R;

[0020] The i-th node in layer r The formula for calculation is:

[0021] W r-1 These are the parameters of the (r-1)th layer in the global graph model;

[0022] The N depth maps and N RGB maps corresponding to the same pedestrian are processed by a global feature extraction network 100 to obtain N global deep features and N RGB global deep features, which are then used as the values ​​of nodes in the first graph convolutional layer. The 2N nodes of the Rth graph convolutional layer are then connected to obtain the transition state feature G between the global deep features of the depth map and the global deep features of the RGB map. d,r .

[0023] Furthermore, the local feature extraction network 200 also includes a depth map local spatial context extraction module 211 and an RGB map local spatial context extraction module 221;

[0024] The depth map local spatial context extraction module 211 is used to extract local features T of the depth map. d Enhancement is performed to obtain enhanced local features L in the depth map. d The RGB image local spatial context extraction module 221 is used to extract local features T of the RGB image. r Enhancement is performed to obtain enhanced local features L in the RGB image. r The local loss calculation module 230 calculates the loss based on L. d and L r Calculate the local softmax loss L LS and local triplet loss L LT ;

[0025] The depth map local spatial context extraction module 211 extracts the local features T of the depth map. d Adding the depth map location encoding vector to obtain the local fusion feature Seq of the depth map. d Seq d After processing by K cascaded depth map enhancement sub-modules, the depth map enhancement local feature L is obtained. d ;

[0026] The input to the first depth map enhancement submodule is Seq. d Seq d Three sequences were obtained by mapping through three fully connected layers respectively. and and The query, key, and value are respectively input into a first multi-head attention network. Enhanced local features are obtained after processing by the first multi-head attention network, which has N1 parallel attention units. The enhanced local features are then compared with the input Seq. dThe sequence Seq′ is obtained by residual connection and regularization. d Seq′ d After multilayer perceptron and Seq′ d The output sequence of the first depth map enhancement submodule is obtained through residual connection and regularization; the output sequence is then processed by the second to the Kth depth map enhancement submodules to finally output the depth map enhanced local feature L. d .

[0027] Furthermore, the local feature extraction network 200 also includes a cross-modal attention information extraction module 240 and a local contrastive loss function calculation module 250;

[0028] The cross-modal attention information extraction module 240 extracts local features T from the depth map. d and RGB image local features T r After processing by the cross-modal enhancement submodule, the relationship feature C between the depth map and the RGB map is obtained. d,r The cross-modal enhancement submodule transforms the local features T of a modality... m1 The sequence Seq is obtained after mapping through a fully connected layer. Q Local features T of another modality m2 The sequence Seq is obtained after mapping through two additional fully connected layers. K and Seq V ; m1, m2 ∈ {d, r} and m1 ≠ m2; Seq Q Seq K and Seq V The query, key, and value are respectively input into the second multi-head attention network. Enhanced cross-modal local features are obtained after processing by the second multi-head attention network, which has N² parallel attention units. The enhanced cross-modal local features are regularized to obtain the sequence Seq′. Seq′ is then passed through a multilayer perceptron and connected to it via a residual connection, followed by regularization, to obtain the output sequence C of the cross-modal enhancement submodule. d,r ;

[0029] The local contrast loss function calculation module 250 calculates the loss function based on the relationship characteristics C between the depth map and the RGB image. d,r Calculate the local contrast loss L LC ;

[0030] The goal of iterative training is to minimize the second loss function:

[0031]

[0032] On the other hand, the present invention also discloses a system for implementing the above-mentioned pedestrian re-identification method based on cross-modal constraints, comprising: a global feature extraction network 100, a local feature extraction network 200, and a cross-modal transition network 300;

[0033] The global feature extraction network 100 includes a depth map global branch 110, an RGB map global branch 120, and a global loss calculation module 130; the depth map global branch 110 is used to extract global deep features D from the depth map. d It includes a cascaded shallow feature extraction module 111 and a deep feature extraction module 112; the RGB image global branch 120 is used to extract the global deep feature D of the RGB image. r It includes a cascaded RGB image shallow feature extraction module 121 and an RGB image deep feature extraction module 122; the global loss calculation module 130 is used to calculate the global deep feature D of the depth map. d and RGB image global deep feature D r Calculate the global softmax loss L GS and global Triplet loss L GT ;

[0034] The local feature extraction network 200 includes a depth map adaptive pooling module 210, an RGB image adaptive pooling module 220, and a local loss calculation module 230; the depth map adaptive pooling module 210 is used to extract local features T of the depth map based on the global deep features of the depth map. d The RGB image adaptive pooling module 220 is used to perform global deep feature D of the RGB image. r Extracting local features T from RGB image r The local loss calculation module 230 is used to calculate the local loss based on the local features T of the depth map. d and RGB image local features T r Calculate the local softmax loss L LS and local triplet loss L LT ;

[0035] The cross-modal transition network 300 includes a global graph model 310 and a graph softmax loss calculation module 320; the global graph model is used to obtain global deep features D of the depth map. d and RGB image global deep feature D r Transitional state characteristics G d,r The graph softmax loss calculation module is used to calculate the transition state feature G. d,r Calculate the graph softmax loss L GR ;

[0036] The global feature extraction network (100), local feature extraction network (200) and cross-modal transition network (300) are optimized for parameters through iterative training. During the iterative training process, each batch of samples includes images of M pedestrians, where each pedestrian includes N depth maps and N RGB images.

[0037] The objective of the iterative training is to minimize the first loss function:

[0038]

[0039] The superscript t represents the number of epochs in the iterative training. L represents the global softmax loss from iteration 1 to t-1. GS and global Triplet loss L GT The average value.

[0040] The present invention also discloses a computer-readable storage medium storing computer instructions, which execute the above-described pedestrian re-identification method when the computer instructions are run.

[0041] The present invention also discloses a pedestrian re-identification device, including a processor and a storage medium, wherein the storage medium is a computer-readable storage medium; the processor loads and executes the instructions and data in the storage medium to implement the above-mentioned pedestrian re-identification method.

[0042] Beneficial effects: The pedestrian re-identification method and system based on cross-modal constraints disclosed in this invention have the following advantages:

[0043] 1. This approach models the relationships between global features of images of the same category but different modalities by constructing a cross-modal global graph model. It integrates discriminative information from heterogeneous modalities to refine the representations of graph nodes as transitional states. Recognition constraints are imposed on these transitional states to maintain their consistency, thereby building a bridge between the two modalities.

[0044] 2. The shallow feature extraction module of the depth map uses a point attention model to perform point-by-point operations on the point distribution map to obtain a point attention map. Then, a point feature extractor is applied to extract point features. This feature extraction module is more suitable for feature extraction from irregular depth maps. It enhances the feature extraction capability of the depth map by fully encoding the sub-pixel channel information of discriminative points.

[0045] 3. Add a depth map local spatial context extraction module and an RGB map local spatial context extraction module to the local feature extraction network to extract self-attention information from the depth map and RGB map, thereby enhancing the representation ability of image features.

[0046] 4. The cross-modal attention information extraction module is used to obtain the interrelationships between local features of cross-modal image pairs. Matching constraints are applied to these relational features to make the relational features of heterogeneous image pairs of the same pedestrian larger, while the relational features of heterogeneous image pairs of different pedestrians smaller, thereby reducing the difference between the two images. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the pedestrian re-identification system based on cross-modal constraints disclosed in Example 1;

[0048] Figure 2 This is a schematic diagram of the components of the shallow feature extraction module for the depth map;

[0049] Figure 3 This is a schematic diagram of the components of the global loss calculation module;

[0050] Figure 4 Example 2 discloses a schematic diagram of the composition of a pedestrian re-identification system based on cross-modal constraints;

[0051] Figure 5(a) is a schematic diagram of the composition of the depth map local spatial context extraction module;

[0052] Figure 5(b) is a schematic diagram of the composition of the depth map enhancement submodule;

[0053] Figure 6 Example 3 discloses a schematic diagram of the composition of a pedestrian re-identification system based on cross-modal constraints;

[0054] Figure 7(a) is a schematic diagram of the cross-modal attention information extraction module;

[0055] Figure 7(b) is a schematic diagram of the composition of the cross-modal enhancement submodule;

[0056] Figure 8 Example 4 discloses a schematic diagram of the composition of a pedestrian re-identification system based on cross-modal constraints;

[0057] Figure 9 This is a schematic diagram of the pedestrian re-identification device disclosed in this invention. Detailed Implementation

[0058] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0059] Example 1:

[0060] This embodiment discloses a pedestrian re-identification method based on cross-modal constraints, including a training phase and an identification phase. The training phase includes:

[0061] Step 1: Construct a global feature extraction network 100, a local feature extraction network 200, and a cross-modal transition network 300 respectively, forming a pedestrian re-identification system based on cross-modal constraints, as shown below. Figure 1 As shown.

[0062] The global feature extraction network includes a depth map global branch 110, an RGB map global branch 120, and a global loss calculation module 130; the depth map global branch 110 is used to extract global deep features D from the depth map. d It includes a cascaded shallow feature extraction module 111 and a deep feature extraction module 112; the RGB image global branch 120 is used to extract the global deep feature D of the RGB image. r It includes a cascaded RGB image shallow feature extraction module 121 and an RGB image deep feature extraction module 122; the global loss calculation module 130 is used to calculate the global deep feature D of the depth map. d and RGB image global deep feature D r Calculate the global softmax loss L GS and global Triplet loss L GT ;

[0063] In this embodiment, the RGB image shallow feature extraction module 121 uses a module consisting of the initial convolutional layer and blocks 1-2 of a ResNet-50 array connected in sequence to extract the global shallow features S corresponding to the input RGB image. r Each pixel in the depth map describes the depth value in the scene, revealing the 3D information of pedestrians. Typical convolutional layers perform calculations on a regular lattice, which is insufficient to represent the 3D information contained in the depth map. Therefore, this embodiment employs a different structure than the RGB image shallow feature extraction module to extract shallow features from the depth map. The depth map shallow feature extraction module 111 employs the following... Figure 2 The structure shown involves the following steps: the input depth map is processed by a point distribution map extraction module to obtain a point distribution map; the point distribution map is then processed by a point attention model to obtain a point attention map; the point attention map is processed by a point feature extraction module to obtain point features; the point distribution map and point features are concatenated and then input into a multilayer perceptron to obtain the global shallow features S of the depth map. dThe point distribution map extraction module consists of the initial convolutional layer and blocks 1-2 of the ResNet-50 array, connected sequentially. The point attention model comprises three 1×1 convolutional layers. Intuitively, discriminative points occupy a larger proportion in most channels. Therefore, the point attention model accumulates the point attention map along the channels, reducing the number of channels in the point attention map to 1. The top D1 points in the accumulated point attention map are selected as sampling points, and the distribution of these D1 sampling points is uneven. For each sampling point, the point feature extraction module extracts its point-wise features, specifically by linear interpolation of the four nearest points around the sampling point. Therefore, the point features obtained in this embodiment fully utilize the sub-pixel information of the selected point encoding in the channel dimension. The point distribution map and point features are concatenated and input into a multilayer perceptron to obtain the global shallow feature S of the depth map. d The shallow feature extraction module 111 for the depth image and the shallow feature extraction module 121 for the RGB image extract features of a specific modality from the two modalities of the image. The deep feature extraction module 112 for the depth image and the deep feature extraction module 122 for the RGB image have the same structure and share weights, and their structure consists of blocks 3-4 of the ResNet-50 network.

[0064] The structure of the global loss calculation module 130 is as follows: Figure 3 As shown, the deep features D of the global depth map d and global RGB deep feature D r The inputs are fed into an average pooling layer to obtain the global features F corresponding to the two modes. gd and F gr F gd and F gr As input to the Triplet loss function calculation module, the global Triplet loss L is calculated according to the literature: Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017. In defense of the triplet loss for person-identification. arXiv preprint arXiv:1703.07737 (2017). GT :

[0065] L GT =max(d p -d n +β,0) (1)

[0066] Where d p and d n, respectively, represent the distance between features of positive and negative sample pairs, and β is the boundary of the Triplet loss function. In this embodiment, β = 0.3.

[0067] F gd and F gr The batch normalization (BN) module and the fully connected layer are input sequentially to obtain the class features N. d and N r N d ∈R u N r ∈R u , where u is the number of pedestrian categories. N d and N r The inputs are respectively entered into the softmax loss function calculation module to calculate the softmax loss function. The specific steps are as follows:

[0068] First, calculate the probability of a training sample belonging to each class k, k∈{1,2,…,u}:

[0069]

[0070] Where N d,k For N d The value on the k-th channel; then the loss function L gs1 The calculation is as follows:

[0071]

[0072] Assuming g is the true class of the training samples, q(k) = 1 when k = g; otherwise, q(k) = 0. gs1 It can maximize the probability of the correct category.

[0073] Similarly, N r The softmax loss function L is calculated in the input softmax loss function calculation module. gs2 Finally, the global softmax loss L is obtained. GS :L GS =L gs1 +L gs2 .

[0074] The local feature extraction network 200 includes a depth map adaptive pooling module 210, an RGB image adaptive pooling module 220, and a local loss calculation module 230; the depth map adaptive pooling module 210 is used to extract local features T of the depth map based on the global deep features of the depth map. d The RGB image adaptive pooling module 220 is used to perform global deep feature D of the RGB image. r Extracting local features T from RGB image rThe local loss calculation module 230 is used to calculate the local loss based on the local features T of the depth map. d and RGB image local features T r Calculate the local softmax loss L LS and local triplet loss L LT ;

[0075] The depth map adaptive pooling module 210 performs global deep feature D on the depth map. d The specific processing includes:

[0076] First, D d The area is uniformly divided into multiple sub-blocks, each sub-block representing a local feature; in this embodiment, D is... d The local features are divided into nine blocks. Then, adaptive pooling is used to downsample each local feature. Specifically, a spatial distribution is adaptively learned for each local feature. After normalizing this distribution using the softmax function, this spatial distribution is used as the weight to downsample the local features. This automatically selects discriminative spatial portions from the local features to represent the local region, thereby increasing the proportion of discriminative regions in that local space. This pooling process outputs a corresponding local attention feature block for each local feature, resulting in nine local attention feature blocks that are concatenated to form the local feature T of the depth map. d .

[0077] The RGB image adaptive pooling module 220 performs global deep feature D on the RGB image. r Perform the same operation to obtain the local feature T of the RGB image. r .

[0078] Local loss calculation module 230 according to T d and T r Calculate the local softmax loss L LS and local triplet loss L LT Steps and global loss calculation module 130 calculate global softmax loss L GS and global Triplet loss L GT The steps are similar, the only difference being the input data.

[0079] The cross-modal transition network 300 includes a global graph model 310 and a graph softmax loss calculation module 320; the global graph model is used to obtain the global deep features D of the depth map. d and RGB image global deep feature D r Transitional state characteristics G d,r The graph softmax loss calculation module is used to calculate the transition state feature G. d,rCalculate the graph softmax loss L GR ;

[0080] RGB images consist of three channels, while depth maps contain only single-channel information. A significant gap exists between these two modalities, posing a major challenge to cross-modal recognition. To reduce the differences between the two types of images, this invention employs a global graph model to model the relationship between global features of the two types of images within the same category, providing a transitional state between the two modalities. The global graph model 310 consists of R layers of graph convolutional layers, each layer containing 2N nodes, where N is the number of depth maps of a pedestrian in a batch of samples during training.

[0081] The i-th node in layer r and the j-th node The edges between them form an adjacency matrix.

[0082] in For calculation and The similarity between nodes can be calculated by dot product or channel concatenation. In this embodiment, XY channel concatenation is used to calculate the similarity. Conv1() is a 1×1 convolutional layer used to encode the similarity between two nodes. It can halve the number of channels of the feature after channel concatenation. The exponential calculation form is used to normalize the similarity between two nodes.

[0083] After obtaining the relationship between two nodes in the same layer, the global graph model iteratively propagates the energies of all nodes to the representation of each node to update the representation of each node. It is able to consider the relationship between the node to be encoded and all other nodes when encoding the representation of each node. i,j=1,2,…,2N,r=1,2,…,R.

[0084] The i-th node in layer r The formula for calculation is:

[0085] W r-1 These are the parameters of the (r-1)th layer in the global graph model. Let j be the j-th node in layer r-1. The N global deep features and N RGB global deep features obtained from the N depth maps and N RGB images corresponding to the same pedestrian are processed by the global feature extraction network 100 and used as the values ​​of nodes in the first graph convolutional layer. After being processed by the second to Rth graph convolutional layers, the features of each node in the Rth layer contain both the relationships between images of the same modality within the same pedestrian category (ID) and the relationships between images of different modalities within the same pedestrian category (ID). Therefore, the processed features are used as transitional modal features between the two types of images to construct a bridge between them. That is, the 2N nodes of the Rth graph convolutional layer are connected to obtain the transitional state feature G between the global deep features of the depth map and the global deep features of the RGB image. d,r In this invention, R is set to 2.

[0086] The graph softmax loss calculation module 320 calculates the transition state feature G based on the transition state characteristics. d,r Calculate the graph softmax loss L GR Steps and L gs1 The calculations are similar, the only difference being the input data.

[0087] Step 2: Iteratively train the global feature extraction network 100, the local feature extraction network 200, and the cross-modal transition network 300 using the training set. During the training process, each batch of samples includes images of M pedestrians, where each pedestrian includes N depth maps and N RGB images.

[0088] The objective of the iterative training is to minimize the first loss function:

[0089]

[0090] The superscript t represents the number of epochs in the iterative training. L represents the global softmax loss from iteration 1 to t-1. GS and global Triplet loss L GT The average value; the softmax loss L in the figure. GR The coefficient is inversely proportional to the value of the global loss function. This operation allows the cross-modal relationship modeling of global features to gradually increase its proportion in the overall training process after these global features have become more discriminative.

[0091] The identification phase includes:

[0092] Step 3: Input the RGB images in the candidate set into the global branch 120 of the RGB image sequentially to obtain the corresponding global deep feature D of the RGB image. r,a and RGB image local features T r,a D r,a and T r,aAfter concatenation, candidate image features F are formed. a =[D r,a ,T r,a ], a = 1, 2, ..., Num, where Num is the number of RGB images in the candidate set; the depth image to be identified is input into the global branch 110 of the depth map to obtain the corresponding global deep feature D′ of the depth map. d and local features T′ of the depth map d , D′ d and T′ d After concatenation, the features of the image to be identified are F′=[D′ d ,T′ d ]; Calculate F′ and F a The Euclidean distance between the two images is used to select the candidate image corresponding to the minimum Euclidean distance as the recognition result of the image to be recognized.

[0093] Example 2:

[0094] The difference between this embodiment and Embodiment 1 is that the local feature extraction network 200 also includes a depth map local spatial context extraction module 211 and an RGB map local spatial context extraction module 221; for example Figure 4 As shown. Local features T of the depth map d and RGB image local features T r After enhancement by the depth map local spatial context extraction module 211 and the RGB image local spatial context extraction module 221 respectively, the enhanced local features L of the depth map are obtained. d Enhanced local features L in RGB images r Then, the local loss calculation module 230 calculates the loss based on the enhanced local features L. d and L r Calculate the local softmax loss L LS and local triplet loss L LT .

[0095] The depth map local spatial context extraction module 211 extracts the local features T of the depth map. d Adding the depth map location encoding vector to obtain the local fusion feature Seq of the depth map. d Seq d It integrates depth information and location information from local features. Depth map location encoding is based on the following literature: A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, ANGomez, The method described in Kaiser and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008, automatically learns this information. Seq d After processing by K cascaded depth map enhancement sub-modules, the depth map enhancement local feature L is obtained. d As shown in Figure 5(a); in this example, the K value is set to 5, that is, the depth map local fusion feature Seq d After 5 levels of enhancement, L was obtained. d .

[0096] As shown in Figure 5(b), the input to the first depth map enhancement submodule is Seq. d Seq d Three sequences were obtained by mapping through three different fully connected layers. and and The query (Q), key (K), and value (V) are input to a first multi-head attention network, respectively. Enhanced local features are obtained after processing by the first multi-head attention network, which has N1 parallel attention units; in this embodiment, N1 is set to 8. Each attention unit fuses content from different local features to refine the data. Each local representation not only preserves the original local spatial expression but also establishes relationships between different local spaces, embedding them into each local representation and thus enhancing the representational power of local features. Each attention unit, by integrating these spatial contexts, can obtain more discriminative local features. Furthermore, N1 parallel attention units can simultaneously acquire these discriminative local features in multiple mapping spaces. The first multi-head attention network combines the enhanced local features obtained by all attention units. Therefore, the local features contain richer information.

[0097] The enhanced local features output by the first multi-head attention network and the Seq of the input. d The sequence Seq′ is obtained by residual connection and regularization (Add & Norm). d Seq′ d After multilayer perceptron (MLP) and Seq' d The output sequence of the first depth map enhancement submodule is obtained through residual connection and regularization; the output sequence is then processed by the second to the Kth depth map enhancement submodules to finally output the depth map enhanced local feature L. d .

[0098] The structure of the RGB image local spatial context extraction module 221 is the same as that of the depth image local spatial context extraction module 211, that is, the local features T of the RGB image r Adding the positional encoding vector of the RGB image to obtain the local fusion feature Seq of the RGB image. r Seq r After 5 levels of enhancement, the enhanced local feature L of the RGB image is obtained. r .

[0099] Example 3:

[0100] The difference between this embodiment and Embodiment 1 is that the local feature extraction network 200 further includes a cross-modal attention information extraction module 240 and a local contrast loss calculation module 250; for example Figure 6 As shown.

[0101] As shown in Figure 7(a), the cross-modal attention information extraction module 240 extracts local features T from the depth map. d and RGB image local features T r After processing by the cross-modal enhancement submodule, the relationship feature C between the depth map and the RGB map is obtained. d,r The cross-modal enhancement submodule transforms the local features T of a modality... m1 The sequence Seq is obtained after mapping through a fully connected layer. Q Local features T of another modality m2 The sequence Seq is obtained after mapping through two additional fully connected layers. K and Seq V ; m1, m2 ∈ {d, r} and m1 ≠ m2; Seq Q Seq K and Seq V The query (Q), key (K), and value (V) are respectively input into the second multi-head attention network. After processing by the second multi-head attention network, enhanced cross-modal local features are obtained. The second multi-head attention network has N² parallel attention units. The enhanced cross-modal local features are regularized to obtain the sequence Seq′. Seq′ is then passed through a multilayer perceptron and connected to Seq′ via a residual connection, and after regularization, the output sequence of the cross-modal enhancement submodule is obtained, which is the relationship feature C between the depth map and the RGB image. d,r The structure of the cross-modal enhancement submodule is shown in Figure 7(b). The second multi-head attention network in the cross-modal enhancement submodule fuses locally discriminative features from different modalities to refine the Seq. QIn each local representation, i.e., in a cross-modal image pair, the influence of all local features of the other modality is considered in each local feature of one image. Each attention unit not only maintains the information of the original modality but also incorporates information from the other modality, integrating the information of both modalities as a heterogeneous image pair relational feature. Furthermore, N² parallel attention units can simultaneously acquire the relational features of these two modalities in multiple mapping spaces. The second multi-head attention module combines the relational features obtained by all units. Therefore, the relational features can contain richer information.

[0102] The local contrast loss function calculation module 250 calculates the loss function based on the relationship between the depth map and the RGB image. d,r Calculate the local contrast loss L LC The specific steps for the calculation are as follows:

[0103] (1) Relational characteristics C d,r After passing through a fully connected layer, multiple pairs of heterogeneous modal relation values ​​S are obtained. d,v S d , v ={S d,v,+ ,S d,v,1 ,S d,v,2 ,…,S d,v,R}; that is, S d,v Includes a positive sample pair of heterogeneous modal relation values ​​S d,v,+ The heterogeneous modal relationship values ​​S for R negative samples d,v,r r = 1, 2, ..., R;

[0104] (2) The local contrast loss function is:

[0105]

[0106] τ is the smoothing parameter for the label, which is set to 0.2 in this embodiment.

[0107] This loss function can reduce the difference between two modalities by maximizing the relational features of cross-modal positive sample pairs. At the same time, it can improve the discriminative power of global features by minimizing the relational values ​​of cross-modal negative sample pairs.

[0108] In this embodiment, the goal of iterative training is to minimize the second loss function L. t :

[0109]

[0110] L t The local contrast loss L in the calculation formula LC The coefficient is inversely proportional to the local loss. This operation allows the cross-modal relationship modeling of local features to gradually increase its proportion in the overall training after these local features have become more discriminative.

[0111] Example 4:

[0112] This embodiment combines the improvements made to Embodiment 1 by Embodiments 2 and 3, forming the following: Figure 8 The pedestrian re-identification system shown is based on cross-modal constraints.

[0113] In the logarithm calculation of the loss function in Examples 1-4, the base is usually a natural number e or 2.

[0114] The pedestrian re-identification method described in the above embodiments was tested in D-RGB mode on the public dataset BIWI, and compared with existing algorithms. The test used depth maps as the probe set and RGB images as the candidate set. The Euclidean distance between the features of the probe and candidate images was calculated, and the candidate sets were sorted from smallest to largest distance. Rank-1 represents the proportion of the first-ranked image in the candidate set that is the correct image. The test results are shown in Table 1.

[0115] Table 1: Experimental Results

[0116] Methods Rank-1 (%) ICMDL 7.1 Corr. Dict 11.3 Cross-modal distillation network 29.2 Example base 29.4 Example 0 30.6 Example 1 31.3 Example 2 32.8 Example 3 35.2 Example 4 41.3

[0117] In Table 1, ICMDL is used for identification using the method in Reference 1: "Peng Zhang, Jingsong Xu, Qiang Wu, Yan Huang, and Jian Zhang. 2019. Top-push constrained modality-adaptive dictionary learning for cross-modality person re-identification. IEEE Transactions on Circuits and Systems for Video Technology 30, 12(2019), 4554–4566."; Corr.Dict is used for identification using the method in Reference 2: "Jiaxuan Zhuo, Junyong Zhu, Jianhuang Lai, and XiaohuaXie. 2017. Person re-identification on heterogeneous camera network. In CCFChinese Conference on Computer Vision. Springer, 280–291."; Cross-modal distillation network is used for identification using the method in Reference 3: "Frank M Hafner, Amran Bhuiyan, Julian FP Kooij, and Eric Granger. 2019. RGB-depth cross-modal person re-identification." The identification method is described in the 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 1–8.

[0118] As can be seen from Table 1, the average recognition accuracy of the embodiments disclosed in this invention is superior to that of current state-of-the-art methods. Based on Embodiment 1, the network performance can be effectively improved by adding a degree map local spatial context extraction module, an RGB map local spatial context extraction module, a cross-modal attention information extraction module, and a local contrast loss calculation module.

[0119] The present invention also discloses a computer-readable storage medium storing computer instructions, which execute the above-described pedestrian re-identification method when the computer instructions are run.

[0120] This invention also discloses a pedestrian re-identification device, such as... Figure 9 As shown, it includes a processor 901 and a storage medium 902, wherein the storage medium is a computer-readable storage medium; the processor loads and executes the instructions and data in the storage medium to implement the above-mentioned pedestrian re-identification method.

Claims

1. A pedestrian re-identification method based on cross-modal constraint, comprising a training stage and an identification stage, characterized in that, The training phase comprises: S1, a global feature extraction network (100) is constructed, the global feature extraction network comprises a depth map global branch (110), an RGB image global branch (120) and a global loss calculation module (130); the depth map global branch (110) is used to extract depth map global deep layer feature D d , comprising a cascaded depth map shallow layer feature extraction module (111) and a depth map deep layer feature extraction module (112); the RGB image global branch (120) is used to extract RGB image global deep layer feature D r , comprising a cascaded RGB image shallow layer feature extraction module (121) and an RGB image deep layer feature extraction module (122); the global loss calculation module (130) is used to calculate global softmax loss L GS and global Triplet loss L GT according to depth map global deep layer feature D d and RGB image global deep layer feature D r ; S2. Construct a local feature extraction network (200), which includes a depth map adaptive pooling module (210), an RGB image adaptive pooling module (220), and a local loss calculation module (230); the depth map adaptive pooling module (210) is used to extract local features T of the depth map based on the global deep features of the depth map. d The RGB image adaptive pooling module (220) is used to perform global deep feature D of the RGB image. r Extracting local features T from RGB image r The local loss calculation module (230) is used to calculate the local loss based on the local features T of the depth map. d and RGB image local features T r Calculate the local softmax loss L LS and local triplet loss L LT ; S3. Construct a cross-modal transition network (300), which includes a global graph model (310) and a graph softmax loss calculation module (320); the global graph model is used to obtain the global deep features D of the depth map. d and RGB image global deep feature D r Transition state characteristics G between d,r The graph softmax loss calculation module (320) is used to calculate the transition state feature G. d,r Calculate the graph softmax loss L GR ; S4, iteratively training the global feature extraction network (100), the local feature extraction network (200) and the cross-modal transition network (300) using the training set, wherein each batch of samples in the training process includes M images of pedestrians, and each pedestrian includes N depth maps and N RGB images; The target of the iterative training is to minimize a first loss function: where the superscript t denotes the epoch number of the iterative training, the global softmax loss L GS and the global Triplet loss L GT the average value of the global softmax loss L The identification phase comprises: S5, input the RGB images in the candidate set into the RGB image global branch (120) in sequence to obtain corresponding RGB image global deep features D r,a and RGB image local features T r,a r,a and T r,a are connected to form candidate image features F a =[D r,a ,T r,a ], a=1, 2, …, Num, Num is the number of RGB images in the candidate set; the depth image to be identified is input into the depth image global branch (110) to obtain corresponding depth image global deep features D′ d and depth image local features T′ d d and T′ d are connected to form the to-be-identified image features F′=[D′ d ,T′ d ]; the Euclidean distance between F′ and F a is calculated, and the candidate image corresponding to the minimum Euclidean distance is selected as the recognition result of the to-be-identified image.​​ 2. The cross-modal constraint based pedestrian re-identification method according to claim 1, characterized in that, The input of the depth map shallow feature extraction module (111) is a depth map, the depth map is obtained by a point distribution map extraction module; the point distribution map is obtained by a point attention model; the point attention map is obtained by a point feature extraction module; the point distribution map and the point feature are connected and input into a multi-layer perception machine to obtain a depth map global shallow feature S d .

3. The cross-modal constraint based pedestrian re-identification method according to claim 1, characterized in that, The global graph model (310) is composed of R layers of graph convolution layers, each layer of graph convolution layer including 2N nodes, N being the number of depth maps of a pedestrian in a batch of samples in the training process; The i-th node P in the r-th layer i r and the j-th node The edge between the i-th node P in the r-th layer and the j-th node wherein P is calculated as i r and Conv1() is a convolution layer with 1x1 kernel to encode the similarity between two nodes. i, j = 1, 2, …, 2N, r = 1, 2, …, R; Pthe i-th node in the r-th layer i r The calculation formula is: W r-1 are parameters for the r-1th layer in the global graph model; The N depth maps and N RGB maps corresponding to the same pedestrian are taken as the values of the nodes in the first layer graph convolution layer, and 2N nodes of the Rth layer graph convolution layer are connected to obtain the transition state feature G between the depth map global deep features and the RGB map global deep features d,r .

4. The cross-modal constraint based pedestrian re-identification method according to claim 1, characterized in that, The local feature extraction network (200) further includes a depth map local spatial context extraction module (211) and an RGB image local spatial context extraction module (221); The depth map local spatial context extraction module (211) is configured to enhance the depth map local feature T d to obtain an enhanced depth map local feature L d The RGB image local spatial context extraction module (221) is configured to enhance the RGB image local feature T r to obtain an enhanced RGB image local feature L r The local loss calculation module (230) is configured to calculate a local softmax loss L d and a local Triplet loss L r based on L LS and L LT . The depth map local spatial context extraction module (211) extracts a depth map local feature T d and a depth map position encoding vector, to obtain a depth map local fusion feature Seq d ; Seq d is processed through K depth map enhancement sub-modules in series, to obtain a depth map enhanced local feature L d ; Wherein The input of the first depth map enhancement sub-module is Seq d , Seq d is mapped through three fully connected layers to obtain three sequences , and , which are respectively input into the first multi-head attention network as query, key and value, and the enhanced local features are obtained through the processing of the first multi-head attention network, wherein the first multi-head attention network has N1 parallel attention units; the enhanced local features and the input Seq d are connected through a residual connection and are regularized to obtain a sequence Seq′ d ; Seq′ d is input into a multi-layer perceptron together with Seq′ d , which are connected through a residual connection and are regularized to obtain the output sequence of the first depth map enhancement sub-module; the output sequence is processed by the second depth map enhancement sub-module to the Kth depth map enhancement sub-module, and finally the depth map enhanced local features L d are output.

5. The cross-modal constraint based pedestrian re-identification method according to claim 1, characterized in that, The local feature extraction network (200) further includes a cross-modal attention information extraction module (240) and a local contrast loss function calculation module (250); The cross-modal attention information extraction module (240) extracts local features T from the depth map. d and RGB image local features T r After processing by the cross-modal enhancement submodule, the relationship feature C between the depth map and the RGB map is obtained. d,r ;in The cross-modal enhancement sub-module maps the local features T m1 of one modality to obtain a sequence Seq Q , and maps the local features T m2 of another modality to obtain a sequence Seq K and Seq V ; m1, m2 ∈ {d, r} and m1 ≠ m2; Seq Q , Seq K and Seq V are respectively input into a second multi-head attention network as a query, a key and a value, and the cross-modal local features are enhanced through processing of the second multi-head attention network, wherein the second multi-head attention network has N2 attention units in parallel; the enhanced cross-modal local features are regularized to obtain a sequence Seq′; and Seq′ is input into a multi-layer perception and connected to Seq′ through a residual connection and regularized to obtain an output sequence C d,r of the cross-modal enhancement sub-module. The local contrast loss function calculation module (250) calculates a local contrast loss L according to the relationship feature C between the depth map and the RGB image d,r The local contrast loss L is calculated as follows: LC ; The target of the iterative training is to minimize a second loss function:

6. A cross-modal constraint based pedestrian re-identification system, characterized in that, Comprise: The global feature extraction network (100), the local feature extraction network (200) and the cross-modal transition network (300); The global feature extraction network (100) comprises a depth map global branch (110), an RGB image global branch (120) and a global loss calculation module (130); the depth map global branch (110) is used for extracting depth map global deep layer features D d , comprising a cascaded depth map shallow layer feature extraction module (111) and a depth map deep layer feature extraction module (112); the RGB image global branch (120) is used for extracting RGB image global deep layer features D r , comprising a cascaded RGB image shallow layer feature extraction module (121) and an RGB image deep layer feature extraction module (122); the global loss calculation module (130) is used for calculating a global softmax loss L d and a global Triplet loss L r according to the depth map global deep layer features D GS and the RGB image global deep layer features D GT ; The local feature extraction network (200) comprises a depth map adaptive pooling module (210), an RGB image adaptive pooling module (220), and a local loss calculation module (230); the depth map adaptive pooling module (210) is used for extracting depth map local features T d from depth map global deep features D r ; the RGB image adaptive pooling module (220) is used for extracting RGB image local features T r from RGB image global deep features D d ; and the local loss calculation module (230) is used for calculating a local softmax loss L r and a local Triplet loss L LS from the depth map local features T LT and the RGB image local features T LT . The cross-modal transition network (300) includes a global graph model (310) and a graph softmax loss calculation module (320); the global graph model is used to obtain the global deep features D of the depth map. d and RGB image global deep feature D r Transition state characteristics G between d,r The graph softmax loss calculation module is used to calculate the transition state feature G. d,r Calculate the graph softmax loss L GR ; The global feature extraction network (100), the local feature extraction network (200) and the cross-modal transition network (300) are iteratively trained for parameter optimization, and each batch of samples in the iterative training process includes M images of pedestrians, and each pedestrian includes N depth maps and N RGB images; The target of the iterative training is to minimize a first loss function: where the superscript t denotes the epoch number of the iterative training, represents the average of the global softmax loss L GS and the global Triplet loss L GT over the first t - 1 iterations.

7. The cross-modal constraint based pedestrian re-identification system according to claim 6, wherein, The local feature extraction network (200) further includes a depth map local spatial context extraction module (211) and an RGB image local spatial context extraction module (221); The depth map local spatial context extraction module (211) is used to extract local features T from the depth map. d Enhancement is performed to obtain enhanced local features L in the depth map. d The RGB image local spatial context extraction module (221) is used to extract local features T of the RGB image. r Enhancement is performed to obtain enhanced local features L in the RGB image. r The local loss calculation module (230) calculates the loss based on L. d and L r Calculate the local softmax loss L LS and local triplet loss L LT ; The depth map local spatial context extraction module (211) extracts a depth map local feature T d and a depth map position encoding vector, to obtain a depth map local fusion feature Seq d ; Seq d is processed through K depth map enhancement sub-modules in series, to obtain a depth map enhanced local feature L d ; Wherein The input to the first depth map enhancement submodule is Seq d Seq d Three sequences were obtained by mapping through three fully connected layers respectively. and and The enhanced local features are obtained after passing through a first multi-head attention network, which has N1 parallel attention units; the enhanced local features are then compared with the input Seq. d The sequence Seq′ is obtained by residual connection and regularization. d Seq′ d After multilayer perceptron and Seq′ d The output sequence of the first depth map enhancement submodule is obtained through residual connection and regularization; the output sequence is then processed by the second to the Kth depth map enhancement submodules to finally output the depth map enhanced local feature L. d .

8. The cross-modal constraint based pedestrian re-identification system according to claim 6, wherein, The local feature extraction network (200) further includes a cross-modal attention information extraction module (240) and a local contrast loss calculation module (250); The cross-modal attention information extraction module (240) extracts local features T from the depth map. d and RGB image local features T r After processing by the cross-modal enhancement submodule, the relationship feature C between the depth map and the RGB map is obtained. d,r ;in The cross-modal enhancement sub-module maps the local features T m1 of one modality to obtain a sequence Seq Q , and maps the local features T m2 of another modality to obtain a sequence Seq K and a sequence Seq V ; m1, m2 ∈ {d, r} and m1 ≠ m2; Seq Q , Seq K and Seq V are respectively input into a second multi-head attention network as a query, a key and a value, and the cross-modal local features are enhanced through processing of the second multi-head attention network, wherein the second multi-head attention network has N2 attention units in parallel; the enhanced cross-modal local features are regularized to obtain a sequence Seq′; and Seq′ is input into a multi-layer perception and connected to Seq′ through a residual connection and regularized to obtain an output sequence C d,r of the cross-modal enhancement sub-module. The local contrast loss function calculation module (250) calculates a local contrast loss L according to the relationship feature C between the depth map and the RGB image d,r The local contrast loss L is calculated as follows: LC ; The target of the iterative training is to minimize a second loss function:

9. A computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions run to perform the pedestrian re-identification method of any one of claims 1 to 5.

10. A pedestrian re-identification device, comprising: Comprise a processor and a storage medium, the storage medium is the computer readable storage medium of claim 9; the processor loads and executes the instructions and data in the storage medium for realizing the pedestrian re-identification method of any one of claims 1 to 5. Comprise a processor and a storage medium, the storage medium is the computer readable storage medium of claim 9; the processor loads and executes the instructions and data in the storage medium for realizing the pedestrian re-identification method of any one of claims 1 to 5.

Images