A cross-modal person re-identification method based on adversarial learning of global and local features
By introducing attention mechanism and adversarial learning modules in cross-modal pedestrian re-identification, deep features are extracted and modal differences are reduced, and the problems of insufficient feature extraction and large modal differences in the prior art are solved, and recognition and retrieval is improved rate and characteristic consistency.
Patent Information
- Application Number
- CN202210493056.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-07
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-05-07
AI Technical Summary
The existing cross-modal pedestrian re-identification technology cannot fully extract global and local features, resulting in poor feature discrimination and cannot effectively eliminate the modal differences between visible light images and infrared images.
Adversarial learning methods based on global and local features are adopted, and deep features are extracted and modal differences are reduced by introducing attention mechanisms and adversarial learning modules. The specific steps include using the attention module to enhance feature extraction, the local feature learning module improves local feature identification, the predictive consistency module maintains the consistency between global features and local features, and reduces the modal differences between visible and infrared features through the adversarial learning module. .
The search rate of cross-modal pedestrian re-identification is improved, feature extraction ability is enhanced, discriminant consistency between global and local features, and the difference between visible light and infrared features is effectively reduced.
Smart Images

Figure CN115063832B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and relates to a method for feature extraction and processing and reducing modal differences, and in particular to a cross-modal pedestrian re-identification method based on adversarial learning of global and local features. Background Art
[0002] Person re-identification (Re-ID) is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. It is considered a sub-problem of image retrieval. Since person re-identification has broad application prospects in the field of computer vision and actual video surveillance applications, it has attracted more and more attention from scholars. In recent years, single-modal person re-identification has developed rapidly and achieved good recognition performance. Single-modal person re-identification generally captures visible light images, mainly solving the problem of view changes caused by the camera, and intra-modal differences caused by different pedestrian postures and backgrounds. Most of the images retrieved by single-modal person re-identification are images taken during the day, but in real life, many scenes often occur at night or in dark environments. For example, many crimes often occur at night. Single-modal person re-identification cannot meet the requirements of these occasions.
[0003] Cross-modal person re-identification uses visible light cameras and infrared cameras to capture visible light images and infrared images. The different wavelengths of the spectra of visible light cameras and infrared cameras will lead to modal differences between the two images. Cross-modal person re-identification must not only solve the intra-modal difference problem of single-modal pedestrian re-identification, but also solve the inter-modal difference problem of heterogeneous data of visible light images and infrared images. To this end, researchers have proposed a series of solutions. For example, Zhu et al. used a dual-channel local feature network to learn local features, and proposed a heterogeneous center loss to constrain the centers of the two heterogeneous modalities, so as to supervise the network to learn modality-invariant information representation (Neurocomputing, 2020, 386: 97-109). Wei et al. used an adaptive segmentation strategy to divide the blocks to obtain local features, and combined the global features with the local features to learn discriminant features and modality-invariant features (IEEE Transactions on Neural Networks and Learning Systems, 2021, pp. 1-12). However, these methods still have the following defects: they cannot fully extract global and local features, have poor feature discrimination, and cannot effectively eliminate modal differences. Summary of the invention
[0004] The purpose of the present invention is to overcome the defects of the prior art and provide a cross-modal pedestrian re-identification method based on global and local features, which reduces the modal difference by extracting deep features, matching local features with global features, and generating adversarial networks.
[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions.
[0006] A pedestrian re-identification method based on adversarial learning modality based on global and local features, comprising the following steps:
[0007] S1. Select the visible light image sample set X v And infrared image sample set X t As training data. Among them, the visible light image sample set is The sample set of infrared images is Each sample set contains M pedestrians, and the label set is Y = {y i |i=1,2,…,N}, N represents the number of visible light images or infrared images;
[0008] S2. On the basis of the convolutional neural network, an attention mechanism module is added to construct a feature extraction network with an attention mechanism; the visible light image samples and infrared image samples are input into the feature extraction network with an attention mechanism to obtain features; the feature extraction network with an attention mechanism includes a modality-specific submodule and a modality-shared submodule Modality specific submodules include visible light submodule and infrared submodule Modal shared submodule Include 3 convolutional blocks;
[0009] S3. The visible light feature f obtained from the feature extraction network with attention mechanism i v , infrared characteristics f i t Input the local feature learning module to operate and obtain the visible light local feature f i vp and infrared local features f i tp , visible light local dimension reduction feature f i vp′ and infrared local dimension reduction feature f i tp′ , visible light local label feature f i vp″ and infrared local label feature f i tp″ , visible light splicing feature f iv′ and infrared stitching feature f i t′ ;
[0010] S4. Visible light features f obtained from the feature extraction network with attention mechanism i v 、Infrared characteristicsf i t Input the prediction consistency module to operate and obtain the visible light global feature f i vg and infrared global feature f i tg , visible light global label feature f i vg′ and infrared global label feature f i tg′ ;
[0011] S5. Concatenate the visible light features f i v′ and infrared stitching feature f i t′ Input is processed by the adversarial learning module.
[0012] Specifically, the process of step S2 includes:
[0013] Step S2.1. Obtain the output feature F of the channel attention module i v′ :The visible light mid-level feature F i v As the input of the channel attention module, the visible light mid-level feature F i v Perform maximum pooling and average pooling operations on the spatial dimension to obtain features and Then the feature and Input into a two-layer fully connected neural network for dimension compression, and then the features after dimension compression are and Features Element by element summation, finally get the 1-dimensional channel attention feature M c ∈R C *1*1 :
[0014]
[0015] Where σ(·) represents the Sigmoid activation function, MLP represents a two-layer fully connected neural network, AvgPool(·) represents average pooling, MaxPool(·) represents maximum pooling, and W1 and W0 represent the weight parameters of the two-layer neural network.
[0016] Finally, the channel attention feature M c The mid-level feature F i v Multiply to get the output feature F of the channel attention module i v′ :
[0017]
[0018] In the formula Multiply the elements of the corresponding positions of the two matrices;
[0019] Step S2.2. Obtaining visible light feature f i v and infrared characteristics f i t :The output feature F of the channel attention module i v′ As the input feature of the spatial attention module, firstly, the input feature F i v′ Perform maximum pooling and average pooling operations on the channel dimension to obtain features and Then the obtained features and Features The concatenation is performed on the channel dimension, and then a 7×7 convolution operation is performed and a sigmoid activation layer is used to obtain a 2D spatial attention feature M. s ∈R 1*H*W :
[0020]
[0021] In the formula, F 7×7 Indicates a convolution operation with a convolution kernel of 7×7, [;] indicates concatenation of features;
[0022] Finally, the spatial attention feature M s With feature F i v′ Multiply to get the output feature F of the spatial attention module i v″ :
[0023]
[0024] Then the output feature F of the spatial attention module i v″ Input convolutional block Get the visible light feature f i v ; For infrared mid-layer features F it Conduct and visible light mid-level features F i v The same operation is used to obtain the infrared feature f i t .
[0025] Specifically, step S3 includes:
[0026] S3.1. For visible light characteristics f i v Divide evenly in the horizontal direction to obtain P visible light local features f of the same size i vp , p = 1, 2, ..., 6; then the local feature of visible light f i vp First, generalized average pooling is used; then a 1×1 convolution kernel is used for convolution to obtain the visible light local dimensionality reduction feature f i vp′ , for infrared features f i t Perform corresponding operations to obtain the infrared local dimensionality reduction feature f i tp′ ;
[0027] S3.2. Local dimension reduction feature f for each part of visible light i vp′ , infrared local dimension reduction feature f i tp′ The triplet loss based on heterogeneous centers is used to improve the discriminability of local features. The loss is as follows:
[0028]
[0029] In the formula, min||·|| + Function representation for difficult sample mining, [·] + =max(·,0), ρ is the interval parameter, ||·||2 is the Euclidean distance between two feature centers, It means summing the results from m=1 to m=M; are the local feature centers of the visible light image and infrared image of the mth pedestrian in the current training batch respectively; They are the pth part of the local features of the kth visible light image and the pth part of the local features of the kth infrared image of the mth pedestrian in the current training batch; there are M visible light local feature centers in total and M infrared local feature centers
[0030] S3.3. Dimensionality reduction of local visible light features f ivp′ Classify and obtain the visible light local label feature f i vp″ , the infrared local label feature f is obtained through the same operation i tp″ ; Based on the visible light local label feature f i vp″ and infrared local label feature f i tp″ The classification loss is:
[0031]
[0032] In the formula, p i (f i vp″ ) represents the visible light local label feature f i vp″ The generated probability distribution, p i (f i tp″ ) represents the infrared local tag feature f i tp″ The generated probability distribution; y i is the true label of the corresponding image, log is the logarithmic operation;
[0033] S3.4. Visible light stitching feature f i v′ The local dimension reduction feature f of each part of visible light i vp′ To splice ([f i v1′ ;f i v2′ ;f i v3′ ;f i v4′ ;f i v5′ ;f i v6′ ]) get, infrared splicing feature f i t′ The local dimension reduction feature f of each part of infrared i tp′ The visible light splicing feature f i v′ , infrared stitching feature f i t′ Adopt triplet loss based on heterogeneous center:
[0034]
[0035] In the formula, Based on visible light stitching feature fi v′ We get, which is the center of the P visible light stitching features of the mth pedestrian, Based on infrared stitching features f i t′ Get, which is the center of the P infrared stitching features of the mth pedestrian.
[0036] Specifically, step S4 includes:
[0037] S4.1. Visible light features f obtained from the feature extraction network with attention mechanism i v Converted to visible light global feature f through average pooling i vg , and then the visible light global feature f i vg The global feature identity prediction module composed of a three-layer fully connected neural network and a classifier is used to obtain the visible light global label feature f output by the third fully connected layer. i vg′ , and the identity category probability output by the classifier is the identity category probability of the i-th visible light global feature; for the infrared feature f i t Conducted with visible light characteristics f i v The same operation obtains the infrared global feature f i tg and infrared global label feature f i tg′ ;That as follows:
[0038]
[0039] In the formula, p g (f i vg′ ) represents the visible light global label feature f i vg′ The generated probability distribution, exp() refers to the exponential function with the natural constant e as the base, The identity label in the global feature identity prediction module is y i The classifier parameters of pedestrians, is the classifier parameter of the mth pedestrian in the global feature identity prediction module;
[0040] Similarly, we can get The identity category probabilities corresponding to the i-th visible light image and infrared image in the local feature mining module are
[0041] Use KL divergence loss L KL To reduce the distance between the two prediction distributions:
[0042]
[0043] In the formula, Measurement distribution and distribution The divergence of the mutual relationship between
[0044] S4.2. Global feature f for visible light i vg And the infrared global feature f i tg The triplet loss based on heterogeneous centers is used to improve the discriminability of global features, and the loss is:
[0045]
[0046] In the formula, is the center of the visible light global feature of the mth pedestrian, is the center of the infrared global feature of the mth pedestrian; and Based on the visible light global feature f i vg and infrared global feature f i tg Get; there are M visible light feature centers and M infrared characteristic centers Based on the visible light global label feature f i vg′ and infrared global label feature f i tg′ The classification loss is:
[0047]
[0048] In the formula, p g (f i vg′ ) represents the visible light global label feature f i vg′ The generated probability distribution, p g (f i tg′ ) represents the infrared global tag feature f i tg′ The generated probability distribution.
[0049] Specifically, step S5 includes:
[0050] S5.1. The modality discriminator D consists of two layers of fully connected neural networks, whose goal is to distinguish whether the features of the input image are visible light modality or infrared modality; the generator G consists of a local feature mining module and a prediction consistency module, whose goal is to minimize the modality difference between visible light features and infrared features while learning the discriminative features; the classification loss of the modality discriminator is:
[0051]
[0052] Where θ D are the parameters of the discriminator;
[0053] The generation loss of the generator G is:
[0054] L gen (θ id ,θ tri ,θ kl )=L id (θ id )+αL hc_tri (θ tri )+βL KD (θ kl ) (13)
[0055] In the formula, α and β are hyperparameters, θ id ,θ tri ,θ kl is the generator parameter;
[0056] S5.2. The generator and the modal discriminator perform a minimax game, and the parameters of the discriminator are fixed at the stage of the generative model, and vice versa; the training process between the generative model and the discriminative model is balanced, and the overall goal of optimization is:
[0057]
[0058]
[0059] In formula (14) Indicates that the variable θ is at the minimum value of the function id and θ tri The value of; in formula (15) Indicates that the variable θ is at the maximum value of the function. D The value of .
[0060] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0061] 1. The present invention uses an attention module to enhance the ability to extract deep features. The attention module includes a channel attention module and a spatial attention module. It is a lightweight general module that can be integrated into various convolutional neural networks for end-to-end training. The attention module enables the network to pay more attention to the target pedestrian itself and grasp the key information.
[0062] 2. The present invention uses a prediction consistency module to match the identity of global features with local features. The prediction consistency module uses KL divergence loss to make the identity prediction of global features to supervise the learning of local features, thereby maintaining the discriminable consistency of global features and local features.
[0063] 3. The present invention uses an adversarial learning module to reduce the modal difference between visible light features and infrared features, and minimizes the modal difference between visible light features and infrared features through the minimax game strategy in adversarial learning.
[0064] 4. By taking the above-mentioned effective measures, the present invention can greatly improve the cross-modal pedestrian re-identification retrieval rate. BRIEF DESCRIPTION OF THE DRAWINGS
[0065] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.
[0066] Figure 2 This is an overall diagram of the attention module of an embodiment of the present invention. DETAILED DESCRIPTION
[0067] The present invention is further described in detail below with reference to the accompanying drawings.
[0068] The present invention provides a pedestrian re-identification method based on global and local features in an adversarial learning mode, such as Figure 1 , Figure 2 As shown, a feature extraction network with an attention mechanism is used to extract features, which includes a modality-specific submodule, a modality-sharing submodule, and an attention mechanism module (including a channel attention module and a spatial attention module), and then the features are processed together by a local feature learning module, a prediction consistency module, and an adversarial learning module. The channel attention module includes a maximum pooling layer, an average pooling layer, a two-layer fully connected neural network, and a sigmoid activation function.
[0069] The method comprises the following steps:
[0070] S1. Select the visible light image sample set X v And infrared image sample set X t As training data. Among them, the visible light image sample set is The sample set of infrared images is Each sample set contains M pedestrians, and the label set is Y = {yi |i=1,2,…,N}, N represents the number of visible light images (infrared images).
[0071] S2. On the basis of the convolutional neural network, an attention mechanism module is added to construct a feature extraction network with an attention mechanism, and the visible light image samples and infrared image samples are input into the feature extraction network with an attention mechanism to obtain features. The feature extraction network with an attention mechanism includes a modality-specific submodule (including a visible light submodule) and infrared submodule ) and modal shared submodules ( Include 3 convolutional blocks), the visible light image samples selected from the training dataset and infrared image samples Input the feature extraction network with attention mechanism to operate, Output visible light mid-level feature F i v Input the attention module and obtain the visible light output feature F of the attention module i v″ Finally, the feature F i v″ Input convolutional block Get the visible light feature f i v , similarly, for the infrared mid-layer feature F i t Conduct and visible light mid-level features F i v The same operation can obtain the infrared feature f i t In the convolutional block and Channel attention modules and spatial attention modules are added between them to enhance the ability to extract deep features.
[0072] The specific process of step S2 includes:
[0073] Step S2.1. Obtain the output feature F of the channel attention module i v′ . Wherein, in step S2, Output visible light mid-level feature F i v ∈R C*H*W (C is the number of channels, H and W are the spatial dimensions), the visible light mid-level feature F i v As the input of the channel attention module, we first perform the visible light mid-level feature F iv Perform maximum pooling and average pooling operations on the spatial dimension to obtain features and Then the feature and Input into a two-layer fully connected neural network for dimension compression, and then the features after dimension compression are and Features Element by element summation, finally get the 1-dimensional channel attention feature M c ∈R C*1*1 . As shown below:
[0074]
[0075] In the formula, σ(·) represents the Sigmoid activation function, MLP represents a two-layer fully connected neural network, AvgPool(·) represents average pooling, MaxPool(·) represents maximum pooling, and W1 and W0 represent the weight parameters of the two-layer neural network. Finally, the channel attention feature M c The mid-level feature F i v Multiply to get the output feature F of the channel attention module i v′ , as shown below:
[0076]
[0077] In the formula Multiply the corresponding elements of two matrices.
[0078] Step S2.2. Obtaining visible light feature f i v and infrared characteristics f i t In step S2, the output feature F of the channel attention module is i v′ As the input feature of the spatial attention module, firstly, the input feature F i v′ Perform maximum pooling and average pooling operations on the channel dimension to obtain features and Then the obtained features and Features The concatenation is performed on the channel dimension, and then a 7×7 convolution operation is performed and a sigmoid activation layer is used to obtain a 2D spatial attention feature M. s ∈R 1*H*W , as shown below:
[0079]
[0080] In the formula, F 7×7 Indicates a convolution operation with a convolution kernel of 7×7, and [;] indicates concatenation of features. Finally, the spatial attention feature M s With feature F i v′ Multiply to get the output feature F of the spatial attention module i v″ , as shown below:
[0081]
[0082] Then the output feature F of the spatial attention module i v″ Input convolutional block Get the visible light feature f i v Similarly, for the infrared mid-layer feature F i t Conduct and visible light mid-level features F i v The same operation (first for the infrared mid-level feature F i t The maximum pooling and average pooling operations are performed on the channel dimension, and the features obtained by the maximum pooling and average pooling operations are spliced, and then the convolution operation with a convolution kernel of 7×7 is passed through the Sigmoid activation layer to obtain the infrared feature f i t .
[0083] S3. The visible light feature f obtained from the feature extraction network with attention mechanism i v , infrared characteristics f i t Input the local feature learning module to operate and obtain the visible light local feature f i vp and infrared local features f i tp , visible light local dimension reduction feature f i vp′ and infrared local dimension reduction feature f i tp′ , visible light local label feature f i vp″ and infrared local label feature f i tp″ , visible light stitching feature f i v′ and infrared stitching feature f i t′ .
[0084] The specific process of step S3 includes:
[0085] S3.1. For visible light characteristics f i v , infrared characteristics f i t The corresponding operation is performed. In step S3, the visible light feature f i v Divide evenly in the horizontal direction to obtain P visible light local features f of the same size i vp , p = 1, 2, ..., 6. Then the local feature of visible light f i vp First, generalized average pooling is used, and then a 1×1 convolution kernel is used for convolution to obtain the visible light local dimensionality reduction feature f i vp′ Similarly, for the infrared feature f i t Conducted with visible light characteristics f i v The same operation (uniform division in the horizontal direction, generalized average pooling, and convolution with a 1×1 convolution kernel) is used to obtain the infrared local dimension reduction feature f i tp′ .
[0086] S3.2. Dimensionality reduction of local visible light features f i vp′ , infrared local dimension reduction feature f i tp′ Calculate the triplet loss based on heterogeneous centers. In step S3, the local dimension reduction feature f of each part of visible light is i vp′ , infrared local dimension reduction feature f i tp′ The triplet loss based on heterogeneous centers is used to improve the discriminability of local features. The loss is as follows:
[0087]
[0088] In the formula, min||·|| + Function representation for difficult sample mining, [·] + =max(·,0), ρ is the interval parameter, ||·||2 is the Euclidean distance between two feature centers, It means summing the results from m=1 to m=M. They are respectively the local feature center of the visible light image and the local feature center of the infrared image of the mth pedestrian in the current training batch. They are the pth part of the local features of the kth visible light image and the pth part of the local features of the kth infrared image of the mth pedestrian in the current training batch. There are M visible light local feature centers and M infrared local feature centers
[0089] S3.3. Visible light local label feature f i vp″ , infrared local label feature f i tp″ Calculate the classification loss. In step S3, the local visible light dimension reduction feature f i vp′ Classify and obtain the visible light local label feature f i vp″ , for infrared dimension reduction feature f i tp′ Classify and obtain the infrared local label feature f i tp″ Based on the visible light local label feature f i vp″ and infrared local label feature f i tp″ The classification loss is:
[0090]
[0091] p i (f i vp″ ) represents the visible light local label feature f i vp″ The generated probability distribution, p i (f i tp″ ) represents the infrared local tag feature f i tp″ The generated probability distribution y i is the true label of the corresponding image, and log is the logarithmic operation.
[0092] S3.4. Visible light stitching feature f i v′ , infrared stitching feature f i t′ Calculate the triplet loss based on heterogeneous centers. In step S3, the visible light splicing feature f i v′ The local dimension reduction feature f of each part of visible light i vp′ To splice ([f i v1′ ;f i v2′ ;fi v3′ ;f i v4′ ;f i v5′ ;f i v6′ ]) get, infrared splicing feature f i t′ The local dimension reduction feature f of each part of infrared i tp′ The visible light splicing feature f i v′ , infrared stitching feature f i t′ The triplet loss based on heterogeneous center is adopted, and the loss is as follows:
[0093]
[0094] In the formula, Based on visible light stitching feature f i v′ We get, which is the center of the P visible light stitching features of the mth pedestrian, Based on infrared stitching features f i t′ Get, which is the center of the P infrared stitching features of the mth pedestrian.
[0095] S4. Visible light features f obtained from the feature extraction network with attention mechanism i v , infrared characteristics f i t Input the prediction consistency module to operate and obtain the visible light global feature f i vg and infrared global feature f i tg , visible light global label feature f i vg′ and infrared global label feature f i tg′ .
[0096] The specific process of step S4 includes:
[0097] S4.1. Prediction consistency operation. In step S4, the visible light feature f obtained from the feature extraction network with attention mechanism is i v Converted to visible light global feature f through average pooling i vg , and then the visible light global feature f i vgThe global feature identity prediction module composed of a three-layer fully connected neural network and a classifier is used to obtain the visible light global label feature f output by the third fully connected layer. i vg′ , and the identity category probability output by the classifier For infrared characteristics f i t Conducted with visible light characteristics f i v The same operation obtains the infrared global feature f i tg and infrared global label feature f i tg′ . as follows:
[0098]
[0099] In the formula, p g (f i vg′ ) represents the visible light global label feature f i vg′ The generated probability distribution, exp() refers to the exponential function with the natural constant e as the base, The identity label in the global feature identity prediction module is y i The classifier parameters of pedestrians, is the classifier parameter of the mth pedestrian in the global feature identity prediction module. Similarly, we can get The identity category probabilities corresponding to the i-th visible light image and infrared image in the local feature mining module are
[0100] In step S4, the KL divergence loss L is used. KL To reduce the distance between the two prediction distributions, L KL The loss is:
[0101]
[0102] In the formula, Measurement distribution and distribution The divergence of the mutual relationship between .
[0103] S4.2. Global feature f for visible light i vg , infrared global feature f i tg Calculate the triplet loss based on heterogeneous centers. In step S4, the visible light global feature f i vg And the infrared global feature f itg The triplet loss based on heterogeneous centers is used to improve the discriminability of global features. The loss is as follows:
[0104]
[0105] In the formula, is the center of the visible light global feature of the mth pedestrian, is the center of the infrared global feature of the mth pedestrian. and Based on the visible light global feature f i vg and infrared global feature f i tg There are M visible light feature centers. and M infrared characteristic centers Based on the visible light global label feature f i vg′ and infrared global label feature f i tg′ The classification loss is:
[0106]
[0107] In the formula, p g (f i vg′ ) represents the visible light global label feature f i vg′ The generated probability distribution, p g (f i tg′ ) represents the infrared global tag feature f i tg′ The generated probability distribution.
[0108] S5. Concatenate the visible light features f i v′ and infrared stitching feature f i t′ Input is processed by the adversarial learning module.
[0109] The specific process of step S5 includes:
[0110] S5.1. Introduce the generator and the discriminator. In step S5, the modal discriminator D includes a two-layer fully connected neural network, and the generator G includes a local feature mining module and a prediction consistency module. The goal of the discriminator is to distinguish whether the features of the input image are visible light modality or infrared modality. The goal of the generator is to minimize the modal difference between visible light features and infrared features while learning the identification features. The classification loss of the modal discriminator is:
[0111]
[0112] Where θ D is the parameter of the discriminator. The generation loss of the generator is:
[0113] L gen (θ id ,θ tri ,θ kl )=L id (θ id )+αL hc_tri (θ tri )+βL KD (θ kl ) (13)
[0114] In the formula, α, β are hyper parameters, θ id ,θ tri ,θ kl The generator parameters.
[0115] S5.2. Optimize the generator and the discriminator. In step S5, the generator and the modal discriminator perform a minimax game, and the parameters of the discriminator are fixed at the stage of the generative model, and vice versa. The overall goal of the optimization is to balance the training process between the generative model and the discriminative model:
[0116]
[0117]
[0118] In formula (14) Indicates that the variable θ is at the minimum value of the function id and θ tri The value of, in formula (15) Indicates that the variable θ is at the maximum value of the function. D The value of .
[0119] In summary, the present invention uses the attention module to enhance the ability to extract deep features, uses the prediction consistency module to match the global features with the local features, thereby maintaining the discriminable consistency of the global and local features, and reduces the modal differences between visible light and infrared features through the adversarial learning module, thereby greatly improving the retrieval rate of cross-modal pedestrian re-identification.
Claims
1. A pedestrian re-identification method based on adversarial learning mode based on global and local features, It is characterized in that The following steps are involved: S1. Select the visible light image sample set X v And infrared image sample set X t As training data; the visible light image sample set is The sample set of infrared images is Each sample set contains M pedestrians, and the label set is Y = {y i |i=1,2,…,N}, N represents the number of visible light images or infrared images; S2. On the basis of the convolutional neural network, an attention mechanism module is added to construct a feature extraction network with an attention mechanism; the visible light image samples and infrared image samples are input into the feature extraction network with an attention mechanism to obtain features; the feature extraction network with an attention mechanism includes a modality-specific submodule and a modality-shared submodule Modality specific submodules include visible light submodule and infrared submodule Modal shared submodule Include 3 convolutional blocks; S3. The visible light feature f obtained from the feature extraction network with attention mechanism i v , infrared characteristics f i t Input the local feature learning module to operate and obtain the visible light local feature f i vp and infrared local features f i tp , visible light local dimension reduction feature f i vp′ and infrared local dimension reduction feature f i tp′ , visible light local label feature f i vp″ and infrared local label feature f i tp″ , visible light splicing feature f i v′ and infrared stitching feature f i t′ ; S4. Visible light features f obtained from the feature extraction network with attention mechanism i v 、Infrared characteristicsf i t Input the prediction consistency module to operate and obtain the visible light global feature f i vg and infrared global feature f i tg , visible light global label feature f i vg′ and infrared global label feature f i tg′ ; S5. Concatenate the visible light features f i v′ and infrared stitching feature f i t′ Input adversarial learning module for processing; The step S4 specifically includes: S4.
1. Visible light features f obtained from the feature extraction network with attention mechanism i v Converted to visible light global feature f through average pooling i vg , and then the visible light global feature f i vg The global feature identity prediction module composed of a three-layer fully connected neural network and a classifier is used to obtain the visible light global label feature f output by the third fully connected layer. i vg′ , and the identity category probability output by the classifier is the identity category probability of the i-th visible light global feature; for the infrared feature f i t Conducted with visible light characteristics f i v The same operation obtains the infrared global feature f i tg and infrared global label feature f i tg′ ;in, as follows: In the formula, Represents the visible light global label feature f i vg′ The generated probability distribution, exp() refers to the exponential function with the natural constant e as the base, The identity label in the global feature identity prediction module is y i The classifier parameters of pedestrians, is the classifier parameter of the mth pedestrian in the global feature identity prediction module; Similarly, we can get The identity category probabilities corresponding to the i-th visible light image and infrared image in the local feature mining module are Use KL divergence loss L KL To reduce the distance between the two prediction distributions: In the formula, Measurement distribution and distribution The divergence of the mutual relationships between S4.
2. Global feature f for visible light i vg And the infrared global feature f i tg The triplet loss based on heterogeneous centers is used to improve the discriminability of global features, and the loss is: In the formula, is the center of the visible light global feature of the mth pedestrian, is the center of the infrared global feature of the mth pedestrian; and Based on the visible light global feature f i vg and infrared global feature f i tg Get; there are M visible light feature centers and M infrared characteristic centers Based on the visible light global label feature f i vg′ and infrared global label feature f i tg′ The classification loss is: In the formula, p g (f i vg′ ) represents the visible light global label feature fi vg′ The generated probability distribution, p g (fi tg′ ) represents the infrared global tag feature f i tg′ The generated probability distribution.
2. According to claim 1, a pedestrian re-identification method based on adversarial learning modality based on global and local features, It is characterized in that The specific process of step S2 includes: Step S2.
1. Obtain the output feature F of the channel attention module i v′ :The visible light mid-level feature F i v As the input of the channel attention module, the visible light mid-level feature F i v Perform maximum pooling and average pooling operations on the spatial dimension to obtain features and Then the feature and Input into a two-layer fully connected neural network for dimension compression, and then the features after dimension compression are and Features Element by element summation, finally get the 1-dimensional channel attention feature M c ∈R C*1*1 : Where σ(·) represents the Sigmoid activation function, MLP represents a two-layer fully connected neural network, AvgPool(·) represents average pooling, MaxPool(·) represents maximum pooling, and W 1 , W 0 Represents the weight parameters of the two-layer neural network; Finally, the channel attention feature M c The mid-level feature F i v Multiply to get the output feature F of the channel attention module i v′ : In the formula Multiply the elements of the two matrices at corresponding positions; Step S2.
2. Obtaining visible light feature f i v and infrared characteristics f i t :The output feature F of the channel attention module i v′ As the input feature of the spatial attention module, firstly, the input feature F i v′ Perform maximum pooling and average pooling operations on the channel dimension to obtain features and Then the obtained features and Features The concatenation is performed on the channel dimension, and then a 7×7 convolution operation is performed and a sigmoid activation layer is used to obtain a 2D spatial attention feature M. s ∈R 1*H*W : In the formula, F 7×7 Indicates a convolution operation with a convolution kernel of 7×7, [;] indicates concatenation of features; Finally, the spatial attention feature M s With feature F i v′ Multiply to get the output feature F of the spatial attention module i v″ : Then the output feature F of the spatial attention module i v″ Input convolutional block Get the visible light feature f i v ; For infrared mid-layer features F i t Conduct and visible light mid-level features F i v The same operation is used to obtain the infrared feature f i t .
3. According to claim 1, a pedestrian re-identification method based on global and local features in adversarial learning mode, It is characterized in that The step S3 specifically includes: S3.
1. For visible light characteristics f i v Divide evenly in the horizontal direction to obtain P visible light local features f of the same size i vp , p = 1, 2, ..., 6; then the local feature of visible light f i vp First, generalized average pooling is used; then a 1×1 convolution kernel is used for convolution to obtain the visible light local dimensionality reduction feature f i vp′ , for infrared features f i t Perform corresponding operations to obtain the infrared local dimension reduction feature f i tp′ ; S3.
2. Local dimension reduction feature f for each part of visible light i vp′ , infrared local dimension reduction feature f i tp′ The triplet loss based on heterogeneous centers is used to improve the discriminability of local features. The loss is as follows: In the formula, min||·|| + Function representation for difficult sample mining, [·] + =max(·,0), ρ is the interval parameter, ||·|| 2 is the Euclidean distance between two feature centers, It means summing the results from m=1 to m=M; are the local feature centers of the visible light image and infrared image of the mth pedestrian in the current training batch respectively; They are the pth part of the local features of the kth visible light image and the pth part of the local features of the kth infrared image of the mth pedestrian in the current training batch; there are M visible light local feature centers in total and M infrared local feature centers S3.
3. Dimensionality reduction of local visible light features f i vp′ Classify and obtain the visible light local label feature f i vp″ , the infrared local label feature f is obtained through the same operation i tp″ ; Based on the visible light local label feature f i vp″ and infrared local label feature f i tp″ The classification loss is: In the formula, p i (f i vp″ ) represents the visible light local label feature f i vp″ The generated probability distribution, p i (f i tp″ ) represents the infrared local tag feature f i tp″ The generated probability distribution; y i is the true label of the corresponding image, log is the logarithmic operation; S3.
4. Visible light stitching feature f i v′ The local dimension reduction feature f of each part of visible light i vp′ To splice ([f i v1′ ;f i v2′ ;f i v3′ ;f i v4′ ;f i v5′ ;f i v6′ ]) get, infrared splicing feature f i t′ The local dimension reduction feature f of each part of infrared i tp′ The visible light splicing feature f i v′ , infrared splicing feature f i t′ Adopt triplet loss based on heterogeneous center: In the formula, Based on visible light stitching feature f i v′ We get, which is the center of the P visible light stitching features of the mth pedestrian, Based on infrared stitching features f i t′ Get, which is the center of the P infrared stitching features of the mth pedestrian.
4. According to claim 1, a pedestrian re-identification method based on adversarial learning modality based on global and local features, It is characterized in that The step S5 specifically includes: S5.
1. The modality discriminator D consists of two layers of fully connected neural networks, whose goal is to distinguish whether the features of the input image are visible light modality or infrared modality; the generator G consists of a local feature mining module and a prediction consistency module, whose goal is to minimize the modality difference between visible light features and infrared features while learning the discriminative features; the classification loss of the modality discriminator is: Where θ D are the parameters of the discriminator; The generation loss of the generator G is: L gen (i id ,i tri ,i kl )=L id (i id )+αL hc_tri (i tri )+βL KD (i kl ) (13) In the formula, α and β are hyperparameters, θ id ,θ tri ,θ kl is the generator parameter; S5.
2. The generator and the modal discriminator perform a minimax game, and the parameters of the discriminator are fixed at the stage of the generative model, and vice versa; the training process between the generative model and the discriminative model is balanced, and the overall goal of optimization is: In formula (14) Indicates that the variable θ is at the minimum value of the function id and θ tri The value of; in formula (15) Indicates that the variable θ is at the maximum value of the function. D The value of .
