Contour map guided cross-modal pedestrian re-identification method and system

Abstract

The application discloses a cross-modal pedestrian re-identification method and system based on contour map guidance, and belongs to the technical field of pedestrian re-identification. The application adopts an additive and multiplicative collaborative operation mode to weight image features on corresponding contour features. The additive and multiplicative collaborative operation can combine the respective advantages of image features and contour features and save the original image information. A feature energy function is defined to weight the fused features. The feature energy function calculates the dispersion degree of each neuron of the fused features through a mean square deviation. After being weighted by the energy function, the features at the contour edge position will generate greater weights. The features at the contour edge position obtain more attention in subsequent calculations, and the obtained contour guided features and original image features are used for loss calculation together, so that the recognition accuracy of the cross-modal pedestrian re-identification is improved. The problem that the difference between multi-modal images reduces the pedestrian re-identification accuracy in the prior art is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian re-identification, in particular to a cross-modal pedestrian re-identification method and system based on contour map guidance. BACKGROUND

[0002] The statements in this section merely refer to the background of the present application and do not necessarily constitute the prior art.

[0003] Pedestrian re-identification (ReID) is a crucial task in video surveillance and smart cities. The goal of ReID is to retrieve the same identity pedestrian image from a gallery of pedestrian images captured across cameras given a query set of pedestrian images. Most of the current research focuses on matching RGB pedestrian images captured by visible light cameras. However, in low-light conditions, the imaging effect of visible light cameras is poor, and the captured images will produce a large number of noise points, or even cannot take a complete pedestrian image, thereby causing the accuracy of the model matching pedestrians to drop sharply. In recent years, many monitoring systems will automatically switch the camera to near-infrared mode in low-light environments to capture the complete appearance information of pedestrians at night. The accumulation of more and more cross-modal data prompts researchers to shift their attention to visible-infrared pedestrian re-identification (VI-ReID). VI-ReID can match the target pedestrian even in poor lighting conditions by matching pedestrian images from two different modalities. Compared with the traditional ReID task, VI-ReID also faces new challenges: the color difference between the two modalities of pedestrian images at the channel level will make it difficult for the model to mine the identity-distinguishing pedestrian features. Improving the recognition accuracy of cross-modal pedestrian re-identification models is of great significance to urban intelligent security.

[0004] How to alleviate the impact of modal difference and achieve good pedestrian recognition effect is a problem that needs to be solved at present. The current mainstream methods mainly fall into two categories:

[0005] The first category of methods is to design a reasonable network architecture to extract identity-distinguishing features, and directly map the features to a unified feature space for similarity comparison to match pedestrians with the same identity. Fu et al. introduced a NAS method to automatically search for the best separation scheme of the BN layer for feature extraction network, and Park et al. used the similarity between dense cross-modal features to align the features, which alleviated the difference between modalities and further enhanced the discriminability of the person representation. However, the difference between the two modal images is nonlinear, and simply extracting pedestrian image features by designing network structure for similarity comparison is difficult to minimize the impact of modal difference.

[0006] The second type of method is an image generation-based method, which aims to generate relevant images and jointly input them into a feature extraction network to alleviate the modal difference. Wang et al. proposed a cycle-GAN network to generate images of complementary modalities, i.e., generating a corresponding near-infrared modality pedestrian image from a visible light pedestrian image and generating a visible light pedestrian image from a near-infrared pedestrian image, so as to regard the cross-modality problem as a single-modality pedestrian matching problem; Zhang et al. proposed a method of encoding the identity pairing images of the two modalities into a unified high-dimensional space by a pair of encoders and calculating the feature distance, and then generating an intermediate modality image by a decoder, and jointly inputting the intermediate modality image into the feature extraction network to alleviate the influence of the modal difference. However, since the two modal pedestrian images of the cross-modality pedestrian re-identification task are not aligned at the pixel level, the quality of the images generated by the GAN network or the encoder method is often difficult to guarantee. Ma et al. proposed introducing a pedestrian contour map, which is a pixel-level aligned image, into the cross-modality pedestrian re-identification task, but they simply added the image features and contour features in different feature extraction stages to fuse them, resulting in that the pedestrian contour map is not fully expressed in the fused features. SUMMARY

[0007] In order to solve the problems in the prior art, the present application provides a cross-modality pedestrian re-identification method based on contour map guidance, a system, an electronic device and a computer readable storage medium. The contour detection model is used to extract the corresponding contour maps of the two modal pedestrian images, and the contour maps are jointly input into the feature extraction network with the original pedestrian images. The contour maps and the original images are effectively fused by using the multiplication cooperation method in different feature extraction stages, and the energy function of the fused features is defined to re-weight the fused features to strengthen the representation ability of the contour information. The influence of the modal difference on the pedestrian re-identification accuracy is alleviated by fusing the contour information.

[0008] In a first aspect, the present application provides a cross-modality pedestrian re-identification method based on contour map guidance.

[0009] The cross-modality pedestrian re-identification method based on contour map guidance comprises the following steps.

[0010] Obtaining a visible light modality pedestrian image and a near-infrared modality pedestrian image, and generating a pedestrian contour map according to the visible light modality pedestrian image and the near-infrared modality pedestrian image.

[0011] Extracting a visible light image feature and a near-infrared image feature according to the visible light modality pedestrian image and the near-infrared modality pedestrian image.

[0012] Obtaining contour-guided features by using a multiplication cooperation method according to the pedestrian contour map, the visible light image feature and the near-infrared image feature; and updating the contour-guided features by defining a feature energy function.

[0013] horizontally segmenting the visible light image features to obtain visible light image local features; horizontally segmenting the near-infrared image features to obtain near-infrared image local features; horizontally segmenting the updated contour guidance features to obtain contour guidance local features;

[0014] performing identity prediction on the pedestrians according to the contour guidance local features, the visible light image local features and the near-infrared image local features to obtain pedestrian re-identification results.

[0015] Further, the generating the pedestrian contour map according to the visible light modality pedestrian image and the near-infrared modality pedestrian image comprises:

[0016] inputting the visible light modality pedestrian image into a contour detector to obtain a visible light modality pedestrian contour map;

[0017] inputting the near-infrared modality pedestrian image into the contour detector to obtain a near-infrared modality pedestrian contour map;

[0018] splicing the visible light modality pedestrian contour map and the corresponding near-infrared modality pedestrian contour map to obtain the pedestrian contour map.

[0019] Further, the extracting visible light image features and near-infrared image features according to the visible light modality pedestrian image and the near-infrared modality pedestrian image comprises:

[0020] inputting the visible light modality pedestrian image and the near-infrared modality pedestrian image into a double-flow feature extraction network;

[0021] a first branch of the double-flow feature extraction network extracts the visible light modality pedestrian image to obtain visible light image features; a second branch of the double-flow feature extraction network extracts the near-infrared modality pedestrian image to obtain near-infrared image features;

[0022] wherein, a backbone network of the double-flow feature extraction network is ResNet-50, the first branch and the second branch do not share parameters of a first convolution block to extract modality private features, and the first branch and the second branch share parameters of a residual block to extract modality common features.

[0023] Further, the obtaining contour guidance features according to the pedestrian contour map, the visible light image features and the near-infrared image features comprises:

[0024] inputting the pedestrian contour map into a third branch for processing to obtain contour features, wherein a backbone network of the third branch is ResNet-50;

[0025] weighting the visible light image features and the near-infrared image features to corresponding contour features by a multiplication and addition collaborative method to obtain contour guidance features.

[0026] Further, the definition of the feature energy function updates the contour guiding feature, and the update of the contour guiding feature is specifically:

[0027] The feature energy function is defined, the weight of the contour guiding feature is determined according to the feature energy function and the dispersion degree of the neuron, and the contour guiding feature is updated according to the weight.

[0028] Further, the feature energy function is represented as:

[0029]

[0030] Wherein, F is a feature vector, α, β and γ are all hyperparameters, μ and φ 2 is the dispersion degree of the neuron in the contour guiding feature.

[0031] Further, the identity prediction of the pedestrian according to the contour guiding local feature, the visible light image local feature and the near-infrared image local feature, and the pedestrian re-identification result are obtained, and the identity prediction of the pedestrian according to the contour guiding local feature, the visible light image local feature and the near-infrared image local feature, and the pedestrian re-identification result are obtained.

[0032] The visible light image local feature and the near-infrared image local feature are reweighted by the non-local attention, and the reweighted visible light image local feature and the near-infrared image local feature are input into the global pooling layer to obtain the visible light image local feature vector and the near-infrared image local feature vector.

[0033] The contour guiding local feature is input into the global pooling layer to obtain the contour guiding local feature vector.

[0034] The contour guiding local feature vector, the visible light image local feature vector and the near-infrared image local feature vector are input into the full connection layer for identity prediction to obtain the pedestrian re-identification result.

[0035] In a second aspect, the present application provides a cross-modal pedestrian re-identification system based on contour map guidance;

[0036] The cross-modal pedestrian re-identification system based on contour map guidance comprises:

[0037] The contour map generation module is configured to obtain a visible light modal pedestrian image and a near-infrared modal pedestrian image, and generate a pedestrian contour map according to the visible light modal pedestrian image and the near-infrared modal pedestrian image; the feature fusion module is configured to extract a visible light image feature and a near-infrared image feature according to the visible light modal pedestrian image and the near-infrared modal pedestrian image; obtain a contour guiding feature by a multiplication and addition collaborative method according to the pedestrian contour map, the visible light image feature and the near-infrared image feature; and update the contour guiding feature by defining a feature energy function.

[0038] The pedestrian re-identification module is configured to: perform horizontal segmentation on the visible light image features to obtain visible light image local features; perform horizontal segmentation on the near-infrared image features to obtain near-infrared image local features; perform horizontal segmentation on the updated contour guided features to obtain contour guided local features; and perform identity prediction on the pedestrians according to the contour guided local features, the visible light image local features and the near-infrared image local features to obtain a pedestrian re-identification result.

[0039] In a third aspect, the present application provides an electronic device;

[0040] An electronic device comprises a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are run by the processor, the steps of the contour map guided cross-modal pedestrian re-identification method described above are completed.

[0041] In a fourth aspect, the present application provides a computer readable storage medium;

[0042] A computer readable storage medium is used to store computer instructions, when the computer instructions are executed by a processor, the steps of the contour map guided cross-modal pedestrian re-identification method described above are completed.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] 1. The technical scheme provided by the present application aims at the non-linear difference problem between visible light and near-infrared in cross-modal pedestrian re-identification, and proposes a multiplication and addition combined method to introduce pedestrian contour maps, effectively fusing pedestrian image and contour features, and relieving the modal difference.

[0045] 2. The technical scheme provided by the present application designs an energy function to further improve the expression ability of contour information in the fusion feature, weights the fusion feature according to the discrete degree between neurons to strengthen the expression ability of contour region neurons, and predicts the pedestrian identity together with the original image features, so as to improve the accuracy of cross-modal pedestrian re-identification. BRIEF DESCRIPTION OF DRAWINGS

[0046] The drawings accompanying the specification of the present application form part of the present application and serve to provide a further understanding of the present application, the illustrative embodiments of the present application and their description serve to explain the present application and do not constitute an improper limitation of the present application.

[0047] Figure 1 The flowchart provided for the embodiments of the present application;

[0048] Figure 2 The network architecture diagram provided for the embodiments of the present application;

[0049] Figure 3An effect comparison schematic diagram of a cross-modal pedestrian re-identification method based on contour map guidance provided by the embodiment of the present application and other methods based on the RegDB dataset;

[0050] Figure 4 An effect comparison schematic diagram of a cross-modal pedestrian re-identification method based on contour map guidance provided by the embodiment of the present application and other methods based on the SYSU-MM 01 dataset. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0052] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that the terms "comprise" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0053] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0054] Embodiment one

[0055] Since the identity-paired pedestrian images of two modalities of visible light and near-infrared are not one-to-one corresponding at the pixel level, and the generation process of the images is also uncontrollable, in the prior art, the method based on GAN is difficult to generate images meeting the actual needs, and cannot effectively alleviate the modal difference; therefore, the present application provides a cross-modal pedestrian re-identification method based on contour map guidance, which effectively fuses the contour map and the original image using the method of multiplication cooperation at different stages of feature extraction, and defines an energy function of the fused features to re-weight the fused features to strengthen the representation ability of the contour information, thereby alleviating the influence of the modal difference on the cross-modal pedestrian re-identification accuracy.

[0056] Next, combined with Figures 1-4 A cross-modal pedestrian re-identification method based on contour map guidance is disclosed in detail in the present embodiment, which comprises the following steps:

[0057] Step 1, obtain the pedestrian image in the visible light mode and the pedestrian image in the near-infrared mode, and generate a pedestrian contour map according to the pedestrian image in the visible light mode and the pedestrian image in the near-infrared mode.

[0058] Specifically, the pedestrian image in the visible light mode and the pedestrian image in the near-infrared mode are input into the contour detector to obtain the corresponding pedestrian contour map.

[0059] Step 1 is described in detail by taking reading data in the cross-modal pedestrian re-identification dataset SYSU-MM01 as an example, and the specific process is as follows:

[0060] S101, read the cross-modal pedestrian re-identification dataset SYSU-MM01, divide the dataset into uniform training batches, and define the pedestrian image in the visible light mode in each training batch as and the pedestrian image in the near-infrared mode as where NXP represents that each training batch includes N pedestrian identities, and for each pedestrian identity, P pedestrian images are randomly selected in the pedestrian image set in the visible light mode and the pedestrian image set in the near-infrared mode, respectively, and all training images are scaled to 384x192x3.

[0061] In this embodiment, the parameter N is set to 4 and the parameter P is set to 4, so the size parameter of each training batch is [32, 384, 192, 3].

[0062] S102, before the pedestrian images in each training batch are input into the feature extraction network, the contour detection model RCF pre-trained on the BSDS500 dataset with VGG-16 as the backbone network is used to perform contour detection on the two modal images to generate the corresponding pedestrian contour map in the visible light mode X vis2c =RCF(X vis ) and the pedestrian contour map in the near-infrared mode X ie2c =RCF(X ir ), the size of X vis2c and X ir2c is also 384x192x3, and each contour map is assigned the same pedestrian identity label as the corresponding original image. X vis2c and X ir2c are spliced in the batch dimension to obtain (X vis2c , X ir2c ) (the size parameter of which is the same as the size parameter of each batch, i.e. [32, 384, 192, 3]).

[0063] Step 2, according to the pedestrian image in the visible light mode and the pedestrian image in the near-infrared mode, extract the visible light image feature and the near-infrared image feature; according to the pedestrian contour map, extract the contour feature.

[0064] Specifically, the identity-paired visible light modality pedestrian image and near-infrared modality pedestrian image are input into a first branch and a second branch of a double-flow feature extraction network, and the pedestrian contour map generated in step 1 is input into an additional third branch; wherein the double-flow feature extraction network takes ResNet-50 as the backbone network, and the third branch also takes ResNet-50 as the backbone network.

[0065] The specific process is as follows:

[0066] S201, input the visible light modality pedestrian image X vis and the near-infrared modality pedestrian image X ir in each training batch into the double-flow feature extraction network with ResNet-50 as the backbone network, and extract the features of X vis The branch for extracting the features is called the first branch, and the visible light image features are obtained through the first branch; the branch for extracting the features is called the second branch, and the near-infrared image features are obtained through the second branch. ir

[0067] The two branches do not share the parameters of the first convolution block of ResNet-50 to extract modality private features, and the remaining residual blocks share the parameters to extract modality common features.

[0068] S202, input the pedestrian contour map (X vis2c , X ir2c ) into an additional feature extraction branch called the third branch to extract the corresponding contour features.

[0069] Step 3, according to the contour features, the visible light image features and the near-infrared image features, obtain the contour-guided features; specifically, in each residual stage of the ResNet-50 network feature extraction, the image features extracted by the first branch and the second branch are fused and weighted to the corresponding contour features extracted by the third branch to obtain the contour-guided features.

[0070] Exemplarily, the specific process is as follows:

[0071] S301, the backbone network of the first branch, the second branch and the third branch is ResNet-50, so the feature size extracted by each visible light modality pedestrian image, near-infrared modality pedestrian image and pedestrian contour map after passing through each convolution layer and residual stage is the same (ResNet-50 includes one convolution operation and four subsequent residual stages). In the four residual stages of ResNet-50, the visible light image features and the near-infrared image features of a single pedestrian extracted by the first branch and the second branch in each residual stage are defined as and The contour features extracted by the third branch in each residual stage are defined as ​

[0072] S302, the feature generated by each residual stage, the visible light image feature F extracted by the first branch vis and the near-infrared image feature F extracted by the second branch ir corresponding contour feature F extracted by the third branch is weighted by the multiplication and addition collaborative method ir2c and F vis2c The formula is expressed as follows:

[0073]

[0074]

[0075] wherein, represents the addition of corresponding elements, represents the multiplication of corresponding elements, and Conv represents a convolution operation with a 3x3 size convolution kernel.

[0076] The contour feature is first smoothed by a 3x3 convolution, and then multiplied by the image feature to obtain a preliminary enhanced feature; the preliminary enhanced feature is further added to the original image feature and the original contour feature to obtain the final fusion result.

[0077] In this embodiment, the image feature and the contour feature are fused by combining multiplication and addition and using the residual idea.

[0078] Step 4, defining a feature energy function to update the contour-guided feature.

[0079] Specifically, for the contour-guided feature obtained by fusing each residual stage in step 3, an energy function is defined and the feature is reweighted to highlight the representation ability of the contour information in the contour-guided feature. The feature energy function is represented as follows:

[0080]

[0081] wherein,

[0082]

[0083] F∈R H×W×C represents the contour-guided feature of each residual stage after fusion of a single pedestrian image, H, W, and C represent the length, width size, and channel number of the contour-guided feature; F ijk is the value of the feature vector F at the (i, j, k) position, and the corresponding hyperparameters a, b, and g are set to 5, 0.0001, and 0.5, respectively, and 2 The contour-guided feature F∈R H×W×CThe dispersion degree of all neurons, if a neuron has a large dispersion degree, it means that it is different from others. For the contour guided feature after fusion, the neurons located at the contour position will have a significant difference in value compared to the neurons not located at the contour position, thereby giving a larger energy function E.

[0084] Based on the energy function, a new weight is given to the contour guided feature using the formula: F = σ(E) O F.

[0085] Where O represents Hadamard product, and σ(·) represents relu function activation.

[0086] Therefore, the neurons at the contour position will be given a higher weight and thus get more attention. The fusion feature after energy function weighting strengthens the representation ability of the contour information therein and is sent to the next residual stage of the third branch feature extraction network.

[0087] Step 5, horizontally segmenting the visible light image features to obtain local features of the visible light image; horizontally segmenting the near-infrared image features to obtain local features of the near-infrared image; and horizontally segmenting the updated contour guided features to obtain contour guided local features.

[0088] Specifically, the features extracted by the feature extraction network are sent to the dual granularity segmentation module. The contour guided features extracted by the third branch are horizontally segmented into 2 local features by the coarse granularity branch, and the image features extracted by the first branch and the second branch are horizontally segmented into 4 local features by the fine granularity branch, representing the head, body, legs and feet of the pedestrian.

[0089] Here, the visible light image features and the near-infrared image features extracted by the first branch and the second branch are horizontally segmented into 4 local features, so they are called fine granularity segmentation; and the contour guided features extracted by the third branch are horizontally segmented into 2 local features, so they are called coarse granularity segmentation. The two segmentation methods together are called dual granularity segmentation module.

[0090] Specifically, the PCB (Part-based Convolutional Baseline) horizontal segmentation method is used for feature horizontal segmentation.

[0091] For example, the contour guided features finally extracted by the third branch are input into the coarse granularity branch, which horizontally divides the contour guided features into two equal parts in high dimension to obtain the local features of the upper body and the lower body of the pedestrian, which are defined as contour guided local features. The visible light image features extracted by the first branch are input into the fine-grained branch, and the image features are horizontally quartered in high dimension, and cut into four part features of the head, trunk, leg and foot of the pedestrian, and the four local features are defined as visible light image local features The near-infrared image features extracted by the second branch are input into the fine-grained branch, and the image features are horizontally quartered in high dimension, and cut into four part features of the head, trunk, leg and foot of the pedestrian, and the four local features are defined as near-infrared image local features

[0092] Step 5, according to the contour guidance local feature, the visible light image local feature and the near-infrared image local feature, the identity of the pedestrian is predicted to obtain the pedestrian re-identification result. Specifically, it includes:

[0093] Step 501, the non-local attention is used to reweight the visible light image local feature and the near-infrared image local feature to save high-order semantics.

[0094] Specifically, the obtained and are reweighted by non-local attention to obtain and The flow is as follows:

[0095] The obtained and are copied three times by three one-by-one convolution operations u(.), v(.) and z(.),

[0096] The self-similarity is used to calculate the similarity between the local features to obtain a local feature similarity matrix:

[0097]

[0098] Among them, represents the similarity between two local features, which is obtained by transposing the multiplication of the local features after the convolution operation u(.), v(.):

[0099]

[0100] Finally, the similarity matrix is multiplied by the local features obtained after the convolution operation z(.) to complete the non-local attention reweighting:

[0101]

[0102] Step 502, input the reweighted visible light image local feature and the near-infrared image local feature into the global pooling layer to obtain the visible light image local feature vector and the near-infrared image local feature vector; input the contour guidance local feature into the global pooling layer to obtain the contour guidance local feature vector.

[0103] Specifically, the contour-guided local feature The contour-guided local feature vector is obtained through a global pooling layer The contour-guided local feature vector obtained by reweighting is And The visible light image local feature vector is obtained through a global pooling layer And the near-infrared image local feature vector

[0104] Step 503, the visible light image local feature vector The near-infrared image local feature vector And the contour-guided local feature vector The final pedestrian re-identification result is obtained by inputting the full connection layer, and is optimized by the identity loss function and the triplet loss function.

[0105] The identity loss function is as follows:

[0106]

[0107] Where C represents the number of pedestrian identities in the training set, p i is the predicted value of the model for pedestrian identity i, q i is the probability of this class, and a is a constant.

[0108] The triplet loss function is as follows:

[0109]

[0110] Where J is a margin hyperparameter, D(F i ,F j ) represents the Euclidean distance between local features F i and F j , [z] + represents max(z,0), and Z refers to

[0111] Next, in order to verify the advancement of the method described in the embodiment, experiments are carried out based on the RegDB dataset and the SYSU-MM01 dataset.

[0112] Experimental platform: all experiments are implemented on a server with a TitanV GPU, and experiments are carried out on PyCharm software using Python programming language, and the deep learning framework used for programming is pytorch.

[0113] Experiment 1: dataset introduction

[0114] The SYSU-MM 01 dataset contains 491 different pedestrian images captured by four visible light cameras and two near-infrared cameras in indoor and outdoor environments. Each pedestrian is captured by at least one visible light camera and one near-infrared camera. The training set contains 19,659 visible light pedestrian images and 12,792 near-infrared pedestrian images of 395 pedestrian identities. The test set contains 96 pedestrian identities, of which 3,803 near-infrared pedestrian images form the query set. The gallery set is determined by the test mode, which includes a global search mode and an indoor search mode. In the global search mode, all visible light pedestrian images captured by the cameras are used as the gallery set. In the indoor search mode, only the images captured by the two indoor visible light cameras are used as the gallery set.

[0115] The RegDB dataset is a small dataset containing pedestrian images in both visible light and infrared modalities, which contains 412 pedestrian IDs (identity labels), each of which contains 10 visible light images and 10 infrared images. In this experiment, visible light images and infrared images are used as queries, respectively, and the other modality images are used as galleries. The two cases correspond to visible-thermal and thermal-visible, respectively. During the experiment, 206 IDs are randomly assigned for training, and the remaining 206 IDs are used for testing.

[0116] Experiment 2: Experimental details

[0117] To compare the effectiveness of the present embodiment, the SYSU-MM 01 and RegDB datasets are compared with the current mainstream method, and the effect is improved. The comparison indicators are rank-k and mAp, and the comparison results are shown in Figures 3-4

[0118] Embodiment two

[0119] The embodiment discloses a cross-modality pedestrian re-identification system based on contour map guidance, comprising:

[0120] The contour map generation module is configured to: acquire a visible light modality pedestrian image and a near-infrared modality pedestrian image, and generate a pedestrian contour map according to the visible light modality pedestrian image and the near-infrared modality pedestrian image;

[0121] The feature fusion module is configured to: extract visible light image features and near-infrared image features according to the visible light modality pedestrian image and the near-infrared modality pedestrian image; acquire contour guided features by a multiplication cooperation method according to the pedestrian contour map, the visible light image features and the near-infrared image features; and update the contour guided features by defining a feature energy function;

[0122] ​The pedestrian re-identification module is configured to: perform horizontal segmentation on the visible light image features to obtain visible light image local features; perform horizontal segmentation on the near-infrared image features to obtain near-infrared image local features; perform horizontal segmentation on the updated contour-guided features to obtain contour-guided local features; and perform identity prediction on the pedestrians according to the contour-guided local features, the visible light image local features, and the near-infrared image local features to obtain a pedestrian re-identification result.

[0123] It should be noted that the contour map generation module, the feature fusion module, and the pedestrian re-identification module correspond to the steps in Embodiment One, and have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the modules, as part of the system, can be executed in a computer system such as a set of computer executable instructions.

[0124] Embodiment Three

[0125] Embodiment Three of the present application provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor, when the computer instructions are executed by the processor, the steps of the above-mentioned contour map guided cross-modal pedestrian re-identification method are completed.

[0126] Embodiment Four

[0127] Embodiment Four of the present application provides a computer readable storage medium for storing computer instructions, which, when executed by a processor, complete the steps of the above-mentioned contour map guided cross-modal pedestrian re-identification method.

[0128] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.

[0129] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction devices, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 The device that implements the functions specified in one or more flows and / or blocks.Figure 1 the function specified in the one or more blocks.

[0130] These computer program instructions can also be loaded into computer or other programmable data processing devices, to cause a series of operational steps to be performed on the computer or other programmable devices, so that the instructions which are executed on the computer or other programmable devices provide steps for implementing the functions described in the flowcharts Figure 1 the flowcharts or multiple flowcharts and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0131] The above descriptions of the various embodiments are each focused on a particular aspect of the application. Those skilled in the art will readily recognize how best to implement the aspects of the application covered by the variations of these embodiments, in combinations with, or independent of, the other aspects of the application.

[0132] The application described above is merely preferred embodiments of the application, and the application is not limited thereto. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A cross-modal person re-identification method based on contour map guidance, characterized in that, include: Acquire visible light modal pedestrian images and near-infrared modal pedestrian images, and generate pedestrian contour maps based on the visible light modal pedestrian images and near-infrared modal pedestrian images; Based on visible light modal pedestrian images and near-infrared modal pedestrian images, extract visible light image features and near-infrared image features; Based on pedestrian contour maps, visible light image features, and near-infrared image features, contour guidance features are obtained through a multiplicative collaborative method; a feature energy function is defined to update the contour guidance features. The aforementioned multiplication-cooperative method specifically involves first smoothing the contour edges through a 3×3 convolution of the contour features, then performing element-wise multiplication with the image features to obtain preliminary enhanced features, and finally adding the preliminary enhanced features with the original image features and the original contour features to obtain the final fusion result. The specific formula is expressed as follows: ( )) ( in, This represents the visible light image features of a single pedestrian extracted by the first branch during the residual stage of the ResNet-50 network. This represents the near-infrared image features of a single pedestrian extracted by the second branch during the residual stage of the ResNet-50 network. This represents the addition of corresponding elements. `Conv` represents element-wise multiplication, and `Conv` represents a convolution operation with a 3×3 kernel. , , representing the single pedestrian contour features extracted by the third branch in the residual stage of the ResNet-50 network, respectively, and i represents the i-th residual stage of the ResNet-50 network; The characteristic energy function is expressed as: Where F is the feature vector, , and All of these are hyperparameters. and The degree of dispersion of neurons in the contour-guided features; Horizontal segmentation is performed on visible light image features to obtain local visible light image features; horizontal segmentation is performed on near-infrared image features to obtain local near-infrared image features; horizontal segmentation is performed on updated contour-guided features to obtain contour-guided local features; Pedestrian identity is predicted based on contour-guided local features, visible light image local features, and near-infrared image local features, and pedestrian re-identification results are obtained.

2. The cross-modal pedestrian re-identification method based on contour map guidance as described in claim 1, characterized in that, The step of generating a pedestrian contour map based on visible light modal pedestrian images and near-infrared modal pedestrian images includes: Input the visible light modal pedestrian image into the contour detector to obtain the visible light modal pedestrian contour map; Near-infrared modal pedestrian images are input into a contour detector to obtain near-infrared modal pedestrian contour maps; The pedestrian contour map is obtained by stitching together the visible light mode pedestrian contour map and the corresponding near-infrared mode pedestrian contour map.

3. The cross-modal pedestrian re-identification method based on contour map guidance as described in claim 1, characterized in that, The extraction of visible light image features and near-infrared image features based on visible light modal pedestrian images and near-infrared modal pedestrian images includes: Visible light modal pedestrian images and near-infrared modal pedestrian images are input into a two-stream feature extraction network; The first branch of the dual-stream feature extraction network extracts visible light modal pedestrian images to obtain visible light image features; the second branch of the dual-stream feature extraction network extracts near-infrared modal pedestrian images to obtain near-infrared image features. The backbone network of the dual-stream feature extraction network is ResNet-50. The first and second branches do not share the parameters of the first convolutional block to extract modality-private features, while the first and second branches share the parameters of the residual block to extract modality-shared features.

4. The cross-modal pedestrian re-identification method based on contour map guidance as described in claim 1, characterized in that, The process of obtaining contour guidance features based on pedestrian contour maps, visible light image features, and near-infrared image features includes: The pedestrian contour map is input into the third branch for processing to obtain contour features. The backbone network of the third branch is ResNet-50. Contour-guided features are obtained by weighting visible light image features and near-infrared image features to the corresponding contour features using a multiplicative collaborative method.

5. The cross-modal pedestrian re-identification method based on contour map guidance as described in claim 1, characterized in that, The defined feature energy function, used to update the contour-guided features, specifically involves: Define a feature energy function, determine the weights of the contour-guided features based on the feature energy function and the degree of neuron dispersion, and update the contour-guided features according to the weights.

6. The cross-modal pedestrian re-identification method based on contour map guidance as described in claim 1, characterized in that, The step of predicting pedestrian identity based on contour-guided local features, visible light image local features, and near-infrared image local features, and obtaining pedestrian re-identification results includes: The local features of visible light image and near-infrared image are reweighted by non-local attention. The reweighted local features of visible light image and near-infrared image are then input into a global pooling layer to obtain the local feature vectors of visible light image and near-infrared image. The contour-guided local features are input into the global pooling layer to obtain the contour-guided local feature vector; The contour-guided local feature vector, the visible light image local feature vector, and the near-infrared image local feature vector are input into the fully connected layer for identity prediction to obtain the pedestrian re-identification results.

7. A cross-modal pedestrian re-identification system guided by contour maps, characterized in that, include: The contour map generation module is configured to: acquire visible light modal pedestrian images and near-infrared modal pedestrian images, and generate pedestrian contour maps based on the visible light modal pedestrian images and near-infrared modal pedestrian images; The feature fusion module is configured to: extract visible light image features and near-infrared image features from visible light modal pedestrian images and near-infrared modal pedestrian images; obtain contour guidance features through a multiplicative collaborative method based on pedestrian contour maps, visible light image features, and near-infrared image features; and define a feature energy function to update the contour guidance features. The aforementioned multiplication-cooperative method specifically involves first smoothing the contour edges through a 3×3 convolution of the contour features, then performing element-wise multiplication with the image features to obtain preliminary enhanced features, and finally adding the preliminary enhanced features with the original image features and the original contour features to obtain the final fusion result. The specific formula is expressed as follows: ( )) ( in, This represents the visible light image features of a single pedestrian extracted by the first branch during the residual stage of the ResNet-50 network. This represents the near-infrared image features of a single pedestrian extracted by the second branch during the residual stage of the ResNet-50 network. This represents the addition of corresponding elements. `Conv` represents element-wise multiplication, and `Conv` represents a convolution operation with a 3×3 kernel. , , representing the single pedestrian contour features extracted by the third branch in the residual stage of the ResNet-50 network, respectively, and i represents the i-th residual stage of the ResNet-50 network; The characteristic energy function is expressed as: Where F is the feature vector, , and All of these are hyperparameters. and The degree of dispersion of neurons in the contour-guided features; The pedestrian re-identification module is configured to: perform horizontal segmentation on visible light image features to obtain local features of the visible light image; perform horizontal segmentation on near-infrared image features to obtain local features of the near-infrared image; perform horizontal segmentation on updated contour-guided features to obtain contour-guided local features; and predict the identity of the pedestrian based on the contour-guided local features, visible light image local features, and near-infrared image local features to obtain the pedestrian re-identification result.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the steps described in any one of claims 1-6.

Images