Cross-modal pedestrian re-identification method based on multi-scale multi-branch feature utilization

Abstract

The invention relates to a cross-modal pedestrian re-identification method based on multi-scale multi-branch feature utilization, and belongs to the field of cross-modal pedestrian re-identification. The method comprises the following steps: acquiring an original data set, and learning a relationship between each channel of a visible light modal image in the original data set and a single-channel infrared light modal image through a random channel; the processed image is input into a cross-modal feature extraction module, different attributes of different modals are processed through a double-flow network, and then multi-scale features are extracted through a shared network; inputting the multi-scale features into a multi-scale information mining module for feature fusion; and the fusion features are subjected to discriminant feature extraction by a multi-branch joint module, and pedestrian re-identification is carried out through cooperation of the discriminant features extracted by each branch. According to the method, multi-level spatial feature information is combined for feature extraction, the correlation between an infrared mode and a visible light mode in visual representation is increased by using global features and local features, and the retrieval accuracy of cross-modal pedestrian re-identification is improved.

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal pedestrian re-identification, and relates to a cross-modal pedestrian re-identification method utilizing multi-scale and multi-branch features. Background Art

[0002] As an indispensable and top priority in urban construction, intelligent surveillance video technology can greatly improve the safety, comfort and convenience of urban residents' lives through the analysis of surveillance videos. Pedestrian search refers to the task of searching for specific pedestrians under multiple cameras. This technology can determine whether a specified pedestrian appears on multiple cameras. With the increase in public safety needs and the increase in the number of surveillance cameras, traditional manual monitoring can no longer meet the needs of the big data era. Machine learning is used to improve intelligent surveillance technology, among which pedestrian search has important research significance and practical application value. Methods based on deep learning can effectively analyze video surveillance data. Pedestrian re-identification is a hot topic in the field of deep learning. Since it can automatically match the retrieved pedestrians in the video surveillance information and accurately locate the location where the pedestrian appears, it has considerable application prospects in the fields of intelligent monitoring and security in urban construction. With the improvement of camera functions, at night, the camera can automatically switch to infrared images of pedestrians. At this time, the infrared images show more obvious pedestrian features than the visible light images at night. However, infrared images also have shortcomings such as lack of color information and blurry faces and pedestrians. Therefore, biometric recognition technology in cross-modal scenarios has become a new challenge and opportunity. Cross-modal biometric recognition technology needs to consider the differences in image quality, distribution, expression, etc. between different modalities, and design effective algorithms to achieve cross-modal matching and retrieval.

[0003] Cross-modal recognition aims to study the mutual recognition between data of different modalities (infrared, sketches, low-resolution images, color images). The difficult problem that this type of model needs to solve is how to combine the feature information extracted from data of different modalities, and how the model can rely on other modalities to complete the task when one modality fails. In this topic, cross-modal recognition specifically refers to the mutual recognition of images between infrared (near-infrared) modality and visible light (color) modality. Cross-modal recognition is generally called cross-modal pedestrian re-identification in the field of pedestrian re-identification.

[0004] With the continuous development of cross-modal recognition tasks, research in this direction is also facing new challenges. It mainly focuses on how to deal with the differences between images of different modalities. Since the camera in infrared mode only uses infrared light for imaging, the infrared image only contains single-channel information, that is, the intensity information of infrared light. This means that the important visual feature of color is missing in the infrared image, and it can only reflect information such as the structure and texture of the object. Therefore, in cross-modal scenarios, there will be great differences between color images and infrared images, which are inconsistent in content, distribution, expression, etc.

[0005] Cross-modal person re-identification has attracted extensive attention because it is more difficult than traditional single-modal pedestrian re-identification tasks and is closer to real scenes. The main methods include narrowing the differences between cross-modal features by designing network models and processing pedestrian features; improving the recognition accuracy of the model by generating images; and improving the recognition accuracy by designing loss functions. Ye et al. transformed the features of the two modalities extracted into a common space by jointly sharing the modality matrix and optimizing the modality specificity, so that the modality sharing metric can be learned to train the model. In addition, a two-stream network structure that can learn cross-modal sharable features was designed. Wang et al. unified infrared or visible light images into images of the same style by using GAN, then extracted features from the images of the same style, and reduced the appearance differences by feature embedding, thereby narrowing the differences between modalities. Zhu et al. proposed a heterogeneous center loss, which trained the network to learn the invariant information of cross-modal pedestrian image features by narrowing the intra-class center distance of the visible light modality and the infrared light modality in the common feature space, and then combined the identity verification loss to narrow the inter-class and intra-class differences.

[0006] Although some studies have tried to reduce these differences through methods such as generative adversarial networks (GANs) and shared matrix optimization, the quality of generated images, the accuracy of feature conversion, and the generalization ability of the algorithm still need to be improved. In particular, when dealing with highly heterogeneous image pairs, how to ensure that the model can learn invariant features across modalities is a difficulty in current technology. Summary of the invention

[0007] In view of this, an object of the present invention is to provide a cross-modal pedestrian re-identification method utilizing multi-scale and multi-branch features.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A cross-modal person re-identification method using multi-scale and multi-branch features, comprising the following steps:

[0010] S1. Obtaining an original data set, which includes a visible light modality image and an infrared light modality image, and learning a relationship between each channel in the visible light modality image and the single-channel infrared light modality image through a random channel;

[0011] S2, inputting the visible light modality image and the infrared light modality image after random channel processing into the cross-modal feature extraction module, wherein the different attributes of different modalities are first processed by a two-stream network, and then multi-scale features are extracted by a shared network;

[0012] S3, inputting the multi-scale features into a multi-scale information mining module for feature fusion, wherein the multi-scale information mining module includes multiple self-attention layers, and the self-attention layers sequentially fuse features of different scales;

[0013] S4, the fused features are then used by a multi-branch joint module to extract discriminative features, which includes a global branch and a local branch, and global features and local features are obtained by extracting the global branch and the local branch;

[0014] S5. Apply global features and local features to the reverse feedback process during training, and use triplet loss and identity verification loss to iteratively optimize the model; during real-time recognition, calculate the similarity through the loss function and sort it to obtain the re-recognition result.

[0015] Further, in step S1, the original data set obtained is represented as D = {D v ,D r},in, Represents a collection of visible light modality images, each image It consists of three channels; represents a collection of infrared light modality images, is a single-channel infrared light modality image; the relationship between each R, G, and B channel of the visible light modality image and the single-channel infrared image is directly learned through random channel data exchange, and its learning objectives are:

[0016]

[0017] The dataset after learning is represented as

[0018] Further, in step S2, the completed dataset is learned The input is sent to the cross-modal feature extraction module for feature extraction, which at least includes:

[0019] S21, the learned visible light modality image dataset and infrared light modality image dataset They are respectively input into the two-stream network of the cross-modal feature extraction module, and the corresponding low-level features are extracted by the corresponding residual convolution module. and

[0020] S22. Low-level features corresponding to visible light modality and infrared light modality and Then it is input into the shared network, and multiple residual convolution modules in the shared network extract features of different levels {F m1 ,F m2 ,F m3}.

[0021] Furthermore, in the two-stream network of step S21, the branch for processing the visible light modality image includes a residual convolution module and Residual convolution modules with different network parameters For the learned visible light modality image Processing to obtain specific low-level features of visible light modality images

[0022]

[0023] The branch for processing infrared modality images includes a residual convolution module and Residual convolution modules with different network parameters and For the infrared light modality image after learning Processing to obtain specific low-level features of infrared light modality images

[0024]

[0025] in, They are the convolution operations corresponding to the residual convolution modules respectively.

[0026] Further, in the shared network of step S22, at least three residual convolution modules are set To obtain different levels of feature representation, the extracted features are:

[0027]

[0028]

[0029] The final output is a three-level image feature, represented as {F m1 ,F m2 ,F m3}, each layer of feature images is input into the corresponding self-attention layer in turn.

[0030] Further, in step S3, the extracted multi-level features {F m1 ,F m2 ,F m3} are respectively input into the self-attention layer IF of the multi-scale information mining module, and finally the deep features F containing shallow features and middle features are obtained n , where in the first self-attention layer, F m3 Considered Each self-attention layer will and As its own input, it is mapped to the same space through convolutional layers and regularization, and then reduced in dimension through 1*1 convolution, and then shallow features are extracted through a multi-head attention mechanism. For the attention mechanism A, its query Q, key K and value V are expressed as:

[0031]

[0032] where F c (·) represents the dimension reduction operation, represents the equilibrium parameter, F conv (·) indicates a dimension-raising operation.

[0033] Further, in step S4, the extracted deep features F n Input to the multi-branch joint module for discriminative feature extraction. The multi-branch joint module includes a global branch and a local branch. The global branch extracts the discriminative features from the deep feature F n Extract the global feature F g , the local branch is derived from the deep feature F n Extract local features F l ,in,

[0034] In the global branch, the deep features F n The three-dimensional features are reduced to one-dimensional features through generalized mean pooling. The processing process is expressed as:

[0035]

[0036] In the formula, U represents the one-dimensional feature vector after global feature transformation, |·| represents the conversion matrix value, and p is the pooling hyperparameter;

[0037] Then, channel dimension reduction is performed through a 1*1 convolution layer and a batch normalization layer to obtain the global feature F g .

[0038] Furthermore, in the local branch, the deep feature F n First, divide the block into three equal parts horizontally to obtain three local feature vectors F1 n 、F2 n 、F3 n, and then the three local eigenvectors F1 n 、F2 n 、F3 n The three-dimensional features are reduced to one-dimensional features U1, U2, and U3 through generalized mean pooling:

[0039]

[0040] Where i = 1, 2, 3, and then the three one-dimensional features are processed by U1, U2, and U3 channel dimensionality reduction through 1*1 convolution layer and batch normalization layer to obtain local features F l .

[0041] Further, in step S5, the global feature F is extracted g and local features F l Finally, the model parameters are continuously optimized by feeding the triplet loss and identity verification loss back to the model.

[0042] During the recognition process, the Euclidean distance between the input image and each image in the image library after model training is calculated through the loss function, and the images in the library are sorted in ascending order of distance to obtain the first n images with the smallest distance in the library, that is, the images with the highest similarity, recorded as Rank_k. Rank_k represents the hit rate of the first k results that are similar to the query image. The calculation formula of Rank_k is as follows:

[0043]

[0044] In the formula, It indicates the number of correct results among the k query results, and N indicates the total number of pedestrian images in the database.

[0045] Furthermore, the loss function in the training process is designed as:

[0046]

[0047] Among them, L ID represents the authentication loss, which is expressed as:

[0048]

[0049] Where N represents the number of pedestrian images contained in a batch, W j represents the weight vector of the jth class in the fully connected layer, x i Indicates that it belongs to y i The feature vector extracted from the i-th sample of the category, b and T represent the bias and transposition, and M is the number of pedestrian identity categories;

[0050] and are the triplet loss with visible light modality image as anchor point and the triplet loss with infrared light modality image as anchor point, respectively. tr It is expressed as:

[0051]

[0052] Where a, p, n represents a benchmark triplet sample pair, representing the anchor sample image, the positive sample image, and the negative sample image, respectively; D(·) represents the Euclidean distance, F(·) represents the feature extractor, [·] + represents the clipping function max(·,0), and margin is used to regulate the distance between the feature vectors of the same pedestrian in the common space to be smaller than the distance between different pedestrian identities.

[0053] The beneficial effects of the present invention are:

[0054] The present invention obtains features at different levels and scales, extracts the feature information of the infrared modality and the visible light modality at the shallow level, and extracts shared information at the middle and deep levels. When mining related information, the multi-head attention mechanism is used as the basic block to further mine the valuable parts of the shallow information. The combination of global features and local features is used as the final pedestrian features to enhance the expression ability of the fused features, and the identity verification loss and sample triplet loss are used for training, which can improve the accuracy of cross-modal pedestrian re-identification retrieval.

[0055] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 A schematic diagram of the overall framework structure of the cross-modal person re-identification method using multi-scale and multi-branch features of the present invention;

[0058] Figure 2 Schematic diagram of the processing process of the self-attention layer in the multi-scale information mining module of the present invention. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0060] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0061] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0062] See also Figure 1-2 , which is a cross-modal pedestrian re-identification method utilizing multi-scale and multi-branch features.

[0063] Example

[0064] In this embodiment, Figure 1 The overall framework structure diagram of a cross-modal pedestrian re-identification method using multi-scale multi-branch features is shown, which includes the following steps:

[0065] S1. Obtaining an original data set, which includes a visible light modality image and an infrared light modality image, and learning a relationship between each channel in the visible light modality image and the single-channel infrared light modality image through a random channel;

[0066] S2, inputting the visible light modality image and the infrared light modality image after random channel processing into the cross-modal feature extraction module, wherein the different attributes of different modalities are first processed by a two-stream network, and then multi-scale features are extracted by a shared network;

[0067] S3, inputting the multi-scale features into a multi-scale information mining module for feature fusion, wherein the multi-scale information mining module includes multiple self-attention layers, and the self-attention layers sequentially fuse features of different scales;

[0068] S4, the fused features are then used by a multi-branch joint module to extract discriminative features, which includes a global branch and a local branch, and global features and local features are obtained by extracting the global branch and the local branch;

[0069] S5. Apply global features and local features to the reverse feedback process during training, and use triplet loss and identity verification loss to iteratively optimize the model; during real-time recognition, calculate the similarity through the loss function and sort it to obtain the re-recognition result.

[0070] Cross-modal person re-identification refers to the mutual retrieval of data in infrared and visible light modalities, usually using one modality as a query to retrieve relevant data in another modality. The core tasks are: spatial data feature extraction and content correlation measurement between different modal data.

[0071] The cross-modal person re-identification task requires the model to learn more discriminative pedestrian representations. Therefore, the design of this invention combines multi-level spatial feature information in the convolution process to perform all-round feature extraction; at the same time, by using global features and local features at the same time, the correlation between the infrared light modality and the visible light modality (inter-modality) in the visual representation is increased. To a certain extent, the retrieval accuracy of cross-modal person re-identification is improved.

[0072] In step S1 of this embodiment, the ResNet50 network is used to extract visual features. The original data set contains images of visible light and infrared light. The original data set is represented by D = {D ν ,D r}. Represents the collection of visible light modality images in the dataset, each image It consists of three channels. represents the set of infrared light modality images in the dataset, is a single-channel infrared image. In order to narrow the gap between the input image levels while retaining the rich color information, the relationship between each R, G, and B of the visible light image and the single-channel infrared image is directly learned through random channel data exchange, thereby optimizing the relationship between the features of the three-channel visible light image and the single-channel infrared image. The learning target formula is:

[0073]

[0074] The dataset after learning is represented as

[0075] In step S2 of the present embodiment, the purpose of the cross-modal feature extraction module is to obtain feature embedding representations of visible light modality and infrared modality images. Due to different shooting cameras, images of different modalities have significant differences in visual presentation. Visible light modality images contain rich spatial structure information, while infrared modality images contain more spatial texture information. The ResNet50 network is also used to extract image features. Since the input is pedestrian images of two different modalities, the two-stream network must not only capture the specific features of pedestrian images of different modalities, but also consider extracting shared features of pedestrian images of different modalities. In order to handle the different attributes of different modalities, the network parameters of the first two residual convolution blocks in each stream are different to capture modality-specific low-level features. Subsequently, in subsequent residual convolution blocks, the shared network parameters are used to learn the intermediate layer feature representations between modalities. The feature representation of the last three layers of the modality extracted by the ResNet50 shared network is represented as {F m1 ,F m2 ,F m3}, which reduces the number of model parameters and thus shortens the time required for training.

[0076] Specifically, in the two-stream network of the cross-modal feature extraction module, the branch for processing visible light modality images includes a residual convolution module and Residual convolution modules with different network parameters For the learned visible light modality image Processing to obtain specific low-level features of visible light modality images

[0077]

[0078] The branch for processing infrared modality images includes a residual convolution module and Residual convolution modules with different network parameters and For the infrared light modality image after learning Processing to obtain specific low-level features of infrared light modality images

[0079]

[0080] in, They are the convolution operations corresponding to the residual convolution modules respectively.

[0081] In the shared network, at least three residual convolution modules are set To obtain different levels of feature representation, the extracted features are:

[0082]

[0083] The final output is three-layer image features, represented as {F m1 ,F m2 ,F m3}, each layer of feature images is also input into the corresponding self-attention layer in turn.

[0084] In step S3 of this embodiment, as the network depth increases, the extracted features become more and more refined. The goal is to integrate these refined features with features containing shallow information to enrich the feature information in the deep layer of the network. The extracted feature F m3 Considered as right and The feature information of the multi-head attention mechanism is used as the basic block IF to extract comprehensive features. After self-attention extraction, the output value is set to Then and As the input of the next self-attention layer, multi-scale feature extraction is performed. The feature set after the multi-scale information mining module (MIF) composed of triple self-attention information extraction is F n .

[0085] The multi-scale information mining module is composed of three layers of IF. The input of each IF is and F n is the input in the backbone, representing the deep features. m It is the output retained by the shallow network, representing the lost information. Through the convolution layer and regularization, they are mapped to the same space, and then reduced in dimension through 1*1 convolution. The multi-head attention mechanism can extract the valuable parts of the shallow features and explore the correlation between the features. Figure 2 As shown, the attention mechanism is recorded as A, where the query Q, key K and value V are:

[0086]

[0087] Among them, F c It is to reduce the dimension. represents the equilibrium parameter, here we set In order to facilitate the output of IF as the input of the next layer, a convolutional layer F is finally applied conv By increasing the dimension of the features after the attention mechanism, the function of a single IF layer is realized.

[0088] In step S4 of this embodiment, a multi-branch joint method is used to extract features from different granularities, thereby obtaining more powerful discriminant features, mainly including global branches and local branches. g and local features F l There is a complementary relationship between the two, and both play a key role in pedestrian retrieval. The global branch identifies pedestrians from an overall perspective, and the local branch extracts features from local features of pedestrians for identification. For the global branch, the extracted features are reduced to one-dimensional features through generalized mean pooling (GEM), and then channel dimension reduction is performed through a 1*1 convolution layer and a batch normalization layer (BN). The GEM formula is:

[0089]

[0090] Where U represents the transformed one-dimensional feature vector, |·| represents the conversion matrix value, and p is the pooling hyperparameter.

[0091] The 1*1 convolution operation formula is:

[0092] Z=W·U+b

[0093] Z is the output feature map after 1*1 convolution, W is the weight of the convolution kernel, U is the input feature map, and b is the bias.

[0094] The batch normalization formula is:

[0095]

[0096] is the normalized output, μ is the mean of the current batch, σ 2 is the variance of the current batch, ∈ is a small constant to prevent division by zero errors, γ is the scaling parameter, and β is the translation parameter.

[0097] After the fully connected layer, the output is:

[0098]

[0099] F g It is the output of the global feature after a series of dimensionality reduction and full connection, W is the weight matrix, and b is the bias term.

[0100] The local feature branch extracts the multi-scale pedestrian features F n Divide the block into three equal parts in the horizontal direction to obtain three local feature vectors F1 n 、F2 n 、F3 n , and then perform GEM operation, followed by 1*1 convolution layer and BN layer.

[0101] The GeM formula is:

[0102]

[0103] Among them U i is the transformed one-dimensional feature vector, i = 1, 2, 3. F i n is the local eigenvector.

[0104] The 1*1 convolution formula is:

[0105] Z i =W·U i +b

[0106] Z i is the output after convolution, W is the convolution kernel weight matrix, and b is the bias vector.

[0107] The batch normalization formula is:

[0108]

[0109] It is the feature map after convolution.

[0110] After that, the three local feature maps are connected to a fully connected layer. Before the feature map is input into the fully connected layer, it is usually necessary to flatten it from a one-dimensional dimension of 1×n×1×1 to a two-dimensional matrix dimension of 3×n×1×1. The flattened feature is named

[0111] The final output of the fully connected layer is:

[0112]

[0113] F l is the output of the fully connected layer, W is the weight matrix, and b is the bias term.

[0114] In step S5 of this embodiment, ResNet50 is used as the basic model, and a dual-stream network is used to extract specific image features in stage0 and stage1 respectively, and common features are extracted in the following three layers of residual shared parameters. In order to obtain multi-scale fusion features, the deep features and shallow features are integrated through the MIF module to enrich the information of the deep features. Finally, through the multi-branch structure of global features and local features, and the joint loss function constrained feature learning network training, a stronger identification of pedestrians is made. Combining the obtained global features and local features, in a reverse feedback process during the training process, the triple loss and the identity verification loss will feed back the loss to the model during the training process, thereby updating the parameters of the model. By continuously iterating and optimizing the model, it can better understand the similarities and differences in the data. The loss is calculated by the loss function and returned to the model during training, and the parameters are adjusted to continuously optimize the model. Similarly, during testing, the loss function can calculate the similarity and obtain the result by sorting the similarity.

[0115] In this embodiment, the obtained branch features are subjected to triplet loss learning and identity (ID) loss learning. The formula of triplet loss is expressed as:

[0116]

[0117] Where a, p, n represents a benchmark triplet sample pair, representing the anchor sample image, the positive sample image, and the negative sample image, respectively. D(·) represents the Euclidean distance, F(·) represents the feature extractor, [·] + represents the cropping function max(·,0), and margin can regulate the distance between the feature vectors of the same pedestrian in the public space to be smaller than the distance between different pedestrian identities. In addition, the triplet loss function of cross-modal pedestrian re-identification needs to consider two cases: and the visible light image as anchor Therefore, the triplet loss L tr Should be:

[0118]

[0119] After the multi-branch joint module, each branch feature passes through the fully connected layer and is normalized using the identity (ID) verification loss. The formula for identity verification loss is expressed as:

[0120]

[0121] In the formula, N represents the number of pedestrian images contained in a batch, and W j represents the weight vector of the jth class in the fully connected layer, x i Indicates that it belongs to y iThe feature vector extracted from the i-th sample of the category, b and T represent the bias and transposition, and M is the number of pedestrian identity categories.

[0122] Therefore, the overall loss function is expressed as:

[0123]

[0124] The network model undergoes multiple rounds of training. The training process should be to continuously optimize the model parameters by feeding the triplet loss and identity verification loss back to the model after extracting the features. Because the distance between samples of different categories is much greater than the distance between samples of the same category, the identity verification loss is also applicable to the task of judging the similarity between samples.

[0125] Therefore, the recognition process is actually to calculate the Euclidean distance between the input image and each image in the image library after model training through the loss function, and sort the images in the library in ascending order of distance, so as to obtain the first n images with the smallest distance in the library, that is, the images with the highest similarity, recorded as Rank_k. Rank_k represents the hit rate of the first k results similar to the query image. The calculation formula of Rank_k is as follows:

[0126]

[0127] The number of correct results among k query results is recorded as The total number of pedestrian images in the database is denoted as N.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-modal person re-identification method using multi-scale and multi-branch features, characterized by: It includes the following steps: S1. Obtaining an original data set, which includes a visible light modality image and an infrared light modality image, and learning a relationship between each channel in the visible light modality image and the single-channel infrared light modality image through a random channel; S2, inputting the visible light modality image and the infrared light modality image after random channel processing into the cross-modal feature extraction module, wherein the different attributes of different modalities are first processed by a two-stream network, and then multi-scale features are extracted by a shared network; S3, inputting the multi-scale features into a multi-scale information mining module for feature fusion, wherein the multi-scale information mining module includes multiple self-attention layers, and the self-attention layers sequentially fuse features of different scales; S4, the fused features are then used by a multi-branch joint module to extract discriminative features, which includes a global branch and a local branch, and global features and local features are obtained by extracting the global branch and the local branch; S5. Apply global features and local features to the reverse feedback process during training, and use triplet loss and identity verification loss to iteratively optimize the model; during real-time recognition, calculate the similarity through the loss function and sort it to obtain the re-recognition result.

2. The cross-modal person re-identification method using multi-scale and multi-branch features according to claim 1, characterized in that: In step S1, the original data set obtained is represented as D = {D ν ,D r },in, Represents a collection of visible light modality images, each image It consists of three channels; represents a collection of infrared modality images, is a single-channel infrared light modality image; the relationship between each R, G, and B channel of the visible light modality image and the single-channel infrared image is directly learned through random channel data exchange, and its learning objectives are: The dataset after learning is represented as 3. The cross-modal person re-identification method using multi-scale and multi-branch features according to claim 2, characterized in that: In step S2, the completed dataset is learned The input is sent to the cross-modal feature extraction module for feature extraction, which at least includes: S21, the learned visible light modality image dataset and infrared light modality image dataset They are respectively input into the two-stream network of the cross-modal feature extraction module, and the corresponding low-level features are extracted by the corresponding residual convolution module. and S22. Low-level features corresponding to visible light modality and infrared light modality and Then it is input into the shared network, and multiple residual convolution modules in the shared network extract features of different levels {F m1 ,F m2 ,F m3 }.

4. The cross-modal person re-identification method using multi-scale and multi-branch features according to claim 3 is characterized in that: In the two-stream network of step S21, the branch for processing the visible light modality image includes a residual convolution module and Residual convolution modules with different network parameters For the learned visible light modality image Processing to obtain specific low-level features of visible light modality images The branch for processing infrared modality images includes a residual convolution module and Residual convolution modules with different network parameters and For the infrared light modality image after learning Processing to obtain specific low-level features of infrared light modality images in, They are the convolution operations corresponding to the residual convolution modules respectively.

5. The cross-modal person re-identification method using multi-scale and multi-branch features according to claim 4, characterized in that: In the shared network of step S22, at least three residual convolution modules are set To obtain different levels of feature representation, the extracted features are: The final output is a three-level image feature, represented as {F m1 ,F m2 ,F m3 }, each layer of feature images is input into the corresponding self-attention layer in turn.

6. The cross-modal person re-identification method using multi-scale and multi-branch features according to claim 3, characterized in that: In step S3, the extracted multi-level features {F m1 ,F m2 ,F m3 } are respectively input into the self-attention layer IF of the multi-scale information mining module, and finally the deep features F containing shallow features and middle features are obtained n , where in the first self-attention layer, F m3 Considered Each self-attention layer will and As its own input, it is mapped to the same space through convolutional layers and regularization, and then reduced in dimension through 1*1 convolution, and then shallow features are extracted through a multi-head attention mechanism. For the attention mechanism A, its query Q, key K and value V are expressed as: where F c (·) represents the dimension reduction operation, represents the equilibrium parameter, F conv (·) indicates a dimension-raising operation.

7. The cross-modal person re-identification method using multi-scale and multi-branch features according to claim 6, characterized in that: In step S4, the deep features F are extracted n Input to the multi-branch joint module for discriminative feature extraction. The multi-branch joint module includes a global branch and a local branch. The global branch extracts the discriminative features from the deep feature F n Extract the global feature F g , the local branch is derived from the deep feature F n Extract local features F l ,in, In the global branch, the deep features F n The three-dimensional features are reduced to one-dimensional features through generalized mean pooling. The processing process is expressed as: Among them, U represents the one-dimensional feature vector after the global feature is transformed, |·| represents the conversion matrix value, and p is the pooling hyperparameter; Then, channel dimension reduction is performed through a 1*1 convolution layer and a batch normalization layer to obtain the global feature F g .

8. The cross-modal person re-identification method using multi-scale and multi-branch features according to claim 7, characterized in that: In the local branch, the deep feature F n First, divide the block into three equal parts horizontally to obtain three local feature vectors F1 n 、F2 n 、F3 n , and then the three local eigenvectors F1 n 、F2 n 、F3 n The three-dimensional features are reduced to one-dimensional features U1, U2, and U3 through generalized mean pooling: Where i = 1, 2, 3, and then the three one-dimensional features are processed by U1, U2, and U3 channel dimensionality reduction through 1*1 convolution layer and batch normalization layer to obtain local features F l .

9. A cross-modal person re-identification method using multi-scale and multi-branch features according to any one of claim 8, characterized in that: In step S5, the global feature F is extracted g and local features F l Finally, the model parameters are continuously optimized by feeding the triplet loss and identity verification loss back to the model. During the recognition process, the Euclidean distance between the input image and each image in the image library after model training is calculated through the loss function, and the images in the library are sorted in ascending order of distance to obtain the first n images with the smallest distance in the library, that is, the images with the highest similarity, recorded as Rank_k. Rank_k represents the hit rate of the first k results that are similar to the query image. The calculation formula of Rank_k is as follows: In the formula, It indicates the number of correct results among k query results, and N indicates the total number of pedestrian images in the database.

10. A cross-modal person re-identification method using multi-scale and multi-branch features according to any one of claim 9, characterized in that: The loss function during training is designed as: Among them, L ID represents the authentication loss, which is expressed as: Where N represents the number of pedestrian images contained in a batch, W j represents the weight vector of the jth class in the fully connected layer, x i Indicates that it belongs to y i The feature vector extracted from the i-th sample of the category, b and T represent the bias and transposition, and M is the number of pedestrian identity categories; and are the triplet loss with visible light modality image as anchor point and the triplet loss with infrared light modality image as anchor point, respectively. tr It is expressed as: Where a, p, n represents a benchmark triplet sample pair, representing the anchor sample image, the positive sample image, and the negative sample image, respectively; D(·) represents the Euclidean distance, F(·) represents the feature extractor, [·] + represents the clipping function max(·,0), and margin is used to regulate the distance between the feature vectors of the same pedestrian in the common space to be smaller than the distance between different pedestrian identities.

Images