A multi-scale multi-branch feature utilization cross-modal pedestrian re-identification method

Abstract

The present application relates to a kind of multi-scale multi-branch feature utilization's cross-modal pedestrian re-identification method, belong to cross-modal pedestrian re-identification field.It includes: obtaining original data set, and through random channel learning the relationship between each channel of visible light modality image in original data set and single-channel infrared light modality image;The processed image is input into cross-modal feature extraction module, wherein, first, the different attributes of different modalities are processed by double-flow network, then multi-scale feature is extracted by shared network;Multi-scale feature is input into multi-scale information mining module for feature fusion;Fusion feature is extracted by multi-branch joint module again, and the discriminant feature extracted by each branch cooperates to carry out pedestrian re-identification.The present application combines multi-level spatial feature information to extract features, and uses global feature and local feature to increase the correlation between infrared modality and visible light modality in visual representation, improves the search accuracy of cross-modal pedestrian re-identification.

Description

Technical Field

[0001] This invention belongs to the field of cross-modal pedestrian re-identification and relates to a cross-modal pedestrian re-identification method that utilizes multi-scale and multi-branch features. Background Technology

[0002] Intelligent video surveillance technology is an indispensable part of urban construction. By analyzing surveillance videos, it can greatly improve the safety, comfort, and convenience of urban residents' lives. Pedestrian search refers to the task of searching for specific pedestrians under multiple cameras. This technology can determine whether a specified pedestrian appears on multiple cameras. With the increasing public safety demands and the growing number of surveillance cameras, traditional manual monitoring can no longer meet the needs of the big data era. Machine learning is being used to improve intelligent surveillance technology, and pedestrian search has significant research value and practical application value. Deep learning-based methods can effectively analyze video surveillance data. Pedestrian re-identification is a hot topic in the field of deep learning. Because it can automatically match the pedestrian to be searched in video surveillance information and accurately locate the pedestrian's position, it has considerable application prospects in intelligent monitoring and security fields in urban construction. With the improvement of camera functions, cameras can automatically switch to capturing infrared images of pedestrians at night. In this case, the infrared images actually reveal pedestrian features more clearly than visible light images at night. However, infrared images also suffer from drawbacks such as a lack of color information and blurred faces and pedestrians. Therefore, biometric recognition technology in cross-modal scenarios has become a new challenge and opportunity. Cross-modal biometric recognition technology needs to consider the differences in image quality, distribution, and representation between different modalities, and design effective algorithms to achieve cross-modal matching and retrieval.

[0003] Cross-modal recognition aims to study the mutual recognition between data from different modalities (infrared, sketch, low-resolution images, color images). The challenge for such models is how to combine feature information extracted from data from different modalities, and how the model can rely on other modalities to complete the task when one modality fails. In this study, cross-modal recognition specifically refers to the mutual recognition between infrared (near-infrared) and visible light (color) modal images. In the field of person re-identification, cross-modal recognition is generally referred to as cross-modal person re-identification.

[0004] With the continuous development of cross-modal recognition tasks, research in this field also faces new challenges. These mainly focus on how to handle the differences between images from different modalities. Since infrared cameras only utilize infrared light for imaging, infrared images contain only single-channel information, namely the intensity of infrared light. This means that infrared images lack the important visual feature of color, only reflecting information such as the structure and texture of objects. Therefore, in cross-modal scenarios, there will be significant differences between color images and infrared images, inconsistencies in content, distribution, and representation.

[0005] Cross-modal person re-identification has garnered significant attention due to its higher difficulty level compared to traditional single-modal person re-identification and its closer resemblance to real-world scenarios. Key methods include designing network models and processing pedestrian features to reduce cross-modal differences; using image generation to improve model accuracy; and designing loss functions to enhance recognition accuracy. Ye et al. combined modality sharing matrices and optimized modality specificity to transform extracted features from two modalities into a common space, enabling the learning of modality sharing metrics for model training. They also designed a two-stream network structure capable of learning cross-modal shareable features. Wang et al. used GANs to unify infrared or visible light images into images of the same style, then extracted features from these style images and used feature embedding to reduce appearance differences, thereby minimizing inter-modal discrepancies. Zhu et al. proposed a heterogeneous center loss, which trains the network to learn invariant information about cross-modal pedestrian image features by narrowing the intra-class center distance between visible light and infrared modalities in a common feature space. This is then combined with an identity verification loss to reduce inter-class and intra-class differences.

[0006] Although existing research has attempted to reduce these differences through methods such as Generative Adversarial Networks (GANs) and shared matrix optimization, the quality of the generated images, the accuracy of feature transformation, and the generalization ability of the algorithms still need improvement. In particular, ensuring that the model can learn cross-modal invariant features when dealing with highly heterogeneous image pairs remains a challenge for current technology. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide a cross-modal pedestrian re-identification method that utilizes multi-scale and multi-branch features.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] A cross-modal person re-identification method utilizing multi-scale, multi-branch features includes the following steps:

[0010] S1. Obtain the original dataset, which includes visible light modal images and infrared light modal images, and learn the relationship between each channel in the visible light modal image and the single-channel infrared light modal image through random channels;

[0011] S2. Input the visible light modal image and infrared light modal image after random channel processing into the cross-modal feature extraction module. First, the different attributes of different modalities are processed through a dual-stream network, and then multi-scale features are extracted through a shared network.

[0012] S3. Input the multi-scale features into the multi-scale information mining module for feature fusion. The multi-scale information mining module includes multiple self-attention layers, which sequentially fuse features at different scales.

[0013] S4. The fused features are then used by the multi-branch joint module to extract discriminative features, which includes global branches and local branches. Global features and local features are extracted through global branches and local branches.

[0014] S5. Apply global and local features to the back feedback process during training, and use triplet loss and identity verification loss to iteratively optimize the model; during real-time recognition, calculate similarity through the loss function and sort to obtain the re-recognition results.

[0015] Furthermore, in step S1, the obtained original dataset is represented as D = {D} v D r},in, This represents a set of visible light modal images, where each image... It consists of three channels; This represents a set of infrared light modal images. The image is a single-channel infrared modal image. The relationship between each R, G, and B channel of the visible light modal image and the single-channel infrared image is directly learned through random channel data exchange. The learning objective is:

[0016]

[0017] The dataset after learning is represented as follows

[0018] Furthermore, in step S2, the learned dataset... The input is fed into the cross-modal feature extraction module for feature extraction, which includes at least:

[0019] S21. The learned visible light modal image dataset and infrared light modal image dataset The inputs are fed into the two-stream network of the cross-modal feature extraction module, where the corresponding residual convolution modules extract the corresponding low-level features. and

[0020] S22. Low-level features corresponding to visible light modes and infrared light modes. and The data is then input into a shared network, where multiple residual convolutional modules extract features at different levels {F}. m1 ,F m2 ,F m3}

[0021] Furthermore, in the dual-stream network of step S21, a residual convolution module is included on the branch used for processing the visible light modal image. and Using residual convolution modules with different network parameters The learned visible light modal images Processing is performed to obtain specific low-level features of the visible light modal image.

[0022]

[0023] The branch used for processing infrared modal images includes a residual convolution module. and Using residual convolution modules with different network parameters and The learned infrared modal images Processing is performed to obtain specific low-level features of the infrared modal image.

[0024]

[0025] in, These are the convolution operations for the corresponding residual convolution modules.

[0026] Furthermore, in the shared network of step S22, at least three residual convolutional modules are set up. To obtain feature representations at different levels, the extracted features are as follows:

[0027]

[0028]

[0029] The final output is a three-level image feature, represented as {F m1 ,F m2 ,F m3 Each feature image is sequentially input into the corresponding self-attention layer.

[0030] Furthermore, in step S3, the extracted multi-level features {F m1 ,F m2 ,F m3 The inputs are respectively fed into the self-attention layer IF of the multi-scale information mining module, and finally the deep feature F containing shallow and medium-level features is obtained. n In the first self-attention layer, F m3 View as Each self-attention layer will and As its own input, it is mapped to the same space through convolutional layers and regularization, then dimensionality is reduced through 1*1 convolution, and shallow features are extracted through a multi-head attention mechanism. For attention mechanism A, its query Q, key K, and value V are respectively represented as:

[0031]

[0032] Where F c (·) indicates a dimensionality reduction operation. F represents the equilibrium parameter. conv (·) indicates a dimension-up operation.

[0033] Furthermore, in step S4, the extracted deep features F n The input is fed into a multi-branch joint module for discriminative feature extraction. This module includes a global branch and local branches. The global branch extracts features from deep features F. n Extract global features F g Local branches originate from deep features F n Extracting local features F l ,in,

[0034] In the global branch, deep features F n The generalized mean pooling method reduces three-dimensional features to one-dimensional features. The process is as follows:

[0035]

[0036] In the formula, U represents the one-dimensional feature vector after global feature transformation, |·| represents the transformation matrix value, and p is the pooling hyperparameter;

[0037] Then, channel dimensionality reduction is performed using 1*1 convolutional layers and batch normalization layers to obtain the global feature F. g .

[0038] Furthermore, in local branches, deep features F n First, the horizontal direction is divided into three equal parts to obtain three local feature vectors F1. n F2 n F3 nThen, the three local feature vectors F1 n F2 n F3 n The three-dimensional features are reduced to one-dimensional features U1, U2, and U3 by generalized mean pooling:

[0039]

[0040] Where i = 1, 2, 3, and then the three one-dimensional features are reduced in dimensionality through channels U1, U2, and U3 by a 1*1 convolutional layer and a batch normalization layer to obtain the local feature F. l .

[0041] Furthermore, in step S5, the global feature F is extracted. g and local features F l Then, by feeding the triplet loss and authentication loss back to the model, the model parameters are continuously optimized;

[0042] During the recognition process, the Euclidean distance between the input image and each image in the image database after model training is calculated using a loss function. The images in the database are then sorted in ascending order of distance to obtain the top n images with the smallest distance, i.e., the images with the highest similarity, denoted as Rank_k. Rank_k represents the hit rate of similarity to the query image among the top k results. The formula for calculating Rank_k is as follows:

[0043]

[0044] In the formula, This represents the number of correct results found in k query results, and N represents the total number of pedestrian images tested in the database.

[0045] Furthermore, the loss function during training is designed as follows:

[0046]

[0047] Among them, L ID The authentication loss is represented as:

[0048]

[0049] In the formula, N represents the number of pedestrian images contained in a batch, and W j Let x represent the weight vector of class j in the fully connected layer. i Indicates belonging to y i The feature vector extracted from the i-th sample of the category, where b and T represent the bias and transpose, and M is the number of pedestrian identity categories;

[0050] and These are the triplet losses anchored by visible light modal images and infrared light modal images, respectively. The triplet loss L... tr Represented as:

[0051]

[0052] In the formula, a, p, n represent a baseline triplet sample pair, representing the anchor sample map, positive sample map, and negative sample map, respectively; D(·) represents the Euclidean distance, F(·) represents the feature extractor, [·] + The clipping function is max(·,0), and margin is used to ensure that the distance between the feature vectors of the same pedestrian in the public space is less than the distance between the identities of different pedestrians.

[0053] The beneficial effects of this invention are as follows:

[0054] This invention acquires features at different levels and scales, extracting feature information from the infrared and visible light modalities at the shallow level, and extracting shared information at the mid and deep levels. During the mining of associated information, a multi-head attention mechanism is used as a basic block to further mine valuable parts of the shallow information. A combination of global and local features is used as the final pedestrian features to enhance the expressive power of the fused features. Furthermore, identity verification loss and sample triplet loss are used for training, which can improve the accuracy of cross-modal pedestrian re-identification and retrieval.

[0055] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0056] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0057] Figure 1 This is a schematic diagram of the overall framework of the cross-modal person re-identification method utilizing multi-scale and multi-branch features of the present invention.

[0058] Figure 2 This is a schematic diagram of the processing of the self-attention layer in the multi-scale information mining module of the present invention. Detailed Implementation

[0059] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0060] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0061] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0062] Please see Figures 1-2 This is a cross-modal pedestrian re-identification method that utilizes multi-scale and multi-branch features.

[0063] Example

[0064] In this embodiment, as Figure 1 The diagram shown illustrates the overall framework of a cross-modal person re-identification method utilizing multi-scale, multi-branch features, which includes the following steps:

[0065] S1. Obtain the original dataset, which includes visible light modal images and infrared light modal images, and learn the relationship between each channel in the visible light modal image and the single-channel infrared light modal image through random channels;

[0066] S2. Input the visible light modal image and infrared light modal image after random channel processing into the cross-modal feature extraction module. First, the different attributes of different modalities are processed through a dual-stream network, and then multi-scale features are extracted through a shared network.

[0067] S3. Input the multi-scale features into the multi-scale information mining module for feature fusion. The multi-scale information mining module includes multiple self-attention layers, which sequentially fuse features at different scales.

[0068] S4. The fused features are then used by the multi-branch joint module to extract discriminative features, which includes global branches and local branches. Global features and local features are extracted through global branches and local branches.

[0069] S5. Apply global and local features to the back feedback process during training, and use triplet loss and identity verification loss to iteratively optimize the model; during real-time recognition, calculate similarity through the loss function and sort to obtain the re-recognition results.

[0070] Cross-modal person re-identification refers to the retrieval of data between infrared and visible light modalities, typically using one modality as the query to retrieve relevant data from the other modality. The core tasks are: spatial data feature extraction and measurement of the correlation between content from different modalities.

[0071] Cross-modal person re-identification tasks require models to learn more discriminative person representations. Therefore, this invention combines multi-level spatial feature information during convolution to perform comprehensive feature extraction. Simultaneously, it utilizes both global and local features to increase the correlation between infrared and visible light modalities in the visual representation (inter-modal). This improves the retrieval accuracy of cross-modal person re-identification to a certain extent.

[0072] In step S1 of this embodiment, a ResNet50 network is used to extract visual features. The original dataset contains images of both visible and infrared light. The original dataset is represented as D = {D} ν D r} This represents the collection of visible light modal images in the dataset, where each image... It consists of three channels. This represents the set of infrared light modal images in the dataset. It is a single-channel infrared image. To reduce the gap between input image levels while preserving rich color information, the relationship between each R, G, and B of the visible light image and the single-channel infrared image is directly learned through random channel data exchange. This optimizes the relationship between the features of the three-channel visible light image and the single-channel infrared image. The learning objective formula is:

[0073]

[0074] The dataset after learning is represented as follows

[0075] In step S2 of this embodiment, the purpose of the cross-modal feature extraction module is to obtain feature embedding representations of visible light modal and infrared modal images. Due to different cameras, images of different modalities have significant differences in visual presentation. Visible light modal images contain rich spatial structure information, while infrared light modal images have more spatial texture information. The ResNet50 network is also used to extract image features. Since the input consists of pedestrian images of two different modalities, the two-stream network must not only capture the specific features of pedestrian images of different modalities but also consider extracting shared features from pedestrian images of different modalities. To handle the different attributes of different modalities, the network parameters of the first two residual convolutional blocks in each stream are different to capture modality-specific low-level features. Subsequently, in subsequent residual convolutional blocks, the network parameters are shared to learn the intermediate layer feature representations between modalities. The modality-specific image feature representations extracted by the ResNet50 shared network are represented as {F}. m1 ,F m2 ,F m3 This reduces the number of model parameters, thereby shortening the training time.

[0076] Specifically, in the two-stream network of the cross-modal feature extraction module, a residual convolution module is included on the branch used to process visible light modal images. and Using residual convolution modules with different network parameters The learned visible light modal images Processing is performed to obtain specific low-level features of the visible light modal image.

[0077]

[0078] The branch used for processing infrared modal images includes a residual convolution module. and Using residual convolution modules with different network parameters and The learned infrared modal images Processing is performed to obtain specific low-level features of the infrared modal image.

[0079]

[0080] in, These are the convolution operations for the corresponding residual convolution modules.

[0081] In a shared network, at least three residual convolutional modules should be configured. To obtain feature representations at different levels, the extracted features are as follows:

[0082]

[0083] The final output is a three-layer image feature, represented as {F m1 ,F m2 ,F m3 Each feature image is also sequentially input into the corresponding self-attention layer.

[0084] In step S3 of this embodiment, as the network depth increases, the extracted features become increasingly refined. The goal is to integrate these refined features with features containing shallow information to enrich the feature information existing in the deeper layers of the network. In this embodiment, the features will pass through a residual convolution module. Extracted features F m3 Considered right and The feature information is extracted using a multi-head attention mechanism as the basic block IF, and the comprehensive features are extracted and then the output value is set after self-attention extraction. Again and The features are used as input to the next self-attention layer for multi-scale feature extraction. The features after passing through the multi-scale information mining (MIF) module, which consists of triple self-attention information extraction, are set as F. n .

[0085] The multi-scale information mining module consists of three layers of in-process statements (IFs), with each IF receiving the following input: and F n It is the input in the main branch, representing deep features. And F... m This represents the output retained by the shallower layers of the network, indicating lost information. Convolutional layers and regularization map these features to the same spatial dimension, followed by 1x1 convolution for dimensionality reduction. Multi-head attention mechanisms can extract valuable parts from the shallow features and uncover correlations between them. For example... Figure 2 As shown, let the attention mechanism be denoted as A, where the query Q, key K, and value V are respectively:

[0086]

[0087] Where F c It is to reduce the dimensionality. This represents the balance parameter, set here as follows: To facilitate using the output of IF as the input to the next layer, a convolutional layer F is finally applied. conv By increasing the dimension of the features after the attention mechanism is added, the function of a single-layer IF layer is thus realized.

[0088] In step S4 of this embodiment, a multi-branch joint method is used to extract features from different granularities, thereby obtaining more powerful discriminative features, mainly including global branches and local branches. The global feature F of the pedestrian... g and local features F l There is a complementary relationship between the two, both playing a crucial role in pedestrian retrieval. The global branch identifies pedestrians from a holistic perspective, while the local branch extracts features from local pedestrian characteristics for identification. For the global branch, the extracted features are reduced from a 3D feature map to a 1D feature map using Generalized Mean Pooling (GEM), and then further reduced in dimensionality through a 1*1 convolutional layer and a batch normalization (BN) layer. The GEM formula is:

[0089]

[0090] Where U represents the transformed one-dimensional feature vector, |·| represents the transformation matrix value, and p is the pooling hyperparameter.

[0091] The formula for a 1x1 convolution operation is:

[0092] Z = W·U + b

[0093] Z is the output feature map after 1*1 convolution, W is the weight of the convolution kernel, U is the input feature map, and b is the bias.

[0094] The batch normalization formula is:

[0095]

[0096] This is the normalized output, where μ is the mean of the current batch, and σ... 2 is the variance of the current batch, ∈ is a small constant to prevent division by zero errors, γ is the scaling parameter, and β is the translation parameter.

[0097] After passing through a fully connected layer, the output is:

[0098]

[0099] F g It is the output of the global features after a series of dimensionality reductions and fully connected layers, where W is the weight matrix and b is the bias term.

[0100] Local feature branching for the extracted multi-scale pedestrian features F n Divide the horizontal direction into three equal parts to obtain three local feature vectors F1. n F2 n F3 n Then perform the GEM operation, followed by a 1*1 convolutional layer and a BN layer.

[0101] The GeM formula is:

[0102]

[0103] Among them U i This is the transformed one-dimensional feature vector, where i = 1, 2, 3. F i n It is a local feature vector.

[0104] The formula for 1x1 convolution is:

[0105] Z i =W·U i +b

[0106] Z i is the output after convolution, W is the convolution kernel weight matrix, and b is the bias vector.

[0107] The batch normalization formula is:

[0108]

[0109] This is the feature map after convolution.

[0110] Next, the three local feature maps are fed into a fully connected layer. Before inputting the feature maps into the fully connected layer, they typically need to be flattened, transforming them from a 1×n×1×1 one-dimensional matrix into a 3×n×1×1 two-dimensional matrix. The flattened feature maps are named...

[0111] The final output after passing through the fully connected layer is:

[0112]

[0113] F l is the output of the fully connected layer, W is the weight matrix, and b is the bias term.

[0114] In step S5 of this embodiment, ResNet50 is used as the base model. A two-stream network is used to extract specific image features in stage 0 and stage 1 respectively, and common features are extracted in the residual shared parameters of the last three layers. To obtain multi-scale fused features, deep and shallow features are integrated through the MIF module to enrich the information of deep features. Finally, through a multi-branch structure of global and local features, and by constraining the feature learning network training with a joint loss function, a stronger ability to identify pedestrians is achieved. Combining the obtained global and local features, a reverse feedback process is implemented during training. The triplet loss and authentication loss are fed back to the model during training to update the model's parameters. By continuously iterating and optimizing the model, it can better understand the similarities and differences in the data. The loss calculated by the loss function is returned to the model during training, and the parameters are adjusted to continuously optimize the model. Similarly, during testing, the loss function can calculate similarity, and the results are obtained by ranking by similarity.

[0115] In this embodiment, the obtained branch features are subjected to triplet loss learning and identity (ID) loss learning. The formula for triplet loss is expressed as:

[0116]

[0117] Where a, p, n represent a baseline triplet sample pair, representing the anchor sample image, positive sample image, and negative sample image, respectively. D(·) represents the Euclidean distance, F(·) represents the feature extractor, [·] + The clipping function `max(·, 0)` and `margin` ensure that the distance between feature vectors of the same pedestrian in the common space is less than the distance between different pedestrian identities. Furthermore, the triplet loss function for cross-modal pedestrian re-identification needs to consider two cases: those using infrared images as anchor samples... and anchored by visible light images Therefore, the triplet loss L tr It should be:

[0118]

[0119] After passing through the multi-branch joint module, the features of each branch are normalized using the identity (ID) verification loss after passing through a fully connected layer. The formula for the identity verification loss is expressed as:

[0120]

[0121] In the formula, N represents the number of pedestrian images contained in a batch, and W... j Let x represent the weight vector of class j in the fully connected layer. i Indicates belonging to y iThe feature vector extracted from the i-th sample of the category, where b and T represent the bias and transpose, and M is the number of pedestrian identity categories.

[0122] Therefore, the overall loss function is expressed as:

[0123]

[0124] The network model undergoes multiple rounds of training. The training process involves extracting features and then feeding back the triplet loss and authentication loss to the model to continuously optimize its parameters. Because the triplet loss is represented by a distance vector in Euclidean space, the distance between samples of different classes is much greater than the distance between samples of the same class. Similarly, the authentication loss is also suitable for tasks that determine the similarity between samples.

[0125] Therefore, the recognition process actually calculates the Euclidean distance between the input image and each image in the image database after model training using a loss function. The images in the database are then sorted in ascending order of distance to obtain the top n images with the smallest distance, which are the images with the highest similarity, denoted as Rank_k. Rank_k represents the hit rate of similarity to the query image among the top k results. The formula for calculating Rank_k is as follows:

[0126]

[0127] The number of queries that satisfy the condition of finding the correct result among the k query results is denoted as . The total number of pedestrian images in the database is denoted as N.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A cross-modal person re-identification method utilizing multi-scale, multi-branch features, characterized in that: It includes the following steps: S1. Obtain the original dataset, which includes visible light modal images and infrared light modal images, and learn the relationship between each channel in the visible light modal image and the single-channel infrared light modal image through random channels; S2. Input the visible light modal image and infrared light modal image after random channel processing into the cross-modal feature extraction module. First, the different attributes of different modalities are processed through a dual-stream network, and then multi-scale features are extracted through a shared network. S3. Input the multi-scale features into the multi-scale information mining module for feature fusion. The multi-scale information mining module includes multiple self-attention layers, which sequentially fuse features at different scales. S4. The fusion features are then used by the multi-branch joint module to extract discriminative features, which includes global branches and local branches. Global features and local features are extracted through global branches and local branches. S5. Apply global and local features to the back feedback process during training, and iteratively optimize the model using triplet loss and identity verification loss; calculate similarity and sort the results through the loss function during real-time recognition to obtain the re-recognition results; In step S2, the dataset that has been learned is... The input is fed into the cross-modal feature extraction module for feature extraction, which includes at least: S21. The learned visible light modal image dataset and infrared light modal image dataset The inputs are fed into the two-stream network of the cross-modal feature extraction module, where the corresponding residual convolution modules extract the corresponding low-level features. and ; S22. Low-level features corresponding to visible light modes and infrared light modes. and The data is then fed into a shared network, where multiple residual convolutional modules extract features at different levels. ; In step S3, the extracted multi-level features The data are input into the self-attention layer (IF) of the multi-scale information mining module, ultimately yielding deep features containing both shallow and mid-level features. In the first self-attention layer, View as Each self-attention layer will and As its own input, it is mapped to the same space through convolutional layers and regularization, and then passed through 1 1. Convolutional layer is used for dimensionality reduction, and then shallow features are extracted using a multi-head attention mechanism. Specifically, regarding the attention mechanism... The query Q, key K, and value V are represented as follows: in This indicates a dimensionality reduction operation. Represents the balance parameters. This indicates a dimensional ascension operation; In step S4, the extracted deep features The input is fed into a multi-branch joint module for discriminative feature extraction. This module includes a global branch and local branches. The global branch extracts features from deep layers. Extract global features Local branches from deep features Extracting local features ,in, In the global branch, deep features The generalized mean pooling method reduces three-dimensional features to one-dimensional features. The process is as follows: in, This represents a one-dimensional feature vector after global feature transformation. Represents the transformation matrix values. For pooling hyperparameters; Then through 1 1. Convolutional layers and batch normalization layers are used for channel dimensionality reduction to obtain global features. ; Deep features in local branches First, the horizontal direction is divided into three equal parts to obtain three local feature vectors. , , Then, the three local feature vectors , , Generalized mean pooling is used to reduce three-dimensional features to one-dimensional features. , , : Where i = 1, 2, 3, and then through 1 1. Convolutional layers and batch normalization layers process three one-dimensional features. , , Channel dimensionality reduction processing yields local features ; In step S5, global features are extracted. and local features Then, by feeding the triplet loss and authentication loss back to the model, the model parameters are continuously optimized; During the recognition process, the Euclidean distance between the input image and each image in the image database after model training is calculated using a loss function. The images in the database are then sorted in ascending order of distance to obtain the top n images with the smallest distance, i.e., the images with the highest similarity, denoted as . , Indicates the preceding The hit rate of images similar to the query image among the results. The calculation formula is as follows: In the formula, Indicates in The number of results in the query that satisfy the condition of a correct match. This represents the total number of pedestrian images tested in the database.

2. The cross-modal person re-identification method utilizing multi-scale, multi-branch features according to claim 1, characterized in that: In step S1, the obtained raw dataset is represented as follows: ,in, This represents a set of visible light modal images, where each image... It consists of three channels; This represents a set of infrared light modal images. The image is a single-channel infrared modal image. The relationship between each R, G, and B channel of the visible light modal image and the single-channel infrared image is directly learned through random channel data exchange. The learning objective is: The dataset after learning is represented as follows .

3. The cross-modal person re-identification method utilizing multi-scale, multi-branch features according to claim 1, characterized in that: In the dual-stream network of step S21, the branch used for processing visible light modal images includes a residual convolution module. and Through residual convolution modules with different network parameters The learned visible light modal images Processing is performed to obtain specific low-level features of the visible light modal image. : The branch used for processing infrared modal images includes a residual convolution module. and Through residual convolution modules with different network parameters and The learned infrared light modal images Processing is performed to obtain specific low-level features of the infrared modal image. : in, , , , These are the convolution operations for the corresponding residual convolution modules.

4. The cross-modal person re-identification method utilizing multi-scale, multi-branch features according to claim 3, characterized in that: In the shared network of step S22, at least three residual convolutional modules are set up. , , To obtain feature representations at different levels, the extracted features are as follows: The final output is a three-level image feature, represented as Each feature image is sequentially input into the corresponding self-attention layer.

5. The cross-modal person re-identification method utilizing multi-scale, multi-branch features according to any one of claims 4, characterized in that: The loss function during training is designed as follows: in, The authentication loss is represented as: In the formula, This represents the number of pedestrian images contained in a batch. Let represent the weight vector of class j in the fully connected layer. Indicates belonging to Category 1 The feature vectors extracted from each sample, where b and T represent the bias and transpose, and M is the number of pedestrian identity categories; and These are the triplet losses anchored by visible light modal images and by infrared light modal images, respectively. The triplet loss... Represented as: In the formula, This represents a baseline triplet sample pair, which represents the anchor sample map, the positive sample map, and the negative sample map, respectively. Represents Euclidean distance. Indicates feature extractor, Represents the clipping function , The distance between feature vectors of the same pedestrian in the public space is less than the distance between different pedestrian identities.

Images