A low-rank multi-scale multi-modal pedestrian re-identification method, electronic device and storage medium

Abstract

The present invention provides a low-rank multi-scale multi-modal pedestrian re-identification method, electronic device and storage medium, belonging to the field of deep learning technology; including obtaining pedestrian images containing three modes of RGB, near infrared and thermal infrared, and inputting the pedestrian images of the three modes into a three-branch feature extraction network, respectively obtaining RGB modal features extracted from an RGB camera, NI modal features extracted from near infrared and TI modal features extracted from thermal infrared; using a low-rank multi-scale multi-modal network model to calculate the three modal features separately to obtain multi-modal features; using the multi-modal features to perform pedestrian re-identification on pedestrians, and obtaining pedestrian re-identification results of the pedestrians. The present invention absorbs useful information in other modalities through a multi-scale multi-modal interaction module; and fuses information from the other two modalities through a low-rank multi-modal fusion module, so that the model can more efficiently fuse multiple modal features while maintaining low model complexity.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology, and in particular relates to a low-rank multi-scale multi-modal pedestrian re-identification method, an electronic device and a storage medium. Background Art

[0002] With the growth of global energy demand and the increasingly severe environmental problems, the development and utilization of clean and renewable energy has become a global consensus. As an important renewable energy source, wind energy has received widespread attention due to its clean and sustainable characteristics. As the main form of wind energy utilization, wind farms have developed rapidly around the world. However, with the increase in wind power installed capacity, wind farms face a series of challenges in design, construction, operation and maintenance, such as the reliability of wind turbines, efficient utilization of wind energy resources, stability of the power grid, and intelligent management. In this context, the introduction of mobile robot technology, especially pedestrian recognition technology, has brought new possibilities for the intelligent operation and maintenance of wind farms. Combining the elements of mobile robot pedestrian recognition can not only promote technological innovation, but also improve the operating efficiency and safety of wind farms.

[0003] At the same time, in order to ensure all-weather recognition, how to combine RGB images taken during the day with near-infrared (NIR) images taken at night has become a challenge. The popularity of infrared cameras has made RGB and near-infrared cross-modal pedestrian re-identification a new trend, which alleviates the limitations of traditional RGB in low illumination. However, this method also exposes the problem of heterogeneity between modalities, that is, it is impossible to process information from three modalities at the same time. In order to overcome the imaging limitations of complex visual environments, RGB-NI-TI multimodal pedestrian re-identification was attempted, providing each pedestrian with its own three modal information, namely RGB, near-infrared and thermal infrared (TI) images.

[0004] Most existing person re-identification tasks are based on a single modality, such as RGB, or RGB and near-infrared. Although the introduction of near-infrared modality significantly improves the ability in low-light conditions, the importance of modality-related information is ignored. Summary of the invention

[0005] The purpose of the present invention is to introduce more supplementary information to assist the traditional pedestrian re-identification task. Therefore, a pedestrian re-identification method, electronic device and storage medium based on RGB-NI-TI multimodality are proposed.

[0006] The present invention provides a low-rank multi-scale multi-modal pedestrian re-identification method, comprising the following steps:

[0007] Step 1: Obtain pedestrian images containing three modes: RGB, near infrared, and thermal infrared, and input the pedestrian images of the three modes into the three-branch feature extraction network to obtain the RGB modal features extracted from the RGB camera respectively. , NI modal features extracted from near infrared And TI modal features extracted from thermal infrared ;

[0008] Step 2: Use a low-rank multi-scale multi-modal network model to transform the RGB modal features , NI Modal Characteristics and TI modal characteristics Calculate and obtain multimodal features , including the following steps:

[0009] The low-rank multi-scale multi-modal network model includes a multi-scale multi-modal interaction module, a feature embedding module and a low-rank multi-scale multi-modal fusion module;

[0010] RGB modality features , NI Modal Characteristics and TI modal characteristics The multi-scale multi-modal interaction modules are used to perform feature interaction and obtain RGB multi-scale interaction features. , NI multi-scale interactive features and TI multi-scale interactive features ;

[0011] RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features The RGB multi-scale interactive features are respectively embedded in the feature embedding module , NI multi-scale interactive features and TI multi-scale interactive features The global feature context information is embedded into the corresponding local features to obtain the TI modality embedding enhanced features. , RGB modality embedding enhanced features and NI Modality Embedding Enhancements ;

[0012] TI modality embedding enhancements , RGB modality embedding enhanced features and NI Modality Embedding Enhancements The multimodal features are obtained by mutual fusion of low-rank multi-scale multi-modal fusion modules. ;

[0013] Step 3: Multimodal features Perform pedestrian re-identification on the pedestrian to obtain a pedestrian re-identification result of the pedestrian.

[0014] Furthermore, we can obtain multimodal features The specific process is as follows:

[0015] S2.1. Based on the multi-scale multi-modal interaction module, the RGB modal features are respectively , NI Modal Characteristics and TI modal characteristics Perform summation, enhancement and interaction processing in sequence to obtain RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features ;

[0016] S2.2, RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features Perform adaptive average pooling to obtain RGB multi-scale interactive features The global characteristics and Local features, NI multi-scale interactive features The global characteristics and The global features of local features and TI multi-scale interactive features and local features; the relationship-based embedding module transforms the RGB multi-scale interactive features into The global feature context information is embedded into each local feature, and the NI multi-scale interactive features are The global feature context information is embedded into each local feature and the TI multi-scale interactive feature The global feature context information is embedded into its local features to obtain the embedded enhanced features of the TI modality , RGB modality embedding enhanced features and NI Modality Embedding Enhancements ;

[0017] S2.3. Enhanced features for TI modality embedding , RGB modality embedding enhanced features and NI Modality Embedding Enhancements Fusion is performed to obtain fusion features .

[0018] Furthermore, TI multi-scale interaction features The specific process of obtaining is as follows:

[0019] ①, extract the RGB modal features of the last scale and the NI modal characteristics of the last scale Perform pixel-wise summation to obtain interaction features ;

[0020] ② In the interaction characteristics and TI modal characteristics All of the above are convolved in a 1×1 convolutional layer to obtain the interaction features after convolution. And the TI modal features after convolution ;

[0021] ③Introducing channel attention to interaction features To obtain enhanced features ;

[0022] ④、Convolve the TI modal features Add to Enhancement Features To obtain the TI modal interaction features of the last scale ;

[0023] ⑤. Using the same method as steps ① to ④, obtain the TI modal interaction characteristics of the second-to-last scale in sequence And the TI modal interaction characteristics of the third-to-last scale ;

[0024] ⑥、The TI modal interaction features of the last scale , the penultimate scale TI modal interaction features And the TI modal interaction characteristics of the third-to-last scale Directly downsample to the same size through quadratic linear interpolation and concatenate to obtain multi-scale interaction features of TI modality ;

[0025] ⑦ Multi-scale interaction features in TI mode Implement a 1×1 convolution layer on top to obtain multi-scale interactive features after convolution , that is, TI multi-scale interaction feature .

[0026] Furthermore, RGB multi-scale interactive features , NI multi-scale interactive features The acquisition process is specifically related to the TI multi-scale interaction feature The acquisition process is the same.

[0027] Furthermore, TI modality embedding enhances features The specific process of obtaining is as follows:

[0028] ① In the global features and Local Features Implement three 1×1 convolutional layers to obtain the global features after convolution and Local Features ;

[0029] ② Global features after convolution and Local Features Perform a dot product operation and then a normalized exponential function operation to obtain their similarity as a measure of the global features after convolution. and Local Features The weight of the relationship ;

[0030] ③, through the global features after convolution Assign weights and then Global Features Sum and get the sum feature ;

[0031] ④、Summation characteristics and local features Perform sum operation to obtain local enhanced features ;

[0032] ⑤. Use global features For each local feature Enhance and obtain all local enhancement features , all local enhancement features Splicing is performed to obtain the TI modality embedding enhanced features that embed global information ;

[0033] in, .

[0034] Furthermore, RGB modality embedding enhances features and NI Modality Embedding Enhancements The acquisition method and TI modality embedding enhancement feature The method of obtaining is the same.

[0035] Furthermore, a multimodal mixed loss function module is also provided in the low-rank multi-scale multimodal network model, and the multimodal mixed loss function module is used to verify the predicted classification results.

[0036] Furthermore, a multimodal mixed loss function module is used to reduce the total loss of the low-rank multi-scale multimodal network model in the training phase. The calculation process is as follows:

[0037] The cross entropy loss is used to classify the identity information of pedestrian images. Suppose the cross entropy loss of each pedestrian image is Calculated as:

[0038] ;

[0039] in, is the number of local features, is the total number of modes, For the In the modal The true value labels of local features, For the In the modal The predicted identity label of local features;

[0040] The multimodal prototype loss is used to limit the distance between each person's different modal feature distributions. The centers of any two modal feature distributions are taken and their penalty center distance is set as for:

[0041] ;

[0042] ;

[0043] in, For the first group The center of the distribution of any two modal features of an individual, For the second group The center of the distribution of any two modal features of an individual, is the total number of pedestrian categories, For the first group The number of images of any two modalities in an individual, For the second group The number of images of any two modalities in an individual, For the first group Personal Zhang modality image features, For the second group Personal Zhang modality image features, For the Sequential symbols of images in individuals;

[0044] The total loss of the low-rank multi-scale multi-modal network model during the training phase is for:

[0045] ;

[0046] ;

[0047] in, is a hyperparameter used to balance the contribution of the two loss functions, and during the entire training phase, set .

[0048] The present invention also provides an electronic device, comprising:

[0049] Memory, used to store programs;

[0050] A processor is used to implement the low-rank multi-scale multi-modal pedestrian re-identification method as described above by executing the program stored in the memory.

[0051] The present invention also provides a computer-readable storage medium, comprising a program, which can be executed by a processor to implement the low-rank multi-scale multi-modal pedestrian re-identification method as described above.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] The present invention proposes a pedestrian re-identification method based on RGB-NI-TI multimodality, which absorbs useful information from other modalities through a multi-scale multimodal interaction module; then, the information from the other two modalities is fused through a low-rank multimodal fusion module, so that the model can more efficiently fuse multiple modal features while maintaining low model complexity. Specifically, the method includes: by setting a multi-scale multimodal interaction module, first using a convolutional neural network to extract multi-scale feature maps, and then splicing them into a multi-channel feature vector; then by setting a low-rank multimodal fusion module, the module is responsible for integrating information from other modalities;

[0054] Furthermore, since each modality has its own specific advantages, a multimodal prototype loss is proposed to enhance the differences between the modalities, and this loss is introduced on the cross entropy loss, so that the model can learn the specific information of each modality by expanding the inter-class differences and improving the intra-class cross-modal similarity, so as to improve the model's ability to understand cross-modal information. Experimental results on RGBNT201 and other pedestrian re-identification datasets verify the effectiveness of this method.

[0055] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0057] Figure 1 Schematic diagram of the structure of the low-rank multi-scale multi-modal network model in the present invention;

[0058] Figure 2 yes Figure 1 Schematic diagram of the structure of the medium-scale multimodal interaction module;

[0059] Figure 3 yes Figure 1 Schematic diagram of the structure of the feature embedding module;

[0060] Figure 4 yes Figure 1 Schematic diagram of the structure of the medium and low rank multimodal fusion module;

[0061] Figure 5 is a schematic image of a pedestrian re-identification dataset in an embodiment of the present invention;

[0062] FIG6 (a) is a graph showing different hyperparameters in an embodiment of the present invention. The performance distribution diagram of the proposed LRMM on the RGBNT201 dataset is shown below;

[0063] FIG6( b ) is a graph showing different hyperparameters in an embodiment of the present invention. The performance distribution diagram of the proposed LRMM on the Market1501 dataset is shown below;

[0064] Figure 7 are different hyperparameters in the embodiments of the present invention The following is a schematic diagram of the results of the proposed LRMM on the RGBNT201 dataset;

[0065] Figure 8 is a schematic diagram of heat map results generated by LRMM for pedestrian images from the labeled Market1501 dataset in an embodiment of the present invention;

[0066] Fig. 9 2 is a schematic diagram comparing pedestrian re-identification effects in an embodiment of the present invention. DETAILED DESCRIPTION

[0067] In order to make the above-mentioned purposes, features and advantages of the present invention more clear and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. It should be noted that the drawings of the present invention are all simplified and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the implementation of the present invention; the "number" mentioned in the present invention is not limited to the specific number in the examples in the accompanying drawings; the directions or positional relationships indicated by "front", "middle", "back", "left", "right", "up", "down", "top", "bottom", "middle", etc. mentioned in the present invention are based on the directions or positional relationships shown in the drawings of the present invention, and do not indicate or imply that the devices or components referred to must have a specific direction, nor can they be understood as limitations on the present invention.

[0068] The present invention provides a low-rank multi-scale multi-modal pedestrian re-identification method, which specifically relates to a low-rank multi-scale multi-modal pedestrian re-identification method, an electronic device and a storage medium based on a mobile robot scenario, comprising the following steps:

[0069] Step 1: Obtain pedestrian images containing three modes: RGB, near infrared (NI), and thermal infrared (TI), and input the pedestrian images of the three modes into a three-branch feature extraction network (ResNet50 network) to obtain the RGB modal features extracted from the RGB camera respectively. , NI modal features extracted from near infrared And TI modal features extracted from thermal infrared ; where the subscript (1) indicates the characteristic scale.

[0070] Step 2: Use a low-rank multi-scale multi-modal network model to transform the RGB modal features , NI Modal Characteristics and TI modal characteristics Calculate and obtain multimodal features .

[0071] Preferably, see Figures 1 to 4 As shown in the figure, the low-rank multi-scale multimodal network model includes a multi-scale multimodal interaction module, a feature embedding module and a low-rank multimodal fusion module; the low-rank multi-scale multimodal network model is used to transform the RGB modal features. , NI Modal Characteristics and TI modal characteristics Calculate and obtain multimodal features The specific process is as follows:

[0072] S2.1. Based on the multi-scale multi-modal interaction module, the RGB modal features are respectively , NI Modal Characteristics and TI modal characteristics Perform summation, enhancement and interaction processing in sequence to obtain RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features ; Specifically, TI multi-scale interaction features The specific process of obtaining is as follows:

[0073] ①, extract the RGB modal features of the last scale and the NI modal characteristics of the last scale Perform pixel-wise summation to obtain interaction features ;

[0074] ② In the interaction characteristics and TI modal characteristics All of the above are convolved in a 1×1 convolutional layer to obtain the interaction features after convolution. And the TI modal features after convolution ;

[0075] ③ In order to emphasize the positive influence of features from other modalities, it is further proposed to introduce channel attention to the interaction features To obtain enhanced features ; Enhanced features The specific expression is as follows:

[0076] ;

[0077] in, Indicates channel attention.

[0078] ④、Convolve the TI modal features Add to Enhancement Features To obtain the TI modal interaction features of the last scale ; Interaction features of the TI modality at the last scale The specific expression is as follows:

[0079] ;

[0080] ⑤. Using the same method as steps ① to ④, obtain the TI modal interaction characteristics of the second-to-last scale in sequence And the TI modal interaction characteristics of the third-to-last scale ;

[0081] ⑥、The TI modal interaction features of the last scale , the penultimate scale TI modal interaction features And the TI modal interaction characteristics of the third-to-last scale Directly downsample to the same size through quadratic linear interpolation and perform function Concat (concatenation) to obtain the multi-scale interaction features of the TI modality , multi-scale interactive features of TI modality The specific expression is as follows:

[0082] ;

[0083] ⑦ Multi-scale interaction features in TI mode Implement a 1×1 convolution layer on top to obtain multi-scale interactive features after convolution (i.e. TI multi-scale interaction features ).

[0084] Preferably, RGB multi-scale interactive features , NI multi-scale interactive features The acquisition process of TI multi-scale interaction features is detailed in The acquisition process.

[0085] Preferably, after being processed by the multi-scale multimodal interaction module, the multi-scale interaction features of each modality integrate useful information of other modalities at different scales and retain information specific to the current modality.

[0086] S2.2, RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features Perform adaptive average pooling to obtain RGB multi-scale interactive features The global characteristics and Local features, NI multi-scale interactive features The global characteristics and Local features and TI multi-scale interactive features The global characteristics and local features, among which Greater than 1; the relationship-based embedding module embeds its global feature context information into its various local features to obtain TI modality embedding enhanced features respectively. , RGB modality embedding enhanced features and NI Modality Embedding Enhancements .

[0087] In the TI modal branch Local Features and global features Take it as an example, the embedding process is as follows:

[0088] (i) In the global feature and Local Features Implement three 1×1 convolutional layers to obtain the global features after convolution and Local Features ;in, ;

[0089] (ii) Global features after convolution and Local Features Perform a dot product operation and then a normalized exponential function operation to obtain their similarity as a measure of the global features after convolution. and Local Features The weight of the relationship ; Weight The specific expression is as follows:

[0090] ;

[0091] (iii) Through the global features after convolution Assign weights and then Global Features Sum and get the sum feature ;Summation characteristics The specific expression is as follows:

[0092] ;

[0093] (iv) For the summation characteristics and local features Perform sum operation to obtain local enhanced features ; Local enhancement features The specific expression is as follows:

[0094] ;

[0095] (v) Use global features For each local feature Enhance and obtain all local enhancement features , and then these local enhancement features Splicing is performed to obtain the TI modality embedding enhanced features that embed global information ; Embedded enhancement features of TI modality The specific expression is as follows:

[0096] ;

[0097] Preferably, the first Local Features and global features The same steps as the embedding process are used to obtain the RGB modality embedding enhancement features embedded with global information. and NI Modality Embedding Enhancements .

[0098] S2.3. Enhanced features for TI modality embedding , RGB modality embedding enhanced features and NI Modality Embedding Enhancements Fusion is performed to obtain fusion features The specific process is as follows:

[0099] In the multi-scale multimodal interaction module, in order to allow each modal feature to fully absorb the useful features from other modalities, the problem of multiple scales and multiple modalities is introduced, which will inevitably lead to an exponential increase in the number of parameters. In order to solve the problem of poor computational efficiency of tensor-based multimodal fusion methods, a low-rank multimodal fusion method is introduced. By decomposing tensors and weights in parallel, multimodal fusion is performed using modality-specific low-order factors. Avoiding the calculation of high-dimensional tensors reduces memory overhead and reduces the exponential time complexity to linear.

[0100] Formulate multimodal fusion as a multilinear function ,in yes The encoded information of a single modality, is represented as the fused feature, and the goal of multimodal fusion is to integrate the single-modal representation into a compact multimodal representation for downstream work. In order to be able to use a tensor to simulate the interaction between any modal subsets. It is proposed to give each It means adding a 1 after it. So the input tensor Calculated through the representation of a single mode:

[0101] ;

[0102] ;

[0103] in, is the input representation with 1 appended. Input tensor Through a linear layer Generate a vector express:

[0104] ;

[0105] in, is the weight, is the offset. yes rank tensor, so yes A tensor of order , with dimensions , additional Order is the size of the output representation In the process of tensor dot product, we can Seen as indivual Order tensor, that is, it can be divided into , each vector The vectors in the output Contribute a dimension to the output vector ,like Figure 5 There is an exact decomposition pattern into vectors:

[0106] ;

[0107] ;

[0108] The smallest possible decomposition is called the rank of the tensor. The set of vectors is called the rank of the original tensor Factorization. Fixed for , and then use Decomposition Factor To reconstruct a low-rank version of . These vectors can be reassembled into mode-specific low-rank factors. Let , then the mode The corresponding low-rank factor is Then the low-rank weight tensor It can be reconstructed using the following formula:

[0109] ;

[0110] Based on weight tensor The decomposition of , the original calculation fusion feature can be The formula is calculated as follows:

[0111] ;

[0112] in: Representing modality The corresponding low-rank factor, where Representing modality The corresponding encoding information, where is represented as the element-wise product of a series of tensors, and , Expressed as multiplication.

[0113] Only three modes are used in the present invention, so is 3, then , and the fusion features The calculation formula is:

[0114] ;

[0115] in and Respectively represent the low-rank factor and NI modal characteristics of the NI mode, and Represent the low-rank factor and RGB modality features of the RGB modality, respectively. and They represent the low-rank factor and TI modal characteristics of the TI mode respectively.

[0116] Step 3: Multimodal features Perform pedestrian re-identification on the pedestrian to obtain a pedestrian re-identification result of the pedestrian.

[0117] As a further solution of the present invention, in order to verify the predicted classification results, a multimodal mixed loss function module is also provided in the low-rank multi-scale multimodal network model; the multimodal mixed loss function module is used to reduce the total loss of the low-rank multi-scale multimodal network model in the training phase. The calculation process is as follows:

[0118] The purpose of cross-entropy loss (CE) is to extract identity-specific information for classification, but it does not constrain the network to effectively extract modal shared information. Relying solely on cross-entropy loss, the differences between pedestrians can be increased, but there is a huge gap in the feature distributions under different modalities in the same class of pedestrians. In order to limit the distance between the feature distributions of different modalities of each class, a multimodal prototype loss (MMP) is proposed, which can improve the intra-class cross-modal similarity. Under the joint supervision of cross-entropy and multimodal prototype losses, the network is trained to achieve the purpose of expanding inter-class differences and enhancing intra-class cross-modal similarities. The cross-entropy loss is used to classify the identity information of pedestrian images. Suppose the cross-entropy loss of each pedestrian image is Calculated as:

[0119] ;

[0120] in, is the number of local features, is the total number of modes, For the In the modal The true value labels of local features, For the In the modal The predicted identity label of local features;

[0121] The multimodal prototype loss is used to limit the distance between each person's different modal feature distributions. The centers of any two modal feature distributions are taken and their penalty center distance is set as for:

[0122] ;

[0123] ;

[0124] in, For the first group The center of the distribution of any two modal features of an individual, For the second group The center of the distribution of any two modal features of an individual, is the total number of pedestrian categories, For the first group The number of images of any two modalities in an individual, For the second group The number of images of any two modalities in an individual, For the first group Personal Zhang modality image features, For the second group Personal Zhang modality image features, For the Sequential symbols of images in individuals;

[0125] The total loss of the multi-scale multi-modal interaction module during the training phase is (i.e., multimodal mixed loss function) is:

[0126] ;

[0127] ;

[0128] in, is a hyperparameter used to balance the contribution of the two loss functions, and during the entire training phase, set .

[0129] As a further solution of the present invention, the present invention also provides an electronic device, which can be a processing device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the above-mentioned method.

[0130] The electronic device of this embodiment includes a memory, a processor, and a computer program stored in the memory; the processor executes the computer program in the memory to implement the steps of the method in the first embodiment above.

[0131] In some implementations, the memory may be a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory.

[0132] In some other implementations, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0133] As a further solution of the present invention, the present invention further provides a computer-readable storage medium corresponding to the above embodiment, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method in the above embodiment 1 are implemented.

[0134] Computer readable storage media can be tangible devices that hold and store instructions used by instruction execution devices. Computer readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any combination thereof.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present application may be implemented in various computer languages, for example, object-oriented programming languages ​​python and etc.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0138] Example:

[0139] 1. Experimental process

[0140] 1.1 Experimental Setup

[0141] See also Figure 5 As shown in FIG6 (including FIG6 (a) and FIG6 (b)), the present invention implements this model on the PyTorch framework and uses an NVIDIA GeForce RTX4090 GPU to accelerate the process. Three ResNet50 networks pre-trained on ImageNet are used as the backbone for extracting features (i.e., ResNet50 networks pre-trained on ImageNet for RGB camera images, near-infrared images, and infrared images are used as the backbone for extracting features); the mini-batch is set to 8, and the feature map after the multi-modal multi-scale interaction module is evenly divided into 6 strips; the size of each local feature is reduced to 128 by the FC layer. Therefore, the feature dimension of each modality is 6×128=768, and the final feature is 768×3=2304-dim. Stochastic gradient descent (SGD) with momentum of 0.9 and weight decay of 0.0005 is used to fine-tune the network. The initial learning rate is set to 0.001, and the learning rate is decayed by 10 times after training 20 and 40 epochs. The cross entropy loss and multimodal prototype loss are used in the training phase, and the hyperparameters of the final loss are Set to 1.

[0142] 1.2 Datasets and Evaluation Metrics

[0143] Experiments are conducted and evaluated on two public person re-identification datasets: 1) RGBNT201 dataset; 2) Market1501 dataset. These datasets and experimental evaluation indicators are summarized as follows.

[0144] 1) RGBNT201: It is collected on campus from four non-overlapping views, each consisting of three cameras, recording RGB, NI, and TI data simultaneously. It contains 4787 image triplets of 201 people, of which 141 identities are used for training, 30 identities are used for verification, and 30 identities are used for testing.

[0145] 2) Market1501: It includes 32,668 images of 1,501 people collected from six different cameras. The dataset is divided into a training set and a test set: the training set includes 12,936 images of 751 identities, and the test set includes 19,732 images of 750 people. Since Market1501 is an RGB single-modal dataset, it is necessary to use it to construct the multimodal Market1501. In order to simulate the night scene, the brightness of all images in the RGB mode is reduced by 60%. In addition, it is also necessary to generate images of the TI mode and the NI mode. The TI mode images can be generated from the RGB images with the help of cycleGAN, and the NI mode can be generated by converting the RGB images into grayscale images.

[0146] 3) Evaluation indicators: In the experiment, the mean average precision (mAP) and cumulative matching feature curve (CMC) are used to compare the performance of the proposed method with other methods. The similarity between two features is measured by Euclidean distance. Rank-n represents the top n closest samples of the query with the same ID from different cameras according to the distance metric.

[0147] 1.3 Comparison with State-of-the-Art Methods

[0148] The present invention has been compared with existing multimodal pedestrian re-identification methods on RGBNT201 and multimodal Market1501 datasets, including HAMNet, PFNet, and IEEE. In addition, a comparison of single-modal methods is also added, the significance of which is to verify the robustness of this method for multimodal pedestrian re-identification tasks. Single-modal methods include MLFN, HACNN, and OSNet. The specific operation of single-modality is to implement single-modal methods separately for three modal branches, extract the features of each modality, and then connect them in series as the final representation. The experimental results on these datasets are shown in Tables 1 and 2. It is not difficult to see from these comparison results that single-modal methods have not achieved good results. The reason is that single-modal methods cannot effectively mine the complementary information between different modalities and lack the ability to process heterogeneous multimodal information, which is the key problem to be solved in multimodal pedestrian re-identification. Among the multimodal person re-identification methods, HAMNet focuses on enhancing the prediction results of each branch, but ignores the importance of feature interaction between different modalities, while PFNet focuses too much on complementary information and ignores the importance of modality-specific information. IEEE combines the advantages of both, focusing on both complementary information and modality-specific information, and to a certain extent solves the heterogeneity problem between multimodal data. This method introduces the ability of multi-scale multimodal enhancement to interact with information, and adopts a low-rank multimodal fusion strategy to achieve the effect of improving model performance without increasing the number of model parameters. As can be seen from Tables 1 and 2, the performance of this method is better than all the compared methods. On the RGBNT201 dataset, compared with the current most advanced algorithm IEEE, this method's mAP increased by 5.86%, Rank-1 increased by 6.22%, Rank-5 increased by 6.1%, and Rank-10 increased by 8.98%. On the multimodal Market1501 dataset, compared with the current most advanced algorithm IEEE, this method's mAP increased by 1.44%, Rank-1 increased by 0.65%, Rank-5 increased by 1.27%, and Rank-10 increased by 1.06%.

[0149] 1.4 Comparison with Cross-modal Re-ID Methods

[0150] In order to verify the necessity of multimodal person re-identification compared with cross-modal person re-identification, the multimodal dataset RGBNT201 is constructed into two cross-modal datasets, retaining RGB, TI images and retaining RGB, NI photos respectively. In order to verify the effectiveness of this model, the model is compared with three state-of-the-art cross-modal person re-identification methods, including MMP loss, DDAG and MPANet, and compared with the state-of-the-art multimodal person re-identification method IEEE on RGBNT201, as shown in Table 3. It can be seen from Table 3 that the multimodal person re-identification method achieves better performance than the cross-modal person re-identification method by utilizing complementary multimodal information, overcoming the heterogeneity problem that the cross-modal person re-identification method fails to solve, which verifies the effectiveness of the multimodal person re-identification method.

[0151] Table 1 Comparison with existing unimodal and multimodal algorithms on the RGBNT201 dataset (in %)

[0152]

[0153] Table 2 Comparison with existing unimodal and multimodal algorithms on the Market1501 (multimodal version) dataset (in %)

[0154]

[0155] 1.5 Ablation Experiment

[0156] The model of the present invention consists of three key modules: multi-scale multimodal interaction module (MMIM), low-rank multimodal fusion module (LMFM) and multimodal prototype loss (MMP Loss). In order to verify the contribution of each module in the model, ablation experiments were performed on RGBNT201 by gradually introducing each module, as shown in Table 4. The improvement in performance from Table 4 (a) to (b) shows the effectiveness of the multi-scale multimodal interaction module (MMIM), which shows that MMIM can exchange meaningful multimodal complementary information. Compared with Table 4 (b), the significant improvement in Table 4 (c) proves the effectiveness of the low-rank modal fusion module (LMFM) in improving the model. Finally, the multimodal prototype loss (MMP Loss) further enhances the results, as shown in the results of Table 4 (d), which verifies the necessity of introducing multimodal prototype loss.

[0157] 1.6 Evaluation of Multi-scale Multi-modal Interaction Module

[0158] The multi-scale multimodal information interaction module (MMIM) aims to help the current modality's multiple scale features absorb the information of other modalities respectively. In order to evaluate the effectiveness of our proposed multi-scale multimodal interaction module, we compare other fusion methods with our module. These include image summation, feature summation JL-DCF, feature aggregation DAPNet, progressive fusion PFNet and cross-modal interaction IEEE method. The experimental results are shown in Table 5. As can be seen from Table 5, the image summation method has the worst effect. This is because it only fuses shallow information at the image level, and image-level fusion cannot remove the interference of noise, and noise from multiple modalities will be aggregated. Compared with the image summation method, feature-level fusion methods such as feature summation, feature aggregation and progressive fusion can more effectively fuse more modal feature information. Because it is a multimodal task, the information interaction between different modalities in the three-branch structure can greatly improve the recognition performance of the model, which is better than the traditional fusion method. Compared with the cross-modal interaction module, the multi-scale multimodal interaction module we proposed, because it introduces multiple scales, can exchange useful information between each modality more deeply, thus achieving the current optimal result.

[0159] The number of multi-scale layers used will affect the performance of the network structure to some extent. To further verify the superiority of the multi-scale multimodal interaction module (MMIM) network structure, this paper verifies the recognition performance of the proposed method under different multi-scale layer combinations, as shown in Table 6. When the network multi-scale layer is a combination of Layer2, 3, and 4, the mAP can reach 52.28%, Rank-1 reaches 53.35%, Rank-5 reaches 64.59%, and Rank-10 reaches 73.21%. Compared with the combination of Layer3 and 4, the performance is significantly improved. However, when the multi-scale layer is further increased to reach the combination of Layer1, 2, 3, and 4, the performance improvement is not obvious, and the complexity of the network structure will increase, and the computational cost will increase. Therefore, this paper adopts a network structure with a multi-scale Layer2, 3, and 4 combination.

[0160] Table 3 Comparison with the most advanced cross-modal person re-identification methods on two cross-modal datasets reconstructed from RGBNT201 (in %)

[0161]

[0162] Table 4 Ablation experiments on the RGBNT201 dataset. In the table, a refers to the three-branch feature extraction network supervised by cross entropy loss (in %)

[0163]

[0164] Table 5 Comparison of the multi-scale multimodal interaction module (MMIM) of the present invention with different fusion schemes (in %)

[0165]

[0166] Table 6 Performance comparison of different scale layers on the RGBNT201 dataset (in %)

[0167]

[0168] 1.7 Evaluation of Multimodal Mixed Loss

[0169] In order to simultaneously achieve the goal of supervising the network to learn specific feature representations to expand inter-class differences and constrain the intra-class center distance to improve the intra-class cross-modal similarity, this network adopts the joint supervision of CE loss and MMP loss. is a hyperparameter used to balance the two loss functions, which is considered as the weight of the MMP loss in the total loss. Figure 7 Shows different hyperparameters The performance distribution of this network on the RGBNT201 dataset is shown below, from which we can intuitively observe the impact of MMP loss during training. As the MMP loss increases, the weight of the total loss increases, and the network performance also increases. When it is close to 1, we can see that the improvement of network performance gradually stabilizes or even stops. When it is equal to 1, the joint supervision of CE loss and MMP loss can enable the network to achieve the best performance.

[0170] 1.8 Visual Analysis

[0171] 1) Retrieval results: From a qualitative perspective, a visualization of the top 10 ranking results for a given query pedestrian image is also provided. Figure 7 As shown in Figure 1, this method is very robust, and the deep multimodal features can robustly represent the discriminative information of the identity of these captured pedestrians regardless of their poses. Images with green borders belong to the same identity as the given query image, and images with red borders do not match the query image. In general, LRMM can effectively improve the ranking results, where the best matching image is more accurate than the baseline image.

[0172] 2) Heatmap: We also conduct heatmap experiments on LRMM using the Grad-CAM method to provide an intuitive explanation. Fig. 9Figure 2 shows the heatmap of our LRMM on the labeled Market1501 dataset, where the highlighted regions play an important role in identifying the target identity. The first column of each row is the probe image, and its heatmaps predicted by the global feature and six local features are shown in Figure 8 From these figures, we can roughly observe that global features capture the entire image, while local features focus on a certain area of ​​the image to identify pedestrians. They have obvious differences in the key areas for identifying pedestrians, which synergistically improves the ability of LRMM to learn distinguishable features.

[0173] 1.9. Comparative Experiment on Pedestrian Re-ID Effect

[0174] To further demonstrate the effectiveness of the proposed LRMM in person re-identification, we conducted a comparative experiment on the person re-identification effect of LRMM and the Baseline network on the PRW dataset. Fig. 9 As shown in Figure 2. Green is the correct recognition, red is the wrong recognition, and the blue dashed line is the missing recognition. Through the powerful multimodal information interaction, our proposed LRMM has the ability to aggregate multi-modal features, so it is more robust to similar pedestrians, pedestrian occlusions, and blurred small pedestrians.

[0175] 2. Results

[0176] In this paper, we introduce a new RGB-NI-TI based multimodal person re-identification method. To achieve comprehensive multimodal interaction, we propose a multi-scale multimodal interaction module to absorb complementary information from other modalities at the same scale from multiple scales. We further propose a low-rank multimodal fusion module to perform multimodal fusion using modality-specific low-order factors. In addition, in order to expand inter-class differences and improve intra-class cross-modal similarities, we introduce a multimodal prototype loss on the basis of the cross-entropy loss to achieve joint supervision of the two losses. The significant performance gain on the multimodal person re-identification dataset RGBNT201 and two constructed person re-identification datasets demonstrates the superiority and effectiveness of our proposed method. We believe that multimodal person re-identification methods will be the trend of the future, and in future work, we will study more robust multimodal models to cope with more complex and diverse multimodal person re-identification scenarios.

[0177] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A low-rank multi-scale multi-modal person re-identification method, characterized in that: The following steps are involved: Step 1: Obtain pedestrian images containing three modes: RGB, near infrared, and thermal infrared, and input the pedestrian images of the three modes into the three-branch feature extraction network to obtain the RGB modal features extracted from the RGB camera respectively. , NI modal features extracted from near infrared And TI modal features extracted from thermal infrared ; Step 2: Use a low-rank multi-scale multi-modal network model to transform the RGB modal features , NI Modal Characteristics and TI modal characteristics Calculate and obtain multimodal features , including the following steps: The low-rank multi-scale multi-modal network model includes a multi-scale multi-modal interaction module, a feature embedding module and a low-rank multi-scale multi-modal fusion module; RGB modality features , NI Modal Characteristics and TI modal characteristics The multi-scale multi-modal interaction modules are used to perform feature interaction and obtain RGB multi-scale interaction features. , NI multi-scale interactive features and TI multi-scale interactive features ; RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features The RGB multi-scale interactive features are respectively embedded in the feature embedding module , NI multi-scale interactive features and TI multi-scale interactive features The global feature context information is embedded into the corresponding local features to obtain the TI modality embedding enhanced features. , RGB modality embedding enhanced features and NI Modality Embedding Enhancements ; TI modality embedding enhancements , RGB modality embedding enhanced features and NI Modality Embedding Enhancements The multimodal features are obtained by mutual fusion of low-rank multi-scale multi-modal fusion modules. ; Step 3: Multimodal features Perform pedestrian re-identification on the pedestrian to obtain a pedestrian re-identification result of the pedestrian.

2. The low-rank multi-scale multi-modal person re-identification method according to claim 1, characterized in that: Obtaining multimodal features The specific process is as follows: S2.

1. Based on the multi-scale multi-modal interaction module, the RGB modal features are respectively , NI Modal Characteristics and TI modal characteristics Perform summation, enhancement and interaction processing in sequence to obtain RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features ; S2.2, RGB multi-scale interactive features , NI multi-scale interactive features and TI multi-scale interactive features Perform adaptive average pooling to obtain RGB multi-scale interactive features The global characteristics and Local features, NI multi-scale interactive features The global characteristics and Local features and TI multi-scale interactive features The global characteristics and local features; The relation-based embedding module transforms the RGB multi-scale interaction features into The global feature context information is embedded into each local feature, and the NI multi-scale interactive features are The global feature context information is embedded into each local feature and the TI multi-scale interactive feature The global feature context information is embedded into each local feature to obtain the TI modality embedding enhanced features. , RGB modality embedding enhanced features and NI Modality Embedding Enhancements ; S2.

3. Enhanced features for TI modality embedding , RGB modality embedding enhanced features and NI Modality Embedding Enhancements Fusion is performed to obtain fusion features .

3. The low-rank multi-scale multi-modal person re-identification method according to claim 2, characterized in that: TI multi-scale interaction features The specific process of obtaining is as follows: ①, extract the RGB modal features of the last scale and the NI modal characteristics of the latter scale Perform pixel-wise summation to obtain interaction features ; ② In the interaction characteristics and TI modal characteristics All of the above are convolved in a 1×1 convolutional layer to obtain the interaction features after convolution. And the TI modal features after convolution ; ③Introducing channel attention to interaction features To obtain enhanced features ; ④、Convolve the TI modal features Add to Enhancement Features To obtain the TI modal interaction features of the last scale ; ⑤. Using the same method as steps ① to ④, obtain the TI modal interaction characteristics of the second-to-last scale in sequence And the TI modal interaction characteristics of the third-to-last scale ; ⑥、The TI modal interaction features of the last scale , the penultimate scale TI modal interaction features And the TI modal interaction characteristics of the third-to-last scale Directly downsample to the same size through quadratic linear interpolation and concatenate to obtain multi-scale interaction features of TI modality ; ⑦ Multi-scale interaction features in TI mode Implement a 1×1 convolution layer on top to obtain multi-scale interactive features after convolution , that is, TI multi-scale interaction feature .

4. The low-rank multi-scale multi-modal person re-identification method according to claim 3, characterized in that: RGB multi-scale interactive features , NI multi-scale interactive features The acquisition process is specifically related to the TI multi-scale interaction feature The acquisition process is the same.

5. The low-rank multi-scale multi-modal person re-identification method according to claim 2, characterized in that: TI modality embedding enhancements The specific process of obtaining is as follows: ① In the global features and Local Features Implement three 1×1 convolutional layers to obtain the global features after convolution and Local Features ; ② Global features after convolution and Local Features Perform a dot product operation and then a normalized exponential function operation to obtain their similarity as a measure of the global features after convolution. and Local Features The weight of the relationship ; ③, through the global features after convolution Assign weights and then Global Features Sum and get the sum feature ; ④. For summation characteristics and local features Perform sum operation to obtain local enhanced features ; ⑤. Use global features For each local feature Enhance and obtain all local enhancement features , all local enhancement features Splicing is performed to obtain the TI modality embedding enhanced features that embed global information ; in, .

6. The low-rank multi-scale multi-modal person re-identification method according to claim 5, characterized in that: RGB modality embedding enhanced features and NI Modality Embedding Enhancements The acquisition method and TI modality embedding enhancement feature The method of obtaining is the same.

7. The low-rank multi-scale multi-modal person re-identification method according to any one of claims 1 to 6, characterized in that: A multimodal mixed loss function module is also provided in the low-rank multi-scale multimodal network model, and the multimodal mixed loss function module is used to verify the predicted classification results.

8. The low-rank multi-scale multi-modal person re-identification method according to claim 7, characterized in that: The multimodal mixed loss function module is used to reduce the total loss of the low-rank multi-scale multimodal network model in the training phase. The calculation process is as follows: The cross entropy loss is used to classify the identity information of pedestrian images. Suppose the cross entropy loss of each pedestrian image is Calculated as: ； in, is the number of local features, is the total number of modes, For the In the modal The true value labels of local features, For the In the modal The predicted identity label of local features; The multimodal prototype loss is used to limit the distance between the distributions of different modal features of each person. The centers of any two groups of modal feature distributions are taken and the penalty center distance is set as for: ； ； in, For the first group The center of the distribution of any two modal features of an individual, For the second group The center of the distribution of any two modal features of an individual, is the total number of pedestrian categories, For the first group The number of images of any two modalities in an individual, For the second group The number of images of any two modalities in an individual, For the first group Personal Zhang modality image features, For the second group Personal Zhang modality image features, For the Sequential symbols of images in individuals; The total loss of the low-rank multi-scale multi-modal network model during the training phase is for: ； ； in, is a hyperparameter used to balance the contribution of the two loss functions, and during the entire training phase, set .

9. An electronic device, characterized in that: include: Memory, used to store programs; A processor, configured to implement the low-rank multi-scale multi-modal pedestrian re-identification method as described in any one of claims 1 to 8 by executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The invention comprises a program which can be executed by a processor to implement the low-rank multi-scale multi-modal pedestrian re-identification method as described in any one of claims 1 to 8.

Images