A cross-modal pedestrian re-identification method and system based on wavelet transform

Abstract

The present invention discloses a cross-modal pedestrian re-identification method and system based on wavelet transform, which relates to the field of pedestrian re-identification technology, including: constructing a two-stream ResNet-50 backbone network; fusing the shallow features of the two-stream ResNet-50 backbone network through the ICB module to obtain fused features; extracting the fused features through the WEB branch module to obtain the modal shared features of the high and low frequency parts; optimizing the modal shared features output by the WEB branch module and the fused features output by the ICB module through the double-branch center guided loss to obtain the final recognition result. The present invention uses an information compensation module to combine the shallow features of the wavelet transform aggregation network at different stages, compensates for the loss of valuable information in the network feature extraction process, and improves the quality of the final feature vector; uses a wavelet enhancement module and a double-branch center guided loss to guide the network to mine the modal invariant information in the wavelet subgraph, thereby improving the model performance.

Description

Technical Field

[0001] The present invention relates to the technical field of pedestrian re-identification, and more particularly to a cross-modal pedestrian re-identification method and system based on wavelet transform. Background Art

[0002] Person re-identification (ReID) aims to retrieve a specific person from a given set of images across multiple disjoint cameras, and plays an increasingly important role in the field of intelligent video surveillance. Existing ReID methods mainly focus on information retrieval between RGB images. However, with the improvement of people's public safety awareness and the popularity of infrared cameras, more and more surveillance occasions are carried out at night or in low-light conditions. Models designed for visible modalities often face the problem of low accuracy at night or in low-light environments. In this regard, recent studies have paid attention to an important but challenging ReID subtask, namely visible-infrared ReID (VI-ReID), which aims to match the same person across modalities (visible to infrared, infrared to visible).

[0003] The main challenge of VI-ReID is how to extract modality-independent identity feature information of people to resist the huge modality difference between visible light and infrared modalities. Based on the current work, two mainstream methods are usually used to narrow the difference between modalities. One is the image-level method, which converts two different modality images into the same modality to maintain style consistency. However, the introduction of additional noise in the image conversion process will affect the extraction of discriminative features. The other is the feature-level method, which uses neural networks to extract identity features from different modalities and map them to a shared space for alignment. These methods can effectively handle the differences between modalities, but they still cannot avoid losing valuable information in the feature extraction process, which also contains some modality-independent features that can be mined and utilized.

[0004] In recent years, wavelet transform feature learning has shown strong advantages in different tasks such as image classification, medical image segmentation and image denoising. Wavelet transform can decompose feature maps into high-frequency and low-frequency parts. The low-frequency part has less noise and details, while the high-frequency part has clearer object boundary information, such as contours. Wavelet transform of feature maps is conducive to the model to mine shared features between modalities from different angles. At the same time, wavelet transform (DWT) and inverse wavelet transform (IDWT) of feature maps are lossless, and no additional information is lost. Traditional feature extraction-based methods often cannot fully mine the invariant information in the modality and are easily disturbed by noise features. Wavelet transform is an effective time-frequency analysis method. Considering the reversibility of wavelet transform and its ability to retain all information, wavelet transform is used to improve performance in various visual tasks. Although wavelet transform has been used in fields such as image classification, medical image segmentation and image denoising, no related research has been seen in the field of VI-Reid so far.

[0005] Therefore, how to propose a cross-modal pedestrian re-identification method and system based on wavelet transform, combine wavelet transform with attention mechanism, and use wavelet transform to decompose feature maps, mine modality-invariant features in the decomposed low-frequency and high-frequency images, reduce modality differences, and realize VI-ReID is a problem that technical personnel in this field urgently need to solve. Summary of the invention

[0006] In view of this, the present invention provides a cross-modal pedestrian re-identification method and system based on wavelet transform, which combines wavelet transform with attention mechanism, decomposes feature maps by wavelet transform, mines modality-invariant features in the decomposed low-frequency and high-frequency images, reduces modality differences, and realizes VI-ReID. In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A cross-modal person re-identification method based on wavelet transform, comprising:

[0008] Collect the images to be tested for VI-ReID and use them to build a two-stream ResNet-50 backbone network;

[0009] The shallow features of the two-stream ResNet-50 backbone network are fused through the ICB module to obtain the fused features;

[0010] Extracting the fused features through the WEB branch module to obtain the modal shared features of the high and low frequency parts;

[0011] The dual-branch center-guided loss is used to optimize the modal shared features output by the WEB branch module and the fusion features output by the ICB module to obtain the final recognition result.

[0012] Perform visual verification of the final recognition results.

[0013] Optionally, the ICB module is used to fuse shallow features of a two-stream ResNet-50 backbone network. The ICB module integrates self-attention learning with wavelet transform, uses wavelet transform for reversible downsampling, and reduces the spatial dimension of deep features; performs self-attention learning with shallow features, and concatenates the results with deep features that have undergone inverse wavelet transform to obtain fused features.

[0014] Optionally, the wavelet transform includes:

[0015] ;

[0016] The original image is decomposed into LF component, horizontal HF component, vertical HF component and diagonal HF component by wavelet transform, and LL, HL, LH and HH are obtained, where LL represents the low-frequency part, LH, HL, HH represent the high-frequency part, and the decomposed four components are defined as L and H parts.

[0017] Optionally, the step of splicing the result with the deep features after inverse wavelet transformation to obtain fused features includes: adding 4 ICB modules to the backbone network Resnet, the first three ICB modules perform feature fusion before and after each ResnetBlock, and the fourth ICB module performs feature fusion on the outputs of the first and third ICB modules.

[0018] Optionally, the ICB module structure includes:

[0019] The features before and after each ResnetBlock are used as shallow features and deep features respectively, and used together as the input of the ICB module. Convolution and regularization are applied to the input, and it is mapped to the feature space of the same dimension. The number of channels of the deep features is reduced to obtain the reduced features.

[0020] The reduced features are decomposed into n wavelet subbands using Haar discrete wavelet transform;

[0021] Concat n wavelet subbands in the channel dimension to obtain the output of the Haar discrete wavelet transform;

[0022] Convolve the output of the Haar discrete wavelet transform with shallow features to capture the features of the local area and calculate the attention output;

[0023] Design a DWT-Conv-IDWT structure, concatenate the attention output with the output of the DWT-Conv-IDWT structure, and obtain the output of the wavelet attention block;

[0024] The output of the wavelet attention block is convolved to restore the number of channels and added to the deep features of the initial input to obtain the final output.

[0025] Optionally, extracting the fused features through the WEB branch module to obtain the modal shared features of the high and low frequency parts includes:

[0026] Perform harr discrete wavelet transform on the output features after Resnetblock, decompose them into n wavelet subbands, and concat these n wavelet subbands in the channel dimension to obtain the extended result;

[0027] The expanded result is input into the BottleNeck convolution block for subtle modality consistent feature learning. Finally, the inverse wavelet transform is applied to the output of the BottleNeck block structure to restore the output feature map dimension to its original size.

[0028] Optionally, it also includes: splicing the output features after the original Resnetblock and the output features of the WEB branch module together and inputting them into the last Resnetblock to obtain n groups of features generated by the double branches.

[0029] Optionally, the method optimizes the modal sharing features output by the WEB branch module and the fusion features output by the ICB module through the dual-branch center guided loss to obtain the final recognition result, including:

[0030] Calculate the centers of n groups of features generated by the two branches respectively;

[0031] Set the criterion of the dual-branch center-guided loss, and define the distance loss of the centers of n groups of features respectively by measuring the similarity of features through Euclidean distance;

[0032] The dual-branch center-guided loss is obtained by taking the average of the distance losses of the centers of the defined n groups of features.

[0033] Optionally, the total loss is calculated as:

[0034] ,in, is the coefficient of control loss ratio, For identity loss, is the triple entropy loss, which minimizes three losses , and jointly optimize the network.

[0035] Optionally, a cross-modal person re-identification system based on wavelet transform comprises:

[0036] Backbone network construction module: used to collect images to be tested for VI-ReID and build a two-stream ResNet-50 backbone network based on them;

[0037] Feature fusion module: used to fuse the shallow features of the two-stream ResNet-50 backbone network through the ICB module to obtain fused features;

[0038] Feature extraction module: used to extract the fusion features through the WEB branch module to obtain the modal shared features of the high and low frequency parts;

[0039] Optimization module: used to optimize the modal shared features output by the WEB branch module and the fusion features output by the ICB module through the dual-branch center guided loss to obtain the final recognition result;

[0040] Verification module: used to visually verify the final recognition results.

[0041] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a cross-modal pedestrian re-identification method and system based on wavelet transform, which has the following beneficial effects:

[0042] The present invention proposes a cross-modal pedestrian re-identification method based on wavelet transform, comprising: collecting a test image of VI-ReID and constructing a two-stream ResNet-50 backbone network based on it; fusing shallow features of the two-stream ResNet-50 backbone network through an ICB module to obtain fused features; extracting features of the fused features through a WEB branch module to obtain modal shared features of high and low frequency parts; optimizing the modal shared features output by the WEB branch module and the fused features output by the ICB module through a dual-branch center guided loss to obtain a final recognition result; and visually verifying the final recognition result. This paper proposes a new feature mining method based on wavelet transform, which aims to apply it to the cross-modal pedestrian re-identification task by mining the potential information in wavelet subgraphs; an information compensation module is proposed, which combines the shallow features of the network with wavelet transform aggregation at different stages to compensate for the loss of valuable information in the network feature extraction process, which also improves the quality of the final feature vector; a wavelet enhancement module is proposed, and a dual-branch center-guided loss is used to guide the network to fully mine the modality-invariant information in the wavelet subgraphs to improve the performance of the model; extensive experiments on three challenging VI-ReID datasets show that the proposed method outperforms other state-of-the-art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0044] Figure 1 A schematic flow chart of a cross-modal pedestrian re-identification method based on wavelet transform provided by the present invention.

[0045] Figure 2 This is a framework diagram of the wavelet transform feature mining network WFM provided by the present invention.

[0046] FIG3( a ) is a schematic diagram of applying DWT and IDWT to a two-dimensional image provided by the present invention.

[0047] FIG3( b ) is a schematic diagram of a specific DWT process provided by the present invention.

[0048] FIG3( c ) is a schematic diagram of the receptive field of the DWT-Conv-IDWT provided by the present invention.

[0049] Figure 4 This is a structural framework diagram of the ICB module provided by the present invention.

[0050] FIG5( a ) is a schematic diagram of a WEB module provided by the present invention.

[0051] FIG5( b ) is a schematic diagram of the center of calculating four groups of features provided by the present invention.

[0052] FIG. 5( c ) is a diagram of the present invention. Schematic diagram.

[0053] Figure 6 (a) shows the hyperparameters on the SYSU-MM01 dataset provided by the present invention Schematic diagram of the impact on model performance.

[0054] Figure 6 (b) shows the hyperparameters on the SYSU-MM01 dataset provided by the present invention Schematic diagram of the impact on model performance.

[0055] FIG7( a ) is a graph showing the rank-10 retrieval results of the baseline provided by the present invention on the SYSU-MM01 dataset.

[0056] FIG7( b ) is a graph showing the rank-10 retrieval results of the WFM provided by the present invention on the SYSU-MM01 dataset.

[0057] Figure 8 This is a thermodynamic visualization diagram of each module provided by the present invention.

[0058] Fig. 9 Intra-class and inter-class distance diagrams of the cross-modal features provided by the present invention.

[0059] Fig.10 This is a distribution diagram of the features provided by the present invention embedded in a two-dimensional feature space. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0061] The embodiment of the present invention discloses a cross-modal pedestrian re-identification method based on wavelet transform, such as Figure 1 As shown, including:

[0062] Collect the images to be tested for VI-ReID and use them to build a two-stream ResNet-50 backbone network;

[0063] The shallow features of the two-stream ResNet-50 backbone network are fused through the ICB module to obtain the fused features;

[0064] Extracting the fused features through the WEB branch module to obtain the modal shared features of the high and low frequency parts;

[0065] The dual-branch center-guided loss is used to optimize the modal shared features output by the WEB branch module and the fusion features output by the ICB module to obtain the final recognition result.

[0066] Perform visual verification of the final recognition results.

[0067] In a specific implementation, a cross-modal person re-identification method based on wavelet transform is as follows: Figure 2 As shown in the figure, wavelet transform is applied to the cross-modal pedestrian re-identification task, and a wavelet transform feature mining network (WFM) is proposed, which can learn discriminative information from the feature map of wavelet decomposition and mine various subtle modal shared features contained therein, including:

[0068] Information compensation module (ICB), wavelet transform enhancement module (WEB) and dual-branch center guided loss. The ICB module compensates for the loss of valuable information in the network feature extraction process by fusing the current features with the features of the previous stage. The WEB module uses a new branch to enable the network to further mine the modal invariant features in the WEB module. In order to guide the module to mine key information that is different from the other branch, a dual-branch guided loss is proposed to improve the model's feature learning ability. The information compensation module (ICB) and wavelet transform enhancement module (WEB) are used to reduce the modal differences between visible and infrared images in the frequency domain. The dual-branch center guided loss L dbc, which is used to guide the WEB module to discover modal shared features on the feature map reconstructed by wavelet transform. The visualization results show that the model pays more attention to modal invariant features and can reduce the impact of interference information. A large number of experiments show that the proposed WFM achieves excellent performance on three challenging cross-modal datasets.

[0069] Specifically, the wavelet transform-based feature mining network WFM adopts a two-stream ResNet-50 network as the backbone. First, an ICB module is designed to fuse the shallow features of the network. The ICB module integrates self-attention learning with wavelet transform, and uses wavelet transform for reversible downsampling. It reduces the spatial dimension of deep features without losing information, and performs self-attention learning with shallow features. The results are concatenated with deep features that have undergone inverse wavelet transform to obtain fused features. In addition, a WEB branch module is designed, which performs feature extraction on the decomposed wavelet subgraph to mine the modal shared features of the high and low frequency parts and reduce modal differences. Finally, a dual-branch center-guided loss L is proposed. dbc To guide the model to learn additional modality-shared features from the WEB branch to reduce intra-class and inter-class differences.

[0070] In a specific implementation, the wavelet transform specifically includes:

[0071] A two-dimensional image is essentially a two-dimensional discrete non-stationary signal containing different frequency ranges and spatial position information. Wavelet transform can effectively retain this information while decomposing it. As shown in Figure 3(a), the original image is decomposed into LF, horizontal HF, vertical HF, and diagonal HF components (i.e., LL, HL, LH, and HH, where LL represents the low-frequency part and LH, HL, and HH represent the high-frequency part) using wavelet transform. The four decomposed components are defined as L and H:

[0072] ;

[0073] The L part has less noise and details. In contrast, the H part has more noise, but the object boundaries are clearer, which helps the model pay more attention to details. The lossless decomposition of wavelet transform helps the model extract modal sharing features on different components and improve model performance.

[0074] In the specific implementation, as the network depth increases, the extracted features become more and more refined, and some shallow spatial information will be discarded. Reasonable fusion of shallow features containing rich spatial information with deep features can effectively improve the performance of the model. In this regard, an information compensation module ICB based on wavelet transform is designed to reuse the spatial information lost in the shallow layer of the network, such as Figure 4 As shown, DWT and IDWT represent wavelet transform and inverse wavelet transform respectively, and the ICB module specifically includes:

[0075] 1) Take the features before and after each residual block as shallow features and deep features , together as the input of the ICB module, the input and Apply convolution and regularization to map them to a feature space of the same dimension ;

[0076] 2) Further The number of channels is reduced to 1 / 4 of the original, denoted as X;

[0077] 3) Input The Haar discrete wavelet transform (DWT) is used to decompose it into four wavelet subbands. The DWT decomposition process is shown in Figure 3(b). The input feature Figure X Apply a low pass filter along the line

[0078] and high pass filter Encode X into two subbands and Next, along the learned subband and The columns use the same low-pass filter and high pass filter , thus obtaining all 4 wavelet subbands: . Refers to low-frequency components that reflect the underlying object structure at a coarse-grained level. , and Represents the high-frequency components that preserve the texture details of the object at a fine-grained level. Finally, the four wavelet subbands are concatenated in the channel dimension to obtain the output of the DWT .

[0079] 4) Output of DWT and shallow features Convolution is performed to capture the features of local areas and enhance the model's ability to perceive local information in feature maps:

[0080] ;

[0081] ;

[0082] ;

[0083] Among them, G represents a 3×3 convolution, , , is a 1×1 convolution. , , Q, K, V as attention mechanism.

[0084] 5) Yes , After matrix multiplication, the softmax activation function is performed to obtain the attention matrix A:

[0085] ;

[0086] The attention matrix A is combined with Multiply them together to get the attention output.

[0087] 6) Design the DWT-Conv-IDWT structure, as shown in Figure 3(c). Compared with ordinary convolution, this structure has a larger receptive field, thereby increasing the network's local context perception ability. The attention output is concatenated with the output of DWT-Conv-IDWT to obtain the output of the wavelet attention block:

[0088] ;

[0089] 7) Output of wavelet attention block A 1×1 convolution is performed to restore the number of channels and added to the deep features of the initial input to obtain the final output.

[0090] ;

[0091] This embodiment adds 4 ICB modules to the backbone network Resnet. The first three perform feature fusion on each ResnetBlock. The fourth performs feature fusion on the output of the first and third ICB modules. The purpose is to make full use of the wavelet transform to enhance the local perception ability to fuse the important spatial information of the shallow network, so that the network can better focus on the modal sharing features.

[0092] In a specific implementation, the wavelet enhancement module specifically includes:

[0093] As shown in Figure 5(a), after the third residual block of Resnet, a new branch is added to enable the model to mine modality-invariant features from the feature map after wavelet transformation to improve model performance. Specifically, the output features after the third block Perform the Harr discrete wavelet transform (DWT), decompose it into 4 wavelet subbands, and splice these 4 wavelet subbands in the channel dimension, expressed as:

[0094] ;

[0095] in, These are the four wavelet subbands obtained after DWT decomposition, and Concat represents the concatenation operation on the channel.

[0096] Will Input to BottleNeck convolutional block for subtle modality consistent feature learning;

[0097] Applying inverse wavelet transform to the output of BottleNeck convolutional block structure restores the output feature map dimension to its original size. This process is described as:

[0098] ;

[0099] The output features of the original block3 and the output features of the WEB module are spliced ​​together and input into the last block. Finally, four sets of features generated by the two branches can be obtained: .

[0100] In a specific implementation, the specific steps of determining the dual-branch central guidance loss are:

[0101] Although there are clearer details in the decomposed wavelet sub-image, it is difficult for the neural network to learn this information autonomously, which is contrary to the original intention of designing the WEB module. In this regard, the dual-branch center guided loss is used To enhance the mining of subtle modal shared features by WEB. As shown in Figure 5(b), for pedestrians with the same ID, we first calculate The centers of the 4 groups of features:

[0102] ;

[0103] Among them, K represents the number of pedestrians with the same ID in each group, c represents the feature center of all pedestrians with the same ID in each group, V and T represent visible light and infrared modalities respectively, and o and w represent the original features and the wavelet features after the WEB module respectively.

[0104] Designed by Three principles are followed: the visible light modal features and the infrared modal features are as similar as possible, aiming to encourage the network to focus on modal shared features and narrow modal differences; the visible light / infrared modal features generated by the WEB module are as different as possible from the original visible light / infrared modal features, aiming to encourage the two branches of the network to produce complementary features and enrich the feature representation of the network; different ID features are as different as possible, aiming to encourage the network to retain identity discrimination information.

[0105] The Euclidean distance is used to measure the similarity of features. Definition The distance loss is as follows:

[0106] ;

[0107] in, Represents the Euclidean distance between two features, + ensures that the loss value is non-negative. When the calculated value is less than zero, the loss is zero. Represents the interval to control the distance between positive and negative samples.

[0108] As shown in Figure 5(c), Bringing the original visible light modal features closer to the infrared modal features helps reduce the modal differences between VIS and IR images. Pushing the distance between the visible light modal features generated by the WEB module and the original visible modal features helps the model to mine different cross-modal shared features in the WEB module. , The two items ensure that the intra-class distance is smaller than the inter-class distance by pushing the distance between features with different IDs farther.

[0109] Similarly, The distance loss between them is as follows:

[0110] ;

[0111] is defined as:

[0112] .

[0113] In a specific implementation, it also includes calculating the total loss:

[0114] Combined identity loss and triple entropy loss , by minimizing these three losses The sum of , jointly optimize the network, the total loss is expressed as:

[0115] ;

[0116] in, It is the coefficient that controls the loss ratio.

[0117] In a specific implementation, a cross-modal person re-identification system based on wavelet transform includes:

[0118] Backbone network construction module: used to collect images to be tested for VI-ReID and build a two-stream ResNet-50 backbone network based on them;

[0119] Feature fusion module: used to fuse the shallow features of the two-stream ResNet-50 backbone network through the ICB module to obtain fused features;

[0120] Feature extraction module: used to extract the fusion features through the WEB branch module to obtain the modal shared features of the high and low frequency parts;

[0121] Optimization module: used to optimize the modal shared features output by the WEB branch module and the fusion features output by the ICB module through the dual-branch center guided loss to obtain the final recognition result;

[0122] Verification module: used to visually verify the final recognition results.

[0123] In a specific embodiment, the visual verification of the final recognition result specifically includes:

[0124] S1: Dataset and Validation Metrics

[0125] (1) Dataset

[0126] SYSU-MM01 is a large dataset that includes 491 pedestrians from 4 visible cameras and 2 infrared cameras. The training set contains 19659 visible images and 12792 infrared images of 395 people, and the test set contains 96 people. There are two test modes, full search mode and indoor search mode. For the full search mode, all images are used. For the indoor search mode, only indoor images from the first, second, third, and sixth cameras are used.

[0127] RegDB is a small dataset with images captured by one visible light camera and one infrared camera. It contains 8240 images of 412 identities, each with 10 visible images and 10 thermal images. The dataset is randomly divided into two parts, i.e., images of 206 identities are used for training and images of the remaining 206 identities are used for testing.

[0128] LLCM is captured by 9 visible light cameras and 9 infrared cameras in low-light environments, and contains 46,767 images of 1,064 identities. The training set contains 16,946 VIS images and 13,975 IR images of 713 identities, while the test set contains 8,680 VIS images and 7,166 IR images of 351 identities.

[0129] (2) Evaluation indicators

[0130] Cumulative matching features (CMC) and mean average precision (mAP) are used as the main evaluation indicators. CMC reflects the probability that the first n results in the retrieved image appear as correct results, and mAP reflects the average retrieval accuracy of all identities.

[0131] S2: Implementation details: The AGW dataset is used as a baseline and initialized with weights pre-trained on ImageNet. During the training phase, all input images are resized to 3×384×192 and data augmentation is performed by horizontal flipping and random erasing. For each mini-batch, 8 identities are randomly selected, of which 4 VIS images and 4 IR images are selected for each identity. The learning rate is updated from 0.01 to 0.1 in the first 10 training epochs using a warm-up strategy. At the 20th, 60th, and 120th training epochs, the learning rate is set to 0.01, 0.001, and 0.0001, respectively. Stochastic gradient descent SGD is used as the optimizer, and the momentum parameter is set to 0.9. The total number of training epochs is set to 150, and all experiments are implemented on a single NVIDIA RTX3090 GPU using PyTorch.

[0132] S3: Performance comparison of different models

[0133] The WFM model is compared with the most advanced VI-ReID methods on the public VI-ReID datasets SYSU-MM01, RegDB and LLCM datasets. The experimental results are shown in Tables 1, 2 and 3.

[0134] SYSU-MM01: As can be seen from Table 1, the proposed WFM outperforms other state-of-the-art methods, achieving 78.02% Rank-1 and 74.16% mAP in all search modes, and 87.68% Rank-1 and 87.62% mAP in indoor search mode. The proposed method has better results in terms of Rank-1 and mAP in both global and indoor search modes. In addition, compared with the methods [SPOT, CAJ, DEEN] that are also based on two-stream networks, the proposed method also has substantial performance advantages, with Rank-1 and mAP higher than DEEN by 3.32% and 2.36% in all search modes, and Rank-1 and mAP higher than DEEN by 7.38% and 4.32% in indoor search mode. The results also show that the method has better performance in indoor environments. Among them, SPOT is structure-aware positional transformer (SPOT) network; CAJ is Channel augmented joint (CAJ) network; DEEN is Diverse Embedding Expansion Network.

[0135] Table 1 Performance comparison of various models on the SYSU-MM01 dataset

[0136]

[0137] RegDB: From Table 2, we can see that the proposed WFM achieves the best performance among the state-of-the-art methods. The rank-1 and mAP from visible to infrared mode are 92.23% and 86.16%, respectively, and the mAP from infrared to visible mode are 91.94% and 85.70%, respectively. The results show that the proposed method has excellent performance in both visible to infrared and infrared to visible modes, with Rank-1 and mAP higher than DEEN by 1.13% and 1.06% in visible to infrared mode and 2.44% and 2.30% in infrared to visible mode, which shows that the proposed method is very robust to different datasets and query modes.

[0138] Table 2 Performance comparison of various models on the RegDB dataset

[0139]

[0140] LLCM: As can be seen from Table 3, the proposed WFM also achieves excellent performance on the LLCM dataset in low-light complex scenes. The Rank-1 and mAP from visible to infrared mode are 63.61% and 66.12%, respectively, which are 1.11% and 0.32% higher than DEEN, and the mAP from infrared to visible mode are 55.74% and 63.10%, respectively, which are 0.84% ​​and 0.2% higher than DEEN. This shows that the WFM model also shows strong robustness in low-light complex multimodal scenes.

[0141] Table 3 Performance comparison of various models on LLCM dataset

[0142]

[0143] S4: Ablation Experiments: Ablation experiments are conducted to evaluate the contribution of each component in the proposed WFM. For fair comparison, all experiments are conducted on SYSU-MM01, using both full search mode and indoor search mode, using AGW as the baseline for all experiments, and adopting the same data augmentation.

[0144] (1) Effectiveness of each module

[0145] Analyze the key components of WFM, including the Information Compensation Module (ICB), the Wavelet Enhancement Module (WEB), and the Dual-branch Center-Guided Loss . As shown in Table 4, each module improves the performance of the model. Among them, the ICB module fuses shallow features at different stages and reuses the shallow information lost as the network increases in depth. From the first row to the third row, Rank-1 and mAP in all search modes are improved by 5.59% and 3.65% respectively. It is worth noting that adding the WEB module to the model alone can slightly improve the performance, but the results are not ideal. From the first row to the second row, Rank-1 and mAP in all search modes are improved by 3.21% and 0.18% respectively. This is because the model will not actively mine the modal invariant information of the feature map reconstructed by wavelet transform, and a loss is needed to guide the network to mine modal shared features. After adding After that, the performance is further improved. In all search modes, from the second to the fourth row, Rank-1, mAP are increased by an additional 0.86% and 0.74% respectively; from the fifth to the sixth row, Rank-1, mAP are increased by an additional 1.53% and 2.13% respectively, indicating that It greatly guides the model's ability to mine modal shared features.

[0146] Table 4 The impact of each WFM module on model performance

[0147]

[0148] Table 5 Comparison with Non-Local (NL) module

[0149]

[0150] Among them, ICB* represents the ICB module without wavelet transform

[0151] (2) Comparison with Non-Local (NL) module

[0152] The information compensation module (ICB) based on wavelet transform is similar to the structure of the non-local module Non-Local, and its purpose is to fuse the features of different stages. ICB and Non-local were compared experimentally, as shown in Table 5. In all search modes, the Rank1 and mAP of the ICB module are 1.07% and 1.62% higher than those of the Non-Local module, respectively. In indoor search mode, the Rank1 and mAP of the ICB module are 1.65% and 0.96% higher than those of the Non-Local module, respectively. The results verify that the ICB module integrated with wavelet transform has better performance than the Non-Local structure. Comparing the third and fourth rows of Table 5, in all search modes, the wavelet transform improves the Rank-1 of the model by 1.52% and the mAP by 1.84%. It can be seen that the wavelet transform is indispensable in the ICB module and plays an important role.

[0153] (3) Impact of hyperparameters

[0154] Evaluated , The effects of these two hyperparameters on network performance are quantitatively compared, and the results are shown in Figure 6 (a) and Figure 6 (b). It can be seen that as the values ​​of λ and ρ increase, the performance of WFM gradually improves. However, when λ and ρ exceed a certain threshold, the performance of WFM decreases. For this reason, λ=0.8 and ρ=0.2 are set to achieve the best performance.

[0155] S5: Visualization

[0156] (1) Rank Visualization

[0157] To demonstrate the advantages of WFM, the full search test mode method is used on the SYSU-MM01 dataset to compare with the baseline. The 10 images with the highest matching scores are shown in Figure 7 (a) and Figure 7 (b). The WFM method effectively handles modal differences and correctly identifies images of the same person (checked below the image). In contrast, the baseline is insensitive to the attention to modal invariant features and is easily disturbed by non-critical information, such as the pocket-holding posture in the 4th row, resulting in a large number of mismatched person images (crossed below the image). In each retrieval case, the image in the first column is the infrared query, while the rest of the images are visible gallery images. The checked image represents that the image matches the query ID, and the crossed image represents that the image does not match the query ID.

[0158] (2) Heat map visualization

[0159] In order to explore the impact of each module on the network, the heat map of the network after adding each module is shown. The results are as follows Figure 8 As shown, (a) represents the original image. (b) represents the thermal visualization of each module. The square area represents noise information, and the elliptical area represents modal shared features. It can be seen that compared with the baseline, WFM can effectively enhance the network's extraction of modal shared features, represented by ellipses, such as facial information, texture information on clothes, shoe information, etc. At the same time, it reduces the impact of noise information, represented by boxes, such as table corners, blackboards, etc. Columns (c) and (d) reflect that adding WEB or ICB modules alone can also enhance the model's extraction of key features, illustrating the effectiveness of each module of WFM.

[0160] (3) Visualization of intra-class and inter-class distances

[0161] Fig. 9 (a)-(e) show the intra-class and inter-class distances of cross-modal features. The intra-class distance and inter-class distance are represented in dark gray and light gray, respectively. Fig.10(f)-(j) show the distribution of feature embedding in the 2D feature space, where circles and triangles represent visible and IR modalities. A total of 15 people were selected from the test set. Samples of the same color come from the same person. Circle and triangle markers represent images from VIS and IR modes, respectively.

[0162] In order to evaluate the effectiveness of the method, the frequency distribution of intra-class and inter-class distances between VIS and IR features on SYSU-MM01 is analyzed. The distance D between two features can be expressed as:

[0163] ;

[0164] in, Represents cosine similarity. Fig. 9 As shown in (a)-(e), each ID is regarded as a class, and 1000 classes are selected to be plotted as a histogram, where δ1, δ2, δ3, δ4, and δ5 represent the intra-class and inter-class peak distances of initial, baseline, WEB, ICB, and WFM, respectively. The intra-class distance reflects the degree of aggregation of pedestrians of the same class in the feature space (the smaller the value, the higher the degree of aggregation), the inter-class distance reflects the degree of separation of pedestrians of different classes in the feature space (the larger the value, the greater the degree of separation between classes), and the frequency reflects the number of features at this distance. It can be noted that δ1<δ2<δ3<δ4<δ5, which shows that each module strengthens the aggregation of similar features and effectively reduces the modal difference between the VIS and IR images of similar pedestrians, thereby achieving better performance.

[0165] (4) t-SNE visualization

[0166] In order to more intuitively show the distribution of different features in the two-dimensional feature space, the t-SNE tool is used to visualize the feature distribution on the SYSU-MM01 dataset. Fig.10 As shown in (f)-(j), 15 identities are randomly selected from the test set of the SYSU-MM01 dataset. Fig.10 Compared with the baseline feature distribution in (f), as the modules are added, the distance between features of different identities gradually increases, while the distance between features of the same identity gradually decreases. This shows that WFM effectively distinguishes and aggregates the features of the same type of pedestrians and reduces the VIS-IR modality difference.

[0167] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0168] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-modal person re-identification method based on wavelet transform, characterized in that: include: Collect the images to be tested for VI-ReID and use them to build a two-stream ResNet-50 backbone network; The shallow features of the two-stream ResNet-50 backbone network are fused through the ICB module to obtain the fused features; The ICB module is used to fuse the shallow features of the two-stream ResNet-50 backbone network. The ICB module integrates self-attention learning with wavelet transform, uses wavelet transform for reversible downsampling, and reduces the spatial dimension of deep features; performs self-attention learning with shallow features, and splices the results with deep features that have undergone inverse wavelet transform to obtain fused features; The step of concatenating the result with the deep features after inverse wavelet transformation to obtain fused features includes: adding 4 ICB modules to the backbone network Resnet, the first three ICB modules perform feature fusion for each ResnetBlock, and the fourth ICB module performs feature fusion for the outputs of the first and third ICB modules; The ICB module structure includes: The features before and after each ResnetBlock are used as shallow features and deep features respectively, and used together as the input of the ICB module. Convolution and regularization are applied to the input, and it is mapped to the feature space of the same dimension. The number of channels of the deep features is reduced to obtain the reduced features. The reduced features are decomposed into n wavelet subbands using Haar discrete wavelet transform; Concat n wavelet subbands in the channel dimension to obtain the output of the Haar discrete wavelet transform; Convolve the output of the Haar discrete wavelet transform with shallow features to capture the features of the local area and calculate the attention output; Design a DWT-Conv-IDWT structure, concatenate the attention output with the output of the DWT-Conv-IDWT structure, and obtain the output of the wavelet attention block; The output of the wavelet attention block is convolved to restore the number of channels and added to the deep features of the initial input to obtain the final output; Extracting the fused features through the WEB branch module to obtain the modal shared features of the high and low frequency parts; The dual-branch center-guided loss is used to optimize the modal shared features output by the WEB branch module and the fusion features output by the ICB module to obtain the final recognition result. Perform visual verification of the final recognition results.

2. According to the cross-modal pedestrian re-identification method based on wavelet transform according to claim 1, it is characterized in that: The wavelet transform comprises: ； The original image is decomposed into LF component, horizontal HF component, vertical HF component and diagonal HF component by wavelet transform, and LL, HL, LH and HH are obtained, where LL represents the low-frequency part, LH, HL, HH represent the high-frequency part, and the decomposed four components are defined as L and H parts.

3. The cross-modal person re-identification method based on wavelet transform according to claim 1, characterized in that: The method of extracting the fusion features through the WEB branch module to obtain the modal shared features of the high and low frequency parts includes: Perform harr discrete wavelet transform on the output features after Resnetblock, decompose them into n wavelet subbands, and concat these n wavelet subbands in the channel dimension to obtain the extended result; The expanded result is input into the BottleNeck convolution block for subtle modality consistent feature learning. Finally, the inverse wavelet transform is applied to the output of the BottleNeck block structure to restore the output feature map dimension to its original size.

4. The cross-modal person re-identification method based on wavelet transform according to claim 3 is characterized in that: Also includes: The output features after the original Resnetblock and the output features of the WEB branch module are concatenated and input into the last Resnetblock to obtain n sets of features generated by the dual branches.

5. The cross-modal person re-identification method based on wavelet transform according to claim 1, characterized in that: The dual-branch center guided loss is used to optimize the modal shared features output by the WEB branch module and the fusion features output by the ICB module, and the final recognition results include: Calculate the centers of n groups of features generated by the two branches respectively; Set the criterion of the dual-branch center-guided loss, and define the distance loss of the centers of n groups of features respectively by measuring the similarity of features through Euclidean distance; The dual-branch center-guided loss is obtained by taking the average of the distance losses of the centers of the defined n groups of features.

6. The cross-modal person re-identification method based on wavelet transform according to claim 5, characterized in that: It also includes calculating the total loss as: ,in, is the coefficient of control loss ratio, For identity loss, is the triple entropy loss, which minimizes three losses The sum of , jointly optimizes the network, is the dual-branch central guidance loss.

7. A cross-modal person re-identification system based on wavelet transform, characterized in that: include: Backbone network construction module: used to collect images to be tested for VI-ReID and build a two-stream ResNet-50 backbone network based on them; Feature fusion module: used to fuse the shallow features of the two-stream ResNet-50 backbone network through the ICB module to obtain fused features; The ICB module is used to fuse the shallow features of the two-stream ResNet-50 backbone network. The ICB module integrates self-attention learning with wavelet transform, uses wavelet transform for reversible downsampling, and reduces the spatial dimension of deep features; performs self-attention learning with shallow features, and splices the results with deep features that have undergone inverse wavelet transform to obtain fused features; The step of concatenating the result with the deep features after inverse wavelet transformation to obtain fused features includes: adding 4 ICB modules to the backbone network Resnet, the first three ICB modules perform feature fusion for each ResnetBlock, and the fourth ICB module performs feature fusion for the outputs of the first and third ICB modules; The ICB module structure includes: The features before and after each ResnetBlock are used as shallow features and deep features respectively, and used together as the input of the ICB module. Convolution and regularization are applied to the input, and it is mapped to the feature space of the same dimension. The number of channels of the deep features is reduced to obtain the reduced features. The reduced features are decomposed into n wavelet subbands using Haar discrete wavelet transform; Concat n wavelet subbands in the channel dimension to obtain the output of the Haar discrete wavelet transform; Convolve the output of the Haar discrete wavelet transform with shallow features to capture the features of the local area and calculate the attention output; Design a DWT-Conv-IDWT structure, concatenate the attention output with the output of the DWT-Conv-IDWT structure, and obtain the output of the wavelet attention block; The output of the wavelet attention block is convolved to restore the number of channels and added to the deep features of the initial input to obtain the final output; Feature extraction module: used to extract the fusion features through the WEB branch module to obtain the modal shared features of the high and low frequency parts; Optimization module: used to optimize the modal shared features output by the WEB branch module and the fusion features output by the ICB module through the dual-branch center guided loss to obtain the final recognition result; Verification module: used to visually verify the final recognition results.

Images