A cross-modal person re-identification method and system based on multi-frequency domain features

Abstract

The present application discloses a cross-modal pedestrian re-identification method and system based on multi-frequency domain features, which relates to the field of pedestrian re-identification. The method includes: obtaining a pedestrian image set; constructing a cross-modal pedestrian re-identification model; constructing a total loss function; training the cross-modal pedestrian re-identification model based on the pedestrian image training set and the total loss function; inputting the pedestrian image to be identified and the pedestrian image test set into the trained cross-modal pedestrian re-identification model to obtain image features; using the image features, calculating the cosine similarity between the pedestrian image to be identified and each image in the pedestrian image test set; sorting the images in the pedestrian image test set according to the cosine similarity; and obtaining the pedestrian image most similar to the pedestrian image to be identified based on the sorting result to complete cross-modal pedestrian re-identification. The present application can solve the problems of low computational efficiency, high resource consumption, and insufficient feature representation diversity and robustness in the prior art.

Description

Technical Field

[0001] The present application relates to the field of pedestrian re-identification, and in particular to a cross-modal pedestrian re-identification method and system based on multi-frequency domain features. Background Art

[0002] Cross-modality person re-identification (CM-ReID) is a key research direction in computer vision. It aims to identify the same person from images or videos of different modalities across multiple cameras and scenes. Its core task is to address the appearance differences of the same person caused by factors such as different modalities, camera perspective changes, occlusions, lighting variations, and pose changes.

[0003] Existing CM-ReID methods are based on frequency domain analysis, specifically techniques based on frequency decomposition (i.e., separating the signal into low-frequency and high-frequency components). Fourier analysis shows that the amplitude spectrum primarily encodes modality-dependent features (such as color and illumination), while the phase spectrum preserves structural information (such as outline and shape) that is invariant across modalities. Based on this discovery, a series of methods have emerged:

[0004] 1) Spectral Alignment Methods: Early methods such as FDMNet and FDNM focused on aligning amplitude spectra while preserving phase information to narrow modal distances. DSSF³, on the other hand, advocates for jointly learning both amplitude and phase components.

[0005] 2) Frequency decomposition methods: Researchers have recognized that high-frequency and low-frequency components play different roles in learning modal invariant features. Representative methods include:

[0006] FSDF: This method uses discrete cosine transform (DCT) combined with the basic 1×1 convolution operator to perform high and low frequency decomposition and learning.

[0007] BiFFN: This method integrates wavelet transform and computationally expensive graph neural network (GNN) to learn high- and low-frequency information.

[0008] The aforementioned frequency decomposition methods (FSDF and BiFFN) have achieved significant performance improvements by separating and processing high- and low-frequency information, and are currently the leading approach to addressing the modality discrepancy problem in CM-ReID. Although these frequency decomposition-based methods (such as FSDF and BiFFN) have achieved superior performance, they still face two key challenges and shortcomings:

[0009] 1) Feature learning is primarily performed in the spatial domain, failing to fully exploit the efficiency of frequency domain processing: While existing methods conceptually perform frequency decomposition, they still mostly process and learn the resulting high- and low-frequency features in the spatial domain through methods such as convolution or graph networks. This ignores the potential for efficient filtering and processing directly in the frequency domain, resulting in low computational efficiency and a failure to directly manipulate frequency information.

[0010] 2) Frequency decomposition is coarse-grained and limited to a high- and low-frequency dichotomy: Existing work generally employs a limited frequency decomposition strategy, simply dividing features into low- and high-frequency components. This dichotomy can be too coarse, overlooking subtle cross-modal information hidden in finer frequency bands (such as mid-frequency) that is crucial for distinguishing identity. Focusing solely on the high- and low-frequency dichotomy limits the robustness and diversity of features learned by the model.

[0011] In summary, existing CM-ReID methods, particularly when processing frequency features, have room for improvement in terms of computational efficiency and resource consumption. Furthermore, their limited frequency decomposition strategies lead to insufficient diversity and robustness in feature representation. These bottlenecks collectively limit the potential and effectiveness of this technology in more complex and widespread real-world scenarios. Summary of the Invention

[0012] The purpose of this application is to provide a cross-modal pedestrian re-identification method and system based on multi-frequency domain features, which can solve the problems of low computational efficiency, high resource consumption, and insufficient feature representation diversity and robustness in the existing technology, thereby obtaining a method that can accurately, efficiently and robustly identify and track pedestrians in different camera perspectives and cross-modal scenarios.

[0013] To achieve the above objectives, this application provides the following solutions:

[0014] In a first aspect, the present application provides a cross-modal person re-identification method based on multi-frequency domain features, comprising:

[0015] Obtain a pedestrian image set; the pedestrian image set includes a pedestrian image training set and a pedestrian image test set, and the pedestrian image training set and the pedestrian image test set both include modal V and modal I Pedestrian images;

[0016] Construct a cross-modal pedestrian re-identification model; the cross-modal pedestrian re-identification model includes: a ResNet50 model, a high-low frequency modulation module, and a frequency-aware diversity enhancement module; the ResNet50 model includes a first residual module group, a second residual module group, a third residual module group, and a fourth residual module group; the high-low frequency modulation module is located after the third residual module and the fourth residual module in the second residual module group, and after the fourth residual module and the fifth residual module in the third residual module group; the frequency-aware diversity enhancement module is located after the third residual module group;

[0017] Construct the total loss function;

[0018] Training the cross-modal person re-identification model based on the pedestrian image training set and the total loss function;

[0019] Inputting a pedestrian image to be identified and a pedestrian image test set into a trained cross-modal person re-identification model to obtain image features; the pedestrian image to be identified and the pedestrian image test set belong to different modalities;

[0020] Calculating the cosine similarity between the image of the pedestrian to be identified and each image in the pedestrian image test set using the image features;

[0021] sorting the images in the pedestrian image test set according to the cosine similarity;

[0022] According to the sorting results, the pedestrian image most similar to the pedestrian image to be identified is obtained, and cross-modal pedestrian re-identification is completed.

[0023] Optionally, the high and low frequency modulation module specifically includes the following steps:

[0024] Decompose the input features into high-frequency and low-frequency components to obtain high-frequency components and low-frequency components;

[0025] enhancing the high-frequency component and the low-frequency component;

[0026] modulating the enhanced high-frequency component and low-frequency component;

[0027] Fuse the modulated high-frequency component and low-frequency component to obtain the fused features;

[0028] The fused features and the input features are added through residual connection and zero-initialized batch normalization to obtain the output features of the high and low frequency modulation module.

[0029] Optionally, the frequency-aware diversity enhancement module specifically includes the following steps:

[0030] Obtain output features of the third residual module group;

[0031] Decomposing the amplitude of the output feature of the third residual module group into a plurality of non-overlapping regions according to frequency levels;

[0032] The amplitudes in the same area are averaged to obtain multiple average values;

[0033] Splice multiple average values ​​to obtain spliced ​​features;

[0034] Calculate the attention weight of each region based on the concatenated features;

[0035] Weighting the amplitude of each region based on the attention weight to obtain a weighted amplitude feature;

[0036] Combining the weighted amplitude feature with the phase to obtain a combined feature;

[0037] Performing inverse transformation on the combined features using IFFT to obtain inverse transformed features;

[0038] The inverse transformed features are batch normalized to obtain the output features of the frequency-aware diversity enhancement module.

[0039] Optionally, constructing the total loss function includes:

[0040] Constructing cross-modal soft loss functions;

[0041] Construct a cross-modal ranking regularization loss function;

[0042] Construct identity recognition loss function;

[0043] Construct a weighted regularized triplet loss function;

[0044] The final total loss function is obtained based on the cross-modal soft loss function, the cross-modal ranking regularization loss function, the identity recognition loss function and the weighted regularization triplet loss function.

[0045] Optionally, the expression of the cross-modal soft loss function is as follows:

[0046] ;

[0047] in, represents the cross-modal soft loss, Indicates modality V To Modal I Cross-modal soft loss, Indicates modality I To Modal V Cross-modal soft loss.

[0048] Optionally, the expression of the cross-modal ranking regularization loss function is as follows:

[0049] ;

[0050] in, represents the cross-modal ranking regularization loss, Indicates modality V To Modal I Cross-modal ranking regularization loss, Indicates modality I To Modal V Cross-modal ranking regularization loss.

[0051] Optionally, the total loss function is expressed as follows:

[0052] ;

[0053] in, represents the total loss, represents the identity loss, represents the weighted regularized triplet loss, represents the cross-modal soft loss, represents the cross-modal ranking regularization loss, and are all hyperparameters.

[0054] In a second aspect, the present application provides a cross-modal person re-identification system based on multi-frequency domain features, comprising:

[0055] The pedestrian image set acquisition module is used to acquire a pedestrian image training set and a pedestrian image test set; the pedestrian image training set and the pedestrian image test set both include modal V and modal I Pedestrian images;

[0056] A cross-modal pedestrian re-identification model construction module is used to construct a cross-modal pedestrian re-identification model; the cross-modal pedestrian re-identification model includes: a ResNet50 model, a high- and low-frequency modulation module, and a frequency-aware diversity enhancement module; the ResNet50 model includes a first residual module group, a second residual module group, a third residual module group, and a fourth residual module group; the high- and low-frequency modulation module is located after the third residual module and the fourth residual module in the second residual module group, and after the fourth residual module and the fifth residual module in the third residual module group; the frequency-aware diversity enhancement module is located after the third residual module group;

[0057] Total loss function construction module, used to construct the total loss function;

[0058] A training module, configured to train the cross-modal person re-identification model based on the pedestrian image training set and a total loss function;

[0059] An image feature acquisition module is used to input a pedestrian image to be identified and a pedestrian image test set into a trained cross-modal pedestrian re-identification model to obtain image features; the pedestrian image to be identified and the pedestrian image test set belong to different modalities;

[0060] a cosine similarity calculation module, configured to calculate the cosine similarity between the image of the pedestrian to be identified and each image in the pedestrian image test set using the image features;

[0061] a sorting module, configured to sort the images in the pedestrian image test set according to the cosine similarity;

[0062] The cross-modal person re-identification module is used to obtain the pedestrian image most similar to the pedestrian image to be identified based on the sorting results, and complete cross-modal person re-identification.

[0063] Optionally, the high and low frequency modulation module specifically includes the following units:

[0064] The high- and low-frequency decomposition unit is used to decompose the input features into high- and low-frequency components to obtain high-frequency components and low-frequency components;

[0065] an enhancing unit, configured to enhance the high-frequency component and the low-frequency component;

[0066] A modulation unit, configured to modulate the enhanced high-frequency component and low-frequency component;

[0067] A fusion unit is used to fuse the modulated high-frequency component and the low-frequency component to obtain a fused feature;

[0068] The batch normalization unit is used to add the fused features and the original input features through residual connection and zero-initialized batch normalization to obtain the output features of the high and low frequency modulation module.

[0069] Optionally, the frequency-aware diversity enhancement module specifically includes the following units:

[0070] An output feature acquisition unit of the third residual module group, configured to acquire output features of the third residual module group;

[0071] a decomposition unit, decomposing the amplitude of the output feature of the third residual module group into a plurality of non-overlapping regions according to the frequency level;

[0072] An average value calculation unit is used to calculate the average value of the amplitude in the same area to obtain multiple average values;

[0073] A splicing unit, used to splice multiple average values ​​to obtain spliced ​​features;

[0074] an attention weight calculation unit, configured to calculate the attention weight of each region based on the concatenated features;

[0075] a weighting unit, configured to weight the amplitude of each region based on the attention weight to obtain a weighted amplitude feature;

[0076] A feature combining unit, used to combine the weighted amplitude feature with the phase to obtain a combined feature;

[0077] an inverse transformation unit, configured to perform an inverse transformation on the combined features using IFFT to obtain inverse transformed features;

[0078] The batch normalization unit is used to perform batch normalization on the features after inverse transformation to obtain the output features of the frequency-aware diversity enhancement module.

[0079] According to the specific embodiments provided in this application, this application has the following technical effects:

[0080] This application provides a cross-modal pedestrian re-identification method and system based on multi-frequency domain features. By introducing a high- and low-frequency modulation module, low-frequency and high-frequency features can be efficiently decomposed and processed, solving the problem of excessive modal differences in cross-modal feature extraction in traditional methods. The high- and low-frequency modulation module directly filters low-frequency features in the frequency domain and performs lightweight multi-scale mining of high-frequency features in the spatial domain, significantly improving the efficiency and robustness of feature extraction. This application adopts a frequency-aware diversity enhancement module, which enhances feature diversity by weighting and optimizing multi-frequency features in the frequency domain. The frequency-aware diversity enhancement module can effectively reduce the redundancy of feature extraction in a multi-branch structure and further improve the distinguishability of features through frequency-aware weighting.

[0081] In addition, this application adopts a total loss function, namely the cross-modal soft retrieval loss function, which enhances the robustness of the feature by constraining the maximum cross-modal distance of instances of the same category to be less than the minimum same-modal distance. The cross-modal soft retrieval loss function can comprehensively consider the influence of all sample pairs by introducing softmax and softmin operations, avoiding the limitation of traditional hard retrieval loss that only focuses on the maximum and minimum sample pairs;

[0082] This application uses a cross-modal ranking regularization loss function to enhance feature diversity by maximizing the ranking inconsistency between features extracted by different branches. By optimizing the ranking correlation of features, the cross-modal ranking regularization loss function can better handle complex cross-modal variations, further improving the model's performance in cross-modal person re-identification tasks. Furthermore, the differentiability of the cross-modal ranking regularization loss function makes the training process more efficient and stable, further improving the model's training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0084] Figure 1 A flowchart of a cross-modal person re-identification method based on multi-frequency domain features provided in one embodiment of the present application;

[0085] Figure 2 A schematic diagram of the cross-modal person re-identification model structure provided in one embodiment of the present application;

[0086] Figure 3 A schematic diagram of a high and low frequency modulation module provided in one embodiment of the present application;

[0087] Figure 4 A schematic diagram of a frequency-aware diversity enhancement module provided in one embodiment of the present application;

[0088] Figure 5 A schematic diagram of a cross-modal soft loss function provided in one embodiment of the present application;

[0089] Figure 6 A schematic diagram of a cross-modal ranking regularization loss function provided in one embodiment of the present application;

[0090] Figure 7 This is a schematic diagram of a lightweight multi-scale module according to an embodiment of the present application;

[0091] Figure 8 This is a visualization effect diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0093] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0094] In an exemplary embodiment, see Figure 1 and Figure 2, provides a cross-modal pedestrian re-identification method based on multi-frequency domain features, the method comprising the following steps 101 to 108. Among them:

[0095] Step 101: Obtain a pedestrian image set; the pedestrian image set includes a pedestrian image training set and a pedestrian image test set, both of which are modal V and modal I Pedestrian images.

[0096] Specifically, we collect and preprocess a cross-modal person re-identification dataset, which contains V and modal I Pedestrian images and their corresponding pedestrian identity labels. The preprocessing steps include image cropping, normalization, and data augmentation to ensure that the model can learn robust features.

[0097] Step 102: Construct a cross-modal pedestrian re-identification model; the cross-modal pedestrian re-identification model includes: a ResNet50 model, a high- and low-frequency modulation module, and a frequency-aware diversity enhancement module; the ResNet50 model includes a first residual module group, a second residual module group, a third residual module group, and a fourth residual module group; the high- and low-frequency modulation module is located after the third residual module and the fourth residual module in the second residual module group, and after the fourth residual module and the fifth residual module in the third residual module group; the frequency-aware diversity enhancement module is located after the third residual module group.

[0098] In order to learn robust modality-invariant features for CM-ReID, high and low frequency modulation modules (e.g. Figure 3 As shown in Figure 2, the low-frequency and high-frequency components are processed separately to learn overall features in the frequency domain and enrich details in the spatial domain. This module operates through four key stages:

[0099] (1) High and low frequency decomposition

[0100] Given an input feature ( ,in 、 、 、 Represents the four residual modules in the second and third residual module groups of ResNet50 respectively 、 、 、 output of ), where represents a real number, 、 and Represent the number of channels, height, and width of the feature respectively. First, use Fast Fourier Transform (FFT) to convert it to the frequency domain:

[0101] (1)

[0102] in, represents the FFT in the 2D real-valued channel direction, Represents a complex number. Its formal definition is:

[0103] (2)

[0104] in, is an imaginary unit, and are the frequency domain subscripts in the vertical and horizontal directions, and Represents the vertical and horizontal spatial subscripts respectively. Defined as input features Inverse Fast Fourier Transform (IFFT). In the frequency domain, a low-frequency binary mask is predefined (The frequency domain size is )for:

[0105] (3)

[0106] Among them, the hyperparameters Balance low-frequency and high-frequency components, and are the frequency domain subscripts in the vertical and horizontal directions respectively. Correspondingly, the high-frequency binary mask can be obtained by Then the low-frequency and high-frequency components can be decomposed into:

[0107] (4)

[0108] in, Represents point-wise element-wise multiplication.

[0109] (2) Low-frequency adaptation

[0110] Given that low-frequency components carry global style information, using small kernel convolution in the spatial domain is not enough to capture long-range dependencies. At the same time, Transformers exhibit excessively high quadratic computational complexity, especially in shallow network layers. Therefore, this application introduces a parameterized complex weight matrix ( Indicates plural, , and Represent the number of channels, height and width of the complex weight matrix respectively), thereby obtaining the enhanced low-frequency features:

[0111] (5)

[0112] Then, the enhanced low-frequency features are transformed into Convert back to the spatial domain to obtain the enhanced low-frequency spatial domain features:

[0113] (6)

[0114] (3) High-frequency enhancement

[0115] High-frequency features capture the details and contours of people at different scales, which is crucial for extracting modality-invariant features and is very suitable for feature extraction in the spatial domain through convolution. Convolution is not enough to extract multi-scale features. To solve this problem, this application designs a lightweight multi-scale (Lite Multiscale, LM) module (such as Figure 7 as shown), to achieve efficient multi-scale feature learning.

[0116] First, the high frequency component in formula (4) Obtain high-frequency spatial features through IFFT ( , and Represent the number of channels, height, and width of high-frequency spatial features respectively). In order to integrate the sparse high-frequency features across channels, we use Point-wise convolution ( ) to reduce high-frequency spatial characteristics The number of channels is calculated to obtain the high-frequency spatial domain features after dimensionality reduction:

[0117] (7)

[0118] in is the dimensionality reduction factor. Then, we stack different numbers of lightweight Convolution is used to construct multi-scale convolution. , the obtained features are:

[0119] (8)

[0120] in, Indicates deep separation convolution, express Point-wise convolution, express Lightweight Convolution, each lightweight convolution Depend on and In order to integrate scale To solve the problem of spatial aggregation, this application uses a shared Efficient Aggregation Gate (EAG), which first aggregates the spatial dimension using avgpooling and maxpooling, and then concatenates them in the spatial direction to obtain the following features:

[0121] (9)

[0122] in, Represents the splicing operation in the spatial direction. Then, a one-dimensional convolution with a convolution kernel size of 5 is used in the channel dimension ( ), and using the Sigmoid activation function ( ) Generate weighting coefficients:

[0123] (10)

[0124] Use this weighting coefficient to adjust the characteristics of formula (8) Perform weighting in the channel direction and add the weighted features of each scale to obtain the features:

[0125] (11)

[0126] End use Point-wise convolution ( ) Restore the original channel dimension to obtain:

[0127] (12)

[0128] (4) Modulation and fusion based on spatial attention

[0129] According to formulas (6) and (12), the processed low-frequency features can be obtained respectively and high-frequency features Considering the noise inevitably introduced in low-frequency and high-frequency features, Generate low-frequency spatial masks separately and high-frequency spatial mask :

[0130] (13)

[0131] in, The kernel size is The convolution of the input features Number of channels from is reduced to 2, corresponding to the spatial mask , Represents the Sigmoid activation function. These two spatial masks are used to modulate low-frequency features in space. and high-frequency features The modulated low-frequency and high-frequency features are then fused by direct addition to obtain the feature:

[0132] (14)

[0133] In order to stabilize the training, the fused features are finally With the original input features Batch normalization with residual connections and zero initialization ( ) are added together to obtain the output characteristics of the high and low frequency modulation modules :

[0134] (15)

[0135] As mentioned at the beginning of this section, the input features ,in 、 、 、 Represents the four residual modules in the second and third residual module groups of ResNet50 respectively 、 、 、 The output of , accordingly, its final output feature , respectively representing the output characteristics of the high and low frequency modulation modules at the corresponding positions.

[0136] In summary, the high- and low-frequency modulation module effectively integrates comprehensive frequency-domain processing, efficient feature enhancement, and spatial attention-based modulation to achieve robust modality-invariant feature learning, providing significant advantages over existing methods for the CM-ReID task.

[0137] Frequency-aware diversity enhancement module (such as Figure 4 As shown in Figure 2, it adaptively performs attention weighting by perceiving different frequency bands and integrates multiple branches to extract rich modality-invariant features. The input features of this module are , , and Represents the number of channels, height and width of the feature, Represents the residual module in the third residual module group of ResNet50 The module operates through three key stages:

[0138] (1) Multi-frequency decomposition

[0139] First, the amplitude According to the frequency level, it is divided into non-overlapping regions :

[0140] (16)

[0141] in, and Represents the input features respectively The height and width, and are the frequency domain subscripts in the vertical and horizontal directions respectively. Then, the same area Amplitude within Perform the averaging operation:

[0142] (17)

[0143] in, Representing a collection The number of elements in , yes The number of channels. Features Stitched together, we get the features:

[0144] (18)

[0145] in, Represents splicing, features , yes The number of channels.

[0146] (2) Frequency-aware attention

[0147] Next, a lightweight frequency perception module is used to Regions Generate attention weights separately:

[0148] (19)

[0149] in, is the Sigmoid activation function, Represents a one-dimensional convolution with a kernel size of 1, which can be regarded as performing point-by-point convolution between regions. The number of channels from Reduced to 1, the final output attention weight is a dimension of Using this attention weight, the amplitude features of each region can be weighted:

[0150] (20)

[0151] in, Representing a collection Elements in .

[0152] (3) Diversity Enhancer

[0153] To increase feature diversity, parallel attention branches (Formula (19)) generate different attention weights Then, through formula (20), each branch is obtained Different characteristics , these characteristics are respectively related to the phase ( express Take the phase of the feature after FFT) and then use IFFT to perform inverse transform to produce Different characteristics :

[0154] (twenty one)

[0155] in, is an imaginary unit, Expressed as a natural constant An exponential function with base .

[0156] To ensure the stability of optimization, batch normalization (BN) is introduced in each branch to obtain the following features:

[0157] (twenty two)

[0158] In summary, the frequency-aware diversity enhancement module provides superior frequency adaptability and can more accurately preserve the modal invariance characteristics in different frequency bands.

[0159] Step 103: Construct a total loss function.

[0160] The total loss function includes: a cross-modal soft loss function, a cross-modal ranking regularization loss function, an identity recognition loss function, and a weighted regularization triplet loss function.

[0161] Among them, the cross-modal soft retrieval loss function (such as Figure 5 ), aims to make the maximum cross-modal distance of instances of the same category smaller than the minimum same-modal distance. Specifically, for the modality Feature Embedding and modal Feature Embedding , define the maximum cross-modal distance and the minimum same-modal distance as follows (with For example):

[0162] (twenty three)

[0163] in, Indicates modality No. Features and modes No. The Euclidean distance between features, Indicates the The category to which the feature belongs. Introducing boundary values , define the cross-modal hard retrieval loss as:

[0164] (twenty four)

[0165] in, express In order to overcome the limitation of hard retrieval loss that only considers the maximum and minimum sample pairs, this application introduces soft operations, replacing the maximum and minimum operations with softmax and softmin operations, which are defined as follows:

[0166] (25)

[0167] in, is a hyperparameter that controls the degree of smoothing. Indicates modality V No. i Features and modes V No. j The Euclidean distance between features, Indicates the first k The identity tag of the feature, Indicates the first i The identity tag of the feature, Indicates the first j The identity label of each feature is substituted into the cross-modal hard retrieval loss formula to obtain the cross-modal soft retrieval loss:

[0168] (26)

[0169] Also consider and , the final cross-modal soft retrieval loss is:

[0170] (27)

[0171] when hour, and Corresponding to the average distance between the anchor sample and other samples, Equivalent to the mean square error loss (MSEL). When , they correspond to the maximum and minimum distances respectively, then It is equivalent to the cross-modal retrieval loss (CMR). Therefore, the cross-modal soft retrieval loss function proposed in this application can be regarded as a compromise between MSEL and CMR.

[0172] Cross-modal ranking regularization loss function (such as Figure 6 (shown in ) aims to enhance the diversity of features by ranking differences between pairs of features from different modalities. Specifically, given Group Mode Features and Group Mode Features , calculate the distance list between the same category but different modal features in each group (in terms of For example):

[0173] (28)

[0174] in, Indicates the Group Mode The middle Features to Modes No. The Euclidean distance of the features, is the number of samples of the same category in the mini-batch data. Based on the above distance list, we can get the sorted list of features:

[0175] (29)

[0176] in, Indicates the distance in the list The index corresponding to the small distance is:

[0177] (30)

[0178] Group-based and group For different sorted lists, the Spearman rank correlation coefficient is defined as:

[0179] (31)

[0180] in, express The average index of represents the L2 norm of the orientation quantity, Represents the dot product of two vectors. , the smaller the value, the greater the difference between the two groups of sorted lists, thus achieving the goal of sorting diversity. (Formula (28)) gets the sorted list The process of (Formula (29)) is non-differentiable, so we introduce Operation, making the process differentiable:

[0181] (32)

[0182] in, Represents a differentiable sorted list. Based on this, we can obtain a differentiable cross-modal sorting regularization loss function:

[0183] (33)

[0184] in, P Indicates the number of pedestrians with different identities in a small batch of data, K Indicates the number of images of pedestrians with the same identity in a small batch of data, N Indicates the number of parallel branches in the diversity enhancer.

[0185] Also consider and , the final cross-modal ranking regularization loss function is:

[0186] (34)

[0187] The cross-modal ranking regularization loss function can effectively enhance the diversity of features, improve the performance of cross-modal retrieval, and is suitable for a variety of cross-modal retrieval tasks.

[0188] Step 104: Train the cross-modal person re-identification model based on the pedestrian image training set and the total loss function.

[0189] In the CM-ReID task, the dataset contains Pedestrian images and modal Pedestrian images , each mode contains identities, each with Zhang Modal Images and Zhang Modal These images are first passed through the residual module of ResNet50 、 、 、 、 , get the features and features ; Then through a high and low frequency modulation module, the characteristic and features ; Then pass through the residual module of ResNet50 , get the features and features , and then pass through a high and low frequency modulation module again to obtain the characteristic and features ; After that, they pass through the residual module of ResNet50 again 、 、 、 , get the features and features , and after a high and low frequency modulation module, the characteristic and features ; Similarly, after the residual module of ResNet50 , get the features and , and then pass through a high and low frequency modulation module again to obtain the characteristic and features ;Finally, pass the residual module of ResNet50 Thus, the characteristics and features , these two sets of features are obtained through the frequency-aware diversity enhancement module Different sets of characteristics and features Finally, the fourth residual module group of ResNet50 is used Process the features of different branches separately to obtain features ,in After generalized mean pooling and batch normalization layers, we can get:

[0190] (35)

[0191] in, , Represents the feature dimension. represents generalized mean pooling, represents batch normalization, represents the feature after generalized mean pooling, Represents the features after the batch normalization layer.

[0192] During the training process, and Applied to features before and after batch normalization (BN). For features after BN, L2 normalization is first performed and these two loss functions are also applied. Its complete training objective combines multiple loss functions, as follows:

[0193] (36)

[0194] in, is the identity loss, is the weighted regularized triplet loss, is the cross-modal soft retrieval loss, is the cross-modal ranking regularization loss, and is a hyperparameter used to control the trade-off between different loss functions.

[0195] During the testing phase, The concatenation result is taken as the final feature.

[0196] Step 105: Inputting the pedestrian image to be identified and the pedestrian image test set into the trained cross-modal pedestrian re-identification model to obtain image features; the pedestrian image to be identified and the pedestrian image test set belong to different modalities.

[0197] Step 106: Using the image features, calculate the cosine similarity between the image of the pedestrian to be identified and each image in the pedestrian image test set.

[0198] Step 107: Sort the images in the pedestrian image test set according to the cosine similarity.

[0199] Step 108: Based on the sorting results, the pedestrian image that is most similar to the pedestrian image to be identified is obtained to complete cross-modal pedestrian re-identification.

[0200] Based on the same inventive concept, the embodiments of the present application also provide a system for implementing the aforementioned cross-modal pedestrian re-identification system based on multi-frequency domain features. The implementation solution provided by this system is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the cross-modal pedestrian re-identification system based on multi-frequency domain features provided below can be found in the limitations of the cross-modal pedestrian re-identification method based on multi-frequency domain features above, and will not be repeated here.

[0201] In an exemplary embodiment, a cross-modal person re-identification system based on multi-frequency domain features is provided, including:

[0202] The pedestrian image set acquisition module is used to acquire a pedestrian image training set and a pedestrian image test set; the pedestrian image training set and the pedestrian image test set both include modal V and modal I Pedestrian images;

[0203] A cross-modal pedestrian re-identification model construction module is used to construct a cross-modal pedestrian re-identification model; the cross-modal pedestrian re-identification model includes: a ResNet50 model, a high- and low-frequency modulation module, and a frequency-aware diversity enhancement module; the ResNet50 model includes a first residual module group, a second residual module group, a third residual module group, and a fourth residual module group; the high- and low-frequency modulation module is located after the third residual module and the fourth residual module in the second residual module group, and after the fourth residual module and the fifth residual module in the third residual module group; the frequency-aware diversity enhancement module is located after the third residual module group;

[0204] Total loss function construction module, used to construct the total loss function;

[0205] A training module, configured to train the cross-modal person re-identification model based on the pedestrian image training set and a total loss function;

[0206] An image feature acquisition module is used to input a pedestrian image to be identified and a pedestrian image test set into a trained cross-modal pedestrian re-identification model to obtain image features; the pedestrian image to be identified and the pedestrian image test set belong to different modalities;

[0207] a cosine similarity calculation module, configured to calculate the cosine similarity between the image of the pedestrian to be identified and each image in the pedestrian image test set using the image features;

[0208] a sorting module, configured to sort the images in the pedestrian image test set according to the cosine similarity;

[0209] The cross-modal person re-identification module is used to obtain the pedestrian image most similar to the pedestrian image to be identified based on the sorting results, and complete cross-modal person re-identification.

[0210] The high and low frequency modulation module includes the following units:

[0211] The high- and low-frequency decomposition unit is used to decompose the input features into high- and low-frequency components to obtain high-frequency components and low-frequency components;

[0212] an enhancing unit, configured to enhance the high-frequency component and the low-frequency component;

[0213] A modulation unit, configured to modulate the enhanced high-frequency component and low-frequency component;

[0214] A fusion unit is used to fuse the modulated high-frequency component and the low-frequency component to obtain a fused feature;

[0215] The batch normalization unit is used to add the fused features and the original input features through residual connection and zero-initialized batch normalization to obtain the output features of the high and low frequency modulation module.

[0216] The frequency-aware diversity enhancement module includes the following units:

[0217] An output feature acquisition unit of the third residual module group, configured to acquire output features of the third residual module group;

[0218] a decomposition unit, decomposing the amplitude of the output feature of the third residual module group into a plurality of non-overlapping regions according to the frequency level;

[0219] An average value calculation unit is used to calculate the average value of the amplitude in the same area to obtain multiple average values;

[0220] A splicing unit, used to splice multiple average values ​​to obtain spliced ​​features;

[0221] an attention weight calculation unit, configured to calculate the attention weight of each region based on the concatenated features;

[0222] a weighting unit, configured to weight the amplitude of each region based on the attention weight to obtain a weighted amplitude feature;

[0223] A feature combining unit, used to combine the weighted amplitude feature with the phase to obtain a combined feature;

[0224] an inverse transformation unit, configured to perform an inverse transformation on the combined features using IFFT to obtain inverse transformed features;

[0225] The batch normalization unit is used to perform batch normalization on the features after inverse transformation to obtain the output features of the frequency-aware diversity enhancement module.

[0226] To verify the effectiveness of this application, experiments were conducted on the standard datasets SYSU-MM01 and LLCM in the field of cross-modal person re-identification. The experimental results are shown in the following table:

[0227] Table 1 Experimental results of the present invention in the cross-modal person re-identification task

[0228]

[0229] mAP (Mean Average Precision) and Rank-1 are two commonly used indicators for evaluating the performance of information retrieval systems.

[0230] 1. mAP (Mean Average Precision) is a metric that measures the overall performance of a model across multiple categories. It calculates the average precision of the model at different recall levels.

[0231] 2.Rank-1 refers to the proportion of the most likely category (i.e., the first-ranked category) returned by the model for a given input sample that is consistent with the true category.

[0232] These two metrics evaluate model performance from different perspectives. mAP more comprehensively considers the model's performance at different recall levels, while Rank-1 Accuracy focuses on the model's performance on the most likely prediction. In practical applications, the appropriate evaluation metric should be selected based on the specific requirements of the task.

[0233] Furthermore, the visualization results on the SYSU-MM01 dataset are as follows: Figure 8 As shown in the figure. By using Grad-CAM++ technology, it is possible to intuitively show how the model focuses on the key areas of the target. Figure 8 As shown in the figure, data of different modalities have a good attention effect. This result not only intuitively demonstrates the benefits of the present invention, but also further confirms the robustness and accuracy of the model in complex environments.

[0234] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0235] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A cross-modal person re-identification method based on multi-frequency domain features, characterized by: The cross-modal person re-identification method based on multi-frequency domain features includes: Obtain a pedestrian image set; the pedestrian image set includes a pedestrian image training set and a pedestrian image test set, and the pedestrian image training set and the pedestrian image test set both include modal V and modal I Pedestrian images; Construct a cross-modal pedestrian re-identification model; the cross-modal pedestrian re-identification model includes: a ResNet50 model, a high-low frequency modulation module, and a frequency-aware diversity enhancement module; the ResNet50 model includes a first residual module group, a second residual module group, a third residual module group, and a fourth residual module group; the high-low frequency modulation module is located after the third residual module and the fourth residual module in the second residual module group, and after the fourth residual module and the fifth residual module in the third residual module group; the frequency-aware diversity enhancement module is located after the third residual module group; Construct the total loss function; Training the cross-modal person re-identification model based on the pedestrian image training set and the total loss function; Inputting a pedestrian image to be identified and a pedestrian image test set into a trained cross-modal person re-identification model to obtain image features; the pedestrian image to be identified and the pedestrian image test set belong to different modalities; Calculating the cosine similarity between the image of the pedestrian to be identified and each image in the pedestrian image test set using the image features; sorting the images in the pedestrian image test set according to the cosine similarity; According to the sorting results, the pedestrian image that is most similar to the pedestrian image to be identified is obtained to complete cross-modal pedestrian re-identification; The high and low frequency modulation module specifically includes the following steps: Decompose the input features into high-frequency and low-frequency components to obtain high-frequency components and low-frequency components; enhancing the high-frequency component and the low-frequency component; modulating the enhanced high-frequency component and low-frequency component; Fuse the modulated high-frequency component and low-frequency component to obtain the fused features; Adding the fused features and the input features through residual connection and zero-initialized batch normalization to obtain output features of the high and low frequency modulation module; The frequency-aware diversity enhancement module specifically includes the following steps: Obtain output features of the third residual module group; Decomposing the amplitude of the output feature of the third residual module group into a plurality of non-overlapping regions according to frequency levels; The amplitudes in the same area are averaged to obtain multiple average values; Splice multiple average values ​​to obtain spliced ​​features; Calculate the attention weight of each region based on the concatenated features; Weighting the amplitude of each region based on the attention weight to obtain a weighted amplitude feature; Combining the weighted amplitude feature with the phase to obtain a combined feature; Performing inverse transformation on the combined features using IFFT to obtain inverse transformed features; Batch normalization is performed on the inverse transformed features to obtain the output features of the frequency-aware diversity enhancement module; The constructing of the total loss function includes: Constructing cross-modal soft loss functions; Construct a cross-modal ranking regularization loss function; Construct identity recognition loss function; Construct a weighted regularized triplet loss function; The final total loss function is obtained based on the cross-modal soft loss function, the cross-modal ranking regularization loss function, the identity recognition loss function and the weighted regularization triplet loss function.

2. The cross-modal person re-identification method based on multi-frequency domain features according to claim 1, characterized in that: The expression of the cross-modal soft loss function is as follows: ； in, represents the cross-modal soft loss, Indicates modality V To Modal I Cross-modal soft loss, Indicates modality I To Modal V Cross-modal soft loss.

3. The cross-modal person re-identification method based on multi-frequency domain features according to claim 1, characterized in that: The expression of the cross-modal ranking regularization loss function is as follows: ； in, represents the cross-modal ranking regularization loss, Indicates modality V To Modal I Cross-modal ranking regularization loss, Indicates modality I To Modal V Cross-modal ranking regularization loss.

4. The cross-modal person re-identification method based on multi-frequency domain features according to claim 1, characterized in that: The expression of the total loss function is as follows: ； in, represents the total loss, represents the identity loss, represents the weighted regularized triplet loss, represents the cross-modal soft loss, represents the cross-modal ranking regularization loss, and are all hyperparameters.

5. A cross-modal person re-identification system based on multi-frequency domain features, characterized by: The cross-modal person re-identification system based on multi-frequency domain features includes: The pedestrian image set acquisition module is used to acquire a pedestrian image training set and a pedestrian image test set; the pedestrian image training set and the pedestrian image test set both include modal V and modal I Pedestrian images; A cross-modal pedestrian re-identification model construction module is used to construct a cross-modal pedestrian re-identification model; the cross-modal pedestrian re-identification model includes: a ResNet50 model, a high- and low-frequency modulation module, and a frequency-aware diversity enhancement module; the ResNet50 model includes a first residual module group, a second residual module group, a third residual module group, and a fourth residual module group; the high- and low-frequency modulation module is located after the third residual module and the fourth residual module in the second residual module group, and after the fourth residual module and the fifth residual module in the third residual module group; the frequency-aware diversity enhancement module is located after the third residual module group; Total loss function construction module, used to construct the total loss function; A training module, configured to train the cross-modal person re-identification model based on the pedestrian image training set and a total loss function; An image feature acquisition module is used to input a pedestrian image to be identified and a pedestrian image test set into a trained cross-modal pedestrian re-identification model to obtain image features; the pedestrian image to be identified and the pedestrian image test set belong to different modalities; a cosine similarity calculation module, configured to calculate the cosine similarity between the image of the pedestrian to be identified and each image in the pedestrian image test set using the image features; a sorting module, configured to sort the images in the pedestrian image test set according to the cosine similarity; The cross-modal person re-identification module is used to obtain the pedestrian image most similar to the pedestrian image to be identified based on the sorting results, and complete cross-modal pedestrian re-identification; The high and low frequency modulation module specifically includes the following steps: Decompose the input features into high-frequency and low-frequency components to obtain high-frequency components and low-frequency components; enhancing the high-frequency component and the low-frequency component; modulating the enhanced high-frequency component and low-frequency component; Fuse the modulated high-frequency component and low-frequency component to obtain the fused features; Adding the fused features and the input features through residual connection and zero-initialized batch normalization to obtain output features of the high and low frequency modulation module; The frequency-aware diversity enhancement module specifically includes the following steps: Obtain output features of the third residual module group; Decomposing the amplitude of the output feature of the third residual module group into a plurality of non-overlapping regions according to frequency levels; The amplitudes in the same area are averaged to obtain multiple average values; Splice multiple average values ​​to obtain spliced ​​features; Calculate the attention weight of each region based on the concatenated features; Weighting the amplitude of each region based on the attention weight to obtain a weighted amplitude feature; Combining the weighted amplitude feature with the phase to obtain a combined feature; Performing inverse transformation on the combined features using IFFT to obtain inverse transformed features; Batch normalization is performed on the inverse transformed features to obtain the output features of the frequency-aware diversity enhancement module; The constructing of the total loss function includes: Constructing cross-modal soft loss functions; Construct a cross-modal ranking regularization loss function; Construct identity recognition loss function; Construct a weighted regularized triplet loss function; The final total loss function is obtained based on the cross-modal soft loss function, the cross-modal ranking regularization loss function, the identity recognition loss function and the weighted regularization triplet loss function.

6. The cross-modal person re-identification system based on multi-frequency domain features according to claim 5, characterized in that: The high and low frequency modulation module specifically includes the following units: The high- and low-frequency decomposition unit is used to decompose the input features into high- and low-frequency components to obtain high-frequency components and low-frequency components; an enhancing unit, configured to enhance the high-frequency component and the low-frequency component; A modulation unit, configured to modulate the enhanced high-frequency component and low-frequency component; A fusion unit is used to fuse the modulated high-frequency component and the low-frequency component to obtain a fused feature; The batch normalization unit is used to add the fused features and the original input features through residual connection and zero-initialized batch normalization to obtain the output features of the high and low frequency modulation module.

7. The cross-modal person re-identification system based on multi-frequency domain features according to claim 5, characterized in that: The frequency-aware diversity enhancement module specifically includes the following units: An output feature acquisition unit of the third residual module group, configured to acquire output features of the third residual module group; a decomposition unit, decomposing the amplitude of the output feature of the third residual module group into a plurality of non-overlapping regions according to the frequency level; An average value calculation unit is used to calculate the average value of the amplitude in the same area to obtain multiple average values; A splicing unit, used to splice multiple average values ​​to obtain spliced ​​features; an attention weight calculation unit, configured to calculate the attention weight of each region based on the concatenated features; a weighting unit, configured to weight the amplitude of each region based on the attention weight to obtain a weighted amplitude feature; A feature combining unit, used to combine the weighted amplitude feature with the phase to obtain a combined feature; an inverse transformation unit, configured to perform an inverse transformation on the combined features using IFFT to obtain inverse transformed features; The batch normalization unit is used to perform batch normalization on the features after inverse transformation to obtain the output features of the frequency-aware diversity enhancement module.

Images