Frequency domain enhanced cross-modal pedestrian re-identification method, system and device and medium

Abstract

The invention discloses a frequency domain enhanced cross-modal pedestrian re-identification method, system and device and a medium, which are corresponding schemes, in the scheme, through frequency-based data enhancement, the illumination difference between different modals of an image layer can be reduced, and the accuracy of pedestrian re-identification is improved. Frequency domain clues can be extracted and fused through channel frequency attention and spatial frequency attention of a model level, then key frequency domain details are adaptively captured through frequency domain cross-modal feature enhancement, contours and details related to identities are more effectively extracted, irrelevant illumination and colors are eliminated, and on the basis, the identification accuracy is improved. And cross-modal pedestrian re-identification in a real scene can be accurately realized.

Description

Technical Field

[0001] This invention relates to the field of image frequency information mining and fusion technology, and in particular to a frequency domain enhanced cross-modal pedestrian re-identification method, system, device and medium. Background Technology

[0002] Pedestrian re-identification (ReID) aims to match images of the same person captured by different cameras, and it is widely used in surveillance systems. To achieve 24-hour pedestrian re-identification, infrared (IR) cameras are used to capture images of pedestrians at night, which has spurred research into visible-infrared person re-identification (VI-ReID). Unlike single-modal pedestrian re-identification, the large transmodal differences between RGB (red, green, blue) images and IR images, as well as the intramodal variations in IR images, present new challenges to pedestrian re-identification research. The modal differences mainly stem from the different lighting conditions of RGB and IR images. For example, RGB images capture light in the red, green, and blue bands, providing rich color information, while IR images capture light in a single band. Many studies have explored this. However, research has found that differences in light sources are often overlooked. The light sources for RGB images include artificial lighting, sunlight, and ambient reflected light, resulting in clearer details. In contrast, infrared images captured by surveillance cameras rely solely on the camera's light source, which often leads to underexposed and overexposed images. Therefore, the brightness variations caused by different light sources significantly exacerbate the intra-identity gap, posing a serious challenge to the re-identification of pedestrians in visible light and infrared light.

[0003] Most existing methods operate purely in the spatial domain, neglecting frequency information, making it difficult to mitigate modal differences caused by variations in lighting conditions. Wang et al., in "Cross-modality paired-images generation for RGB-infrared person re-identification, AAAI 2020," and Choi et al., in "Hi-CMD: Hierarchical cross-modality disentanglement for visible-infrared person re-identification, CVPR 2022," bridged appearance gaps by generating images from one modality to another. However, due to the lack of cross-modal pose-aligned image pairs, the generated images struggle to retain accurate identity information.

[0004] Other methods attempt to reduce modal discrepancies through data augmentation. Ye et al., in "Channel augmentedjoint learning for visible-infrared recognition. ICCV2021," uniformly generated color-independent images by randomly swapping color channels. Liu et al., in "SFANet: A spectrum-aware featureaugmentation network for visible-infrared person reidentification. TNNLS2021," completely replaced RGB images with grayscale images for feature learning. Qian et al., in "Visible-infrared person reidentification via patch-mixed cross-modality learning. PR2025," segmented two images of the same person from two different modalities into patches and stitched them together to form a new patch. Still other methods aim to reduce modal discrepancies in the feature space through carefully designed feature augmentation modules and cross-modal loss functions. In general, these spatial cross-modal person re-identification methods are steadily advancing the field towards reducing modal discrepancies, but they still cannot fully meet the needs of cross-modal person re-identification in real-world scenarios.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a frequency domain enhanced cross-modal pedestrian re-identification method, system, device, and medium that can adaptively capture key frequency domain details to more effectively extract identity-related contours and details, and accurately achieve cross-modal pedestrian re-identification in real-world scenarios.

[0007] The objective of this invention is achieved through the following technical solution: A frequency-domain enhanced cross-modal person re-identification method includes: Step 1: Transform the images of each mode to Fourier space, swap the amplitude components of the images of different modes in Fourier space, and transform them back to the image domain to obtain the enhanced images of each mode; wherein, the modes include: visible light mode and infrared mode; Step 2: For each modality, after extracting the feature map from the corresponding enhanced image, perform spatial frequency attention operation in Fourier space to extract frequency domain cues and fuse them to obtain the fused feature map. Step 3: For each mode, based on the fused feature map, perform channel frequency attention operation in Fourier space to extract frequency domain cues and fuse them to obtain the final enhanced feature map. Step 4: For each mode, based on the final enhanced feature map, obtain the aggregated features through feature aggregation, and based on the final enhanced feature map, obtain the corresponding enhanced frequency features through frequency calculation and enhancement processing. Step 5: For each modality, use aggregated features and enhanced frequency features for pedestrian re-identification.

[0008] A frequency-domain enhanced cross-modal person re-identification system, used to implement the aforementioned method, includes: The frequency-based data augmentation module is used to transform the images of each mode to Fourier space, exchange the amplitude components of the images of different modes in Fourier space, and transform them back to the image domain to obtain the enhanced images of each mode; wherein, the modes include: visible light mode and infrared mode; The spatial frequency attention module is used to extract feature maps from the corresponding enhanced images for each modality, and then perform spatial frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain fused feature maps. The channel frequency attention module is used to perform channel frequency attention operations in Fourier space for each mode based on the fused feature map to extract frequency domain cues and fuse them to obtain the final enhanced feature map. The frequency enhancement and optimization module is used to obtain aggregated features for each mode by feature aggregation based on the final enhanced feature map, and to obtain the corresponding enhanced frequency features by frequency calculation and enhancement processing based on the final enhanced feature map. The pedestrian re-identification unit is used for pedestrian re-identification for each modality by utilizing aggregated features and enhanced frequency features.

[0009] A processing device includes: one or more processors; and a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0010] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0011] As can be seen from the technical solution provided by the present invention, frequency-based data augmentation can reduce the illumination differences between different modalities at the image level. Channel frequency attention and spatial frequency attention at the model level can extract and fuse frequency domain cues. Furthermore, through frequency domain cross-modal feature enhancement, key frequency domain details can be adaptively captured to more effectively extract identity-related contours and details while excluding irrelevant lighting and colors. Based on this, cross-modal pedestrian re-identification in real-world scenarios can be accurately achieved. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a frequency domain enhanced cross-modal pedestrian re-identification method provided in an embodiment of the present invention.

[0014] Figure 2 This is a schematic diagram of a frequency-based data enhancement module provided in an embodiment of the present invention.

[0015] Figure 3 This is a schematic diagram of the spatial frequency attention module provided in an embodiment of the present invention.

[0016] Figure 4 This is a schematic diagram of the channel frequency attention module provided in an embodiment of the present invention.

[0017] Figure 5 This is a schematic diagram of the frequency enhancement optimization module provided in an embodiment of the present invention.

[0018] Figure 6 This is a schematic diagram of a frequency-domain enhanced cross-modal pedestrian re-identification system provided in an embodiment of the present invention.

[0019] Figure 7 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0021] First, the following explanations are provided for the terms that may be used in this article: The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0022] The term "composed of" excludes any technical features not expressly listed. When used in a claim, it closes the claim to exclude all technical features other than those expressly listed, except for associated conventional impurities. If the term appears only in a clause of a claim, it limits the claim to the elements expressly listed in that clause; elements recited in other clauses are not excluded from the overall claim.

[0023] The following provides a detailed description of a frequency-domain enhanced cross-modal pedestrian re-identification method, system, device, and medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, conventional conditions in the art or conditions recommended by the manufacturer shall apply. Instruments used in the embodiments of the present invention whose manufacturers are not specified are all conventional products that can be purchased commercially.

[0024] Example 1 This invention provides a frequency-domain enhanced cross-modal person re-identification method, such as... Figure 1 As shown, it mainly includes the following steps: Step 1: Frequency-based data augmentation.

[0025] In this embodiment of the invention, the images of each modality are transformed to Fourier space, the amplitude components of the images of different modalities are exchanged in Fourier space, and then transformed back to the image domain to obtain the enhanced images of each modality; wherein, the modalities include: visible light mode and infrared mode.

[0026] In this embodiment of the invention, a preferred implementation for exchanging the amplitude components of images of different modalities in Fourier space is as follows: (1) Image of the current mode through two-dimensional Fourier transform Transform to Fourier space to obtain the corresponding frequency representation. The frequency representation is decomposed into amplitude components. With phase components .

[0027] (2) Image of another modality The amplitude component is denoted as The image of the current mode is obtained by exchanging the amplitude components using the following formula. The amplitude components after the swap: ; in, Image of the current mode The amplitude components after the exchange These represent frequency coordinates, corresponding to the horizontal direction u and the vertical direction v, respectively. For element-wise multiplication; M is a binary mask used to control the frequency of the swapped amplitude components, set... That is: the horizontal index in the binary mask M starts from... to and vertical index to Set all positions within the rectangular area to 1, and set all other positions to 0. The range factor is represented by H and W, which represent the height and width of the image, respectively.

[0028] Then, combine the swapped amplitude components. With phase components To obtain a new frequency representation The new frequency representation is achieved through inverse Fourier transform. Transform to the image domain to obtain the enhanced image. If the current mode is the visible light mode, then from the enhanced image... A channel is randomly selected and copied multiple times to obtain the final enhanced image; if it is an infrared mode, the enhanced image is... This is the final enhanced image.

[0029] Step 2: Spatial frequency attention processing.

[0030] In this embodiment of the invention, for each modality, after extracting the feature map from the corresponding enhanced image, spatial frequency attention operation is performed in Fourier space to extract frequency domain cues and fuse them to obtain the fused feature map.

[0031] The preferred implementation method for this step is as follows: (1) For feature map X, three feature maps Q, K and V in spatial frequency attention are obtained through convolution operation, where Q is the query, K is the key and V is the value.

[0032] (2) Perform two-dimensional Fourier transforms on feature maps Q and K in the spatial dimension to obtain the corresponding frequency domain representations. and .

[0033] (3) Represent the frequency domain Decomposed into two components and , represented as: ; in, For element-wise multiplication, and All are binary masks. ,set up That is, a binary mask Horizontal index from to and vertical index to Set all positions within the rectangular area to 1, and set all other positions to 0. Indicates range factor, and The height and width are represented in the frequency domain.

[0034] (4) For two components and The signals are modulated separately to extract frequency domain cues, and then fused with the feature map V after inverse two-dimensional Fourier transform to obtain the fused feature map. , represented as: ; ; ; in, For components Frequency domain clues obtained after modulation For components Frequency domain clues obtained after modulation and All of these are information obtained by performing an inverse two-dimensional Fourier transform. This represents the inverse two-dimensional Fourier transform. and This represents two batch normalization layers, where A represents the combination of the two batch normalization layers. and Output the resulting feature map; W1 is the weight parameter.

[0035] Step 3: Channel frequency attention processing.

[0036] In this embodiment of the invention, for each mode, based on the fused feature map, channel frequency attention operation is performed in Fourier space to extract frequency domain cues and fuse them to obtain the final enhanced feature map.

[0037] The preferred implementation method for this step is as follows: (1) The fused feature map is denoted as The feature map is obtained after convolution processing. Then utilize feature maps Obtaining three feature maps in channel frequency attention , , ,in, For query, As key, The value is...

[0038] (2) For feature maps With feature map Performing a one-dimensional Fourier transform along the channel dimension yields the corresponding frequency domain representation. and .

[0039] (3) Using frequency domain representation and Frequency domain cues are extracted and subjected to inverse one-dimensional Fourier transform, then compared with the feature map. The feature maps are then fused to obtain the final enhanced feature map. , represented as: ; ; in, Representation using the frequency domain and Extracted frequency domain clues, This is the inverse one-dimensional Fourier transform. For element-wise multiplication, GAP is a batch normalization layer, and GAP is a global average pooling layer. The feature map output by the global average pooling layer; These are the weight parameters.

[0040] Step 4: Frequency enhancement and optimization.

[0041] In this embodiment of the invention, for each mode, based on the final enhanced feature map, aggregated features are obtained through feature aggregation, and based on the final enhanced feature map, corresponding enhanced frequency features are obtained through frequency calculation and enhancement processing.

[0042] In this embodiment of the invention, the step of obtaining aggregated features through feature aggregation based on the final enhanced feature map, and obtaining corresponding enhanced frequency features through frequency calculation and enhancement processing based on the final enhanced feature map, includes: For the image of the current modality The corresponding final enhanced feature map is denoted as The output feature map F is obtained through convolution processing.

[0043] The output feature map F is aggregated using a global average pooling layer to obtain the aggregated feature f: ; And, the frequency representation is obtained through frequency calculation: ; in, Represents a two-dimensional Fourier transform. The frequency representation is obtained from the two-dimensional Fourier transform, and GAP is a global average pooling layer. This represents the frequency representation after global average pooling.

[0044] Introducing second-order moment information to enhance frequency representation and obtain enhanced frequency features , represented as: ; in, This represents the second-order moment information.

[0045] In this embodiment of the invention, step 1 is executed by a frequency-based data augmentation module, step 2 by a spatial frequency attention module, step 3 by a channel frequency attention module, and step 4 by a frequency enhancement optimization module. These four modules, together with the residual network, form a frequency-aware model. The frequency-aware model is pre-trained. Specifically, the frequency enhancement optimization module constructs a training loss based on aggregated features and enhanced frequency features, as well as identity prediction results obtained based on aggregated features and enhanced frequency features, and then uses the training loss to pre-train the frequency-aware model.

[0046] The above-mentioned schemes involved in the embodiments of the present invention are typically implemented in conjunction with related residual networks. Spatial frequency attention modules and channel frequency attention modules are inserted into the residual network, located sequentially at the output ends of intermediate stages K1 and K2, respectively. A data augmentation module is located at the front end of the residual network, and a frequency augmentation optimization module is located at the back end. An enhanced image is input to the residual network, where the output of intermediate stage K1 is the aforementioned feature map X, which serves as the input to the spatial frequency attention module. The spatial frequency attention module outputs the fused feature map. As input to intermediate stage K2, the feature map is processed through deep convolution in intermediate stage K2. And it serves as the input to the channel frequency attention module; the final enhanced feature map output by the channel frequency attention module As the input to the final stage K3, deep convolution processing is performed through the final stage K3 to obtain the output feature map F, which is then used as the input to the frequency enhancement optimization module; among them, the intermediate stages K1 and K2 gradually become deeper.

[0047] For example, a ResNet50 network (a residual network with 50 weight layers) can be selected, which contains four convolutional stages: the middle stage K1 is the second convolutional stage, the middle stage K2 is the third convolutional stage, and the final stage K3 is the fourth convolutional stage.

[0048] Step 5: For each modality, use aggregated features and enhanced frequency features for pedestrian re-identification.

[0049] In the existing pedestrian re-identification process, the image features (in this invention, aggregated features and enhanced frequency features are concatenated) are used for retrieval, and the resulting image list is the pedestrian re-identification result. Considering that the retrieval process involved in this part can be implemented with reference to conventional technology, it will not be described in detail.

[0050] In the above-mentioned scheme provided by the embodiments of the present invention, the illumination difference between different modalities at the image level can be reduced by frequency-based data augmentation. Frequency domain cues can be extracted and fused by channel frequency attention and spatial frequency attention at the model level. Then, key frequency domain details can be adaptively captured by frequency domain cross-modal feature augmentation to more effectively extract identity-related contours and details while excluding irrelevant lighting and colors. On this basis, cross-modal pedestrian re-identification in real-world scenarios can be realized.

[0051] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0052] The core of this invention's frequency-domain enhanced cross-modal person re-identification method lies in its proposed frequency-aware model. This model effectively mines and fuses frequency information at the data, model, and optimization levels. It mainly comprises four parts: a frequency-based data augmentation module, a channel frequency attention module, a spatial frequency attention module, and a frequency enhancement optimization module. Images from each modality are input into the frequency-based data augmentation module to reduce illumination differences between the two modalities at the image level. Subsequently, frequency domain cues are extracted and fused through the channel and spatial frequency attention modules at the model level. Finally, the frequency enhancement optimization module performs frequency-domain cross-modal feature learning at the optimization level. All modules mutually benefit each other, jointly learning robust cross-modal features, ultimately achieving accurate cross-modal person re-identification in real-world scenarios. The following section provides a detailed introduction to these four modules.

[0053] 1. Frequency-based data enhancement module.

[0054] In this embodiment of the invention, visible light and infrared modes are primarily considered. Visible light images capture three wavelengths of light, forming an RGB image composed of red, green, and blue channels, while infrared (IR) images consist of a single channel. This significant difference in image quality increases the burden on the feature extraction network. Simple grayscale transformations can effectively minimize the visual differences between RGB and infrared images from a channel (color) perspective. Some data augmentation techniques further refine this strategy through random channel selection. Studies have found that frequent underexposure and overexposure of infrared images still lead to significant differences between them and grayscale images.

[0055] Therefore, the present invention provides Figure 2 The image shown is a frequency-based data augmentation module. This module transforms an image to simulate another modal of illumination, including variations in color and brightness. It considers that color and brightness information are primarily encoded in the amplitude components within the Fourier frequency domain. Given an image... (Can be any mode), apply Discrete Fourier Transform to the spatial dimension of each channel. Transform the image to Fourier space. The resulting frequency representation... Represented as: ; in, Indicates a virtual unit. These represent the frequency coordinates in the frequency domain, corresponding to the frequency components of the image in the horizontal direction u and the vertical direction v, respectively, where e is the natural constant. Let π be the mathematical constant, and H and W be the height and width of the image, respectively. For image middle Pixel value at the location; and They represent The real and imaginary parts. Amplitude components. and phase components It can be represented as: ; ; Where arctan is the arctangent function.

[0056] Amplitude component Quantization frequency coordinates The amplitude, thus preserving brightness and color information. In obtaining and Subsequently, the amplitude components of the RGB and infrared images in the training batch are swapped to minimize the cross-modal illumination gap. However, swapping the entire amplitude can lead to severe artifacts because it is inconsistent with the phase component.

[0057] Therefore, swapping only the low-frequency components of the amplitude can be expressed as: ; in, Image of the current mode The amplitude components after the exchange for Mid-frequency coordinates The amplitude value, For element-wise multiplication; M is a binary mask used to control the frequency of the amplitude components being swapped, ensuring that the swapping only occurs in the low-frequency components. Set all other positions to 0. The range factor, for example, can be set. .

[0058] In obtaining Then, the original phase is preserved to obtain a new frequency domain representation. : .

[0059] Then, using the inverse Fourier transform Frequency domain representation Transforming to the spatial domain (image domain) yields the enhanced image, which can be represented as: ; in, For the enhanced image middle Pixel value at the location, Frequency domain representation middle Frequency value at the location.

[0060] Finally, a channel is randomly selected from the enhanced RGB image and copied multiple times (e.g., three times) to make it closer to the infrared image, resulting in the final enhanced image; while the enhanced infrared image remains unchanged to preserve the color changes caused by the frequency domain swap. During training, data augmentation is applied to each image with a probability of K (e.g., setting K=0.5). In this way, the illumination patterns of the enhanced RGB and infrared images become similar, significantly reducing the difficulty of subsequent cross-modal feature learning.

[0061] 2. Spatial Frequency Attention Module.

[0062] like Figure 3The diagram illustrates a spatial frequency attention module, which facilitates the capture of frequency domain information by performing Fourier transform operations along the spatial dimension. This spatial frequency attention module is integrated into the feature extraction network to further reduce modal discrepancies at the feature level. Specifically, given a feature map... As input, three 1×1 convolutions project the feature maps Q, K, and V. The feature maps Q and K are then transformed in the spatial dimension using a 2D (2D) Fourier transform to obtain their frequency domain representations. In order to capture high-frequency and low-frequency cues separately, Decomposed into two components and They can be represented as: ; in, and All are binary masks. ,set up Set all other positions to 0. The range factor, for example, can be set. To balance the learning of high-frequency and low-frequency components.

[0063] Subsequently, the high-frequency and low-frequency components are modulated to extract frequency domain cues, and then converted back to the spatial domain, which can be expressed as: ; .

[0064] Finally, feature fusion is performed between the frequency and spatial domains to obtain the fused feature map. : .

[0065] Since the meanings of the symbols involved in this section have already been explained one by one in the preceding text, they will not be repeated here.

[0066] 3. Channel frequency attention module.

[0067] To extract richer frequency domain information, this invention proposes the following... Figure 4 The channel frequency attention module shown is designed to facilitate the capture of frequency domain information by performing Fourier transform operations along the channel dimension. Intuitively, spatial frequency attention performs a 2D Fourier transform independently on each channel, while channel frequency attention performs a 1D (one-dimensional) Fourier transform along the channel dimension. Therefore, they explore frequency information from different perspectives and effectively complement each other.

[0068] Similar to the spatial frequency attention module, given a feature map The three feature maps in the channel frequency attention are obtained through convolution operations. , , ,in, For query, As key, The value is obtained through 1D Fourier transform. Perform a transformation to obtain their frequency domain representation in the channel dimension. , It can be formalized as follows: ; Here, z is a one-dimensional frequency domain coordinate, similar to u and v previously. It is the number of channel dimensions. yes The value of the c-coordinate, where c is the channel coordinate ranging from 0 to... ,akin, yes The value of the c-coordinate.

[0069] Subsequently, frequency domain cues are extracted using a method similar to spatial frequency attention, and the final enhanced feature map is obtained. It is represented as: ; .

[0070] Similarly, the meanings of the symbols involved in this section have been explained one by one in the preceding text, so they will not be repeated here.

[0071] The feature map involved here It refers to the fused feature map. The feature map obtained after deep convolution processing.

[0072] 4. Frequency enhancement and optimization module.

[0073] Similarly, for the final enhanced feature map Deep convolution processing is performed to obtain the output feature map F. This invention obtains aggregated features through global average pooling. Existing methods calculate the loss and optimize the model in the spatial domain. This invention proposes introducing frequency domain constraints during model optimization to further enhance cross-modal feature learning.

[0074] like Figure 5 The diagram shown is a schematic of the frequency enhancement optimization module.

[0075] For the image of the current modality The corresponding output feature map is F, and its frequency representation is obtained by the following formula: ; in, This represents a two-dimensional Fourier transform, and GAP is a global average pooling layer.

[0076] To extract more comprehensive frequency information, second-order moment information is introduced to enhance the frequency representation and obtain enhanced frequency features. , represented as: .

[0077] In this embodiment of the invention, the frequency enhancement optimization module constructs a training loss based on aggregated features and enhanced frequency features, as well as the identity prediction results obtained based on aggregated features and enhanced frequency features, and then uses the training loss to pre-train the frequency perception model end-to-end.

[0078] Specifically, the core of this invention is to simultaneously apply loss constraints to both the aggregated features and the enhanced frequency features. For the i-th image, its aggregated features are denoted as... Enhanced frequency characteristics are denoted as The i-th image can be of any modality; using aggregated features and enhanced frequency characteristics Predict the corresponding identity prediction result and , represented as: , , in and There are two classifiers, corresponding to respectively Figure 5 Classifiers in the frequency domain and classifiers in the frequency domain.

[0079] This invention proposes frequency identity loss as a replacement for traditional identity loss. The calculation method is as follows: ; in, Let i be the identity label (known information) of the i-th image. Calculate the expected value of the i-th image.

[0080] To further eliminate modal differences, a frequency KL divergence loss was introduced. The calculation method is as follows: ;

[0081] in, Let || denote the KL divergence, where || is the separator used in KL divergence calculation to indicate the direction of the divergence, and t represents image t, which serves as a positive sample, representing other modal images with the same identity information as the i-th image in the same batch during training. The identity prediction result corresponding to image t. Using the aggregation features of image t Predicted Utilizing the enhanced frequency features of image t Predicted This indicates that the average is calculated, and the subscript refers to the positive sample.

[0082] Intuitively speaking, frequency KL divergence loss Encourage positive samples across modalities to exhibit similar classification probabilities.

[0083] In addition, frequency domain feature loss was proposed. This is used to apply frequency domain characteristic constraints in Euclidean space, and its calculation method is as follows: ; in, Euclidean distance. It is the margin parameter (e.g., (); k represents image k, which serves as a negative sample, representing other modal images in the same batch during training that have different identity information from the i-th image. and The aggregated features and enhanced frequency features corresponding to image k are... This indicates that the average is calculated, and the subscript refers to the negative sample.

[0084] Intuitively speaking, frequency domain feature loss Positive samples across modalities are encouraged to move closer to each other, while negative samples are pushed further apart.

[0085] The training loss is constructed by combining the above three loss parameters. , represented as: .

[0086] Overall, the frequency enhancement optimization module encourages the model to extract discriminative cues from the frequency domain as a supplement to the spatial domain features.

[0087] As previously mentioned, the four modules and the residual network together form the frequency-aware model. The frequency-aware model is trained using the aforementioned training loss until it converges. The specific training process can refer to conventional techniques, and will not be elaborated upon in this invention. In the inference phase, aggregated features and enhanced frequency features are obtained according to the process described above, and the two are concatenated for retrieval.

[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0089] Example 2 This invention also provides a frequency-domain enhanced cross-modal person re-identification system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 6 As shown, the system mainly includes: The frequency-based data augmentation module is used to transform the images of each mode to Fourier space, exchange the amplitude components of the images of different modes in Fourier space, and transform them back to the image domain to obtain the enhanced images of each mode; wherein, the modes include: visible light mode and infrared mode; The spatial frequency attention module is used to extract feature maps from the corresponding enhanced images for each modality, and then perform spatial frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain fused feature maps. The channel frequency attention module is used to perform channel frequency attention operations in Fourier space for each mode based on the fused feature map to extract frequency domain cues and fuse them to obtain the final enhanced feature map. The frequency enhancement and optimization module is used to obtain aggregated features for each mode by feature aggregation based on the final enhanced feature map, and to obtain the corresponding enhanced frequency features by frequency calculation and enhancement processing based on the final enhanced feature map. The pedestrian re-identification unit is used for pedestrian re-identification for each modality by utilizing aggregated features and enhanced frequency features.

[0090] Since the technical details of the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0091] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0092] Example 3 The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0093] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0094] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example: Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc. The output device can be a display terminal; The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0095] Example 4 The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0096] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0097] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A frequency-domain enhanced cross-modal person re-identification method, characterized in that, include: Step 1: Transform the images of each mode to Fourier space, swap the amplitude components of the images of different modes in Fourier space, and transform them back to the image domain to obtain the enhanced images of each mode; wherein, the modes include: visible light mode and infrared mode; Step 2: For each modality, after extracting the feature map from the corresponding enhanced image, perform spatial frequency attention operation in Fourier space to extract frequency domain cues and fuse them to obtain the fused feature map. Step 3: For each mode, based on the fused feature map, perform channel frequency attention operation in Fourier space to extract frequency domain cues and fuse them to obtain the final enhanced feature map. Step 4: For each mode, based on the final enhanced feature map, obtain the aggregated features through feature aggregation, and based on the final enhanced feature map, obtain the corresponding enhanced frequency features through frequency calculation and enhancement processing. Step 5: For each modality, use aggregated features and enhanced frequency features for pedestrian re-identification.

2. The frequency domain enhanced cross-modal person re-identification method according to claim 1, characterized in that, The step of transforming the images of each modality to Fourier space and exchanging the amplitude components of the images of different modalities in Fourier space includes: Image of the current mode through two-dimensional Fourier transform. Transform to Fourier space to obtain the corresponding frequency representation. The frequency representation is decomposed into amplitude components. With phase components ; Image of another modality The amplitude component is denoted as The image of the current mode is obtained by exchanging the amplitude components using the following formula. The amplitude components after the swap: ； in, Image of the current mode The amplitude components after the exchange These represent frequency coordinates, corresponding to the horizontal direction u and the vertical direction v, respectively. For element-wise multiplication; M is a binary mask used to control the frequency of the swapped amplitude components, set... That is: the horizontal index in the binary mask M starts from... to and vertical index to Set all positions within the rectangular area to 1, and set all other positions to 0. The range factor is represented by H and W, which represent the height and width of the image, respectively.

3. The frequency domain enhanced cross-modal person re-identification method according to claim 1, characterized in that, The step of performing spatial frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain a fused feature map includes: For feature map X, three feature maps Q, K, and V in spatial frequency attention are obtained through convolution operation, where Q is the query, K is the key, and V is the value; Performing two-dimensional Fourier transforms on feature maps Q and K in the spatial dimension yields their corresponding frequency domain representations. and ; Frequency domain representation Decomposed into two components and , represented as: ； in, For element-wise multiplication, and All are binary masks. ,set up That is, a binary mask Horizontal index from to and vertical index to Set all positions within the rectangular area to 1, and set all other positions to 0. Indicates range factor, and The height and width are represented in the frequency domain; For two components and The signals are modulated separately to extract frequency domain cues, and then fused with the feature map V after inverse two-dimensional Fourier transform to obtain the fused feature map. , represented as: ； ； ； in, For components Frequency domain clues obtained after modulation For components Frequency domain clues obtained after modulation and All of these are information obtained by performing an inverse two-dimensional Fourier transform. This represents the inverse two-dimensional Fourier transform. and This represents two batch normalization layers, where A represents the combination of the two batch normalization layers. and Output the resulting feature map; W1 is the weight parameter.

4. The frequency domain enhanced cross-modal person re-identification method according to claim 1, characterized in that, The process of performing channel frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain the final enhanced feature map includes: The fused feature map is denoted as The feature map is obtained after convolution processing. Then utilize feature maps Obtaining three feature maps in channel frequency attention , , ,in, For query, As key, Value; For feature maps With feature map Performing a one-dimensional Fourier transform along the channel dimension yields the corresponding frequency domain representation. and ; Then, frequency domain representation is used. and Frequency domain cues are extracted and subjected to inverse one-dimensional Fourier transform, then compared with the feature map. The feature maps are then fused to obtain the final enhanced feature map. , represented as: ； ； in, Representation using the frequency domain and Extracted frequency domain clues, This is the inverse one-dimensional Fourier transform. For element-wise multiplication, GAP is a batch normalization layer, and GAP is a global average pooling layer. This is the feature map output by the global average pooling layer; These are the weight parameters.

5. The frequency domain enhanced cross-modal person re-identification method according to claim 1, characterized in that, The process of obtaining aggregated features through feature aggregation based on the final enhanced feature map, and obtaining corresponding enhanced frequency features through frequency calculation and enhancement processing based on the final enhanced feature map, includes: For the current modality, the corresponding final enhanced feature map is denoted as: The output feature map F is obtained through convolution processing; The output feature map F is aggregated using a global average pooling layer to obtain the aggregated feature f: ; And, the frequency representation is obtained through frequency calculation: ； in, Represents a two-dimensional Fourier transform. The frequency representation is obtained from the two-dimensional Fourier transform, and GAP is a global average pooling layer. This represents the frequency after global average pooling. Introducing second-order moment information to enhance frequency representation and obtain enhanced frequency features , represented as: ； in, This represents the second-order moment information.

6. A frequency-domain enhanced cross-modal person re-identification method according to any one of claims 1 to 5, characterized in that, Step 1 is executed through a frequency-based data augmentation module, step 2 through a spatial frequency attention module, step 3 through a channel frequency attention module, and step 4 through a frequency enhancement optimization module. These four modules, together with the residual network, form a frequency-aware model. The spatial frequency attention module and the channel frequency attention module are inserted into the residual network, the data augmentation module is located at the front end of the residual network, and the frequency enhancement optimization module is located at the back end of the residual network. The enhanced image is input to the residual network, and the feature map X output by the intermediate stage K1 of the residual network serves as the input to the spatial frequency attention module. The fused feature map output by the spatial frequency attention module... The input to intermediate stage K2 is used for convolution processing, and the output feature map is... As input to the channel frequency attention module; the final enhanced feature map output by the channel frequency attention module. As the input to the final stage K3, the feature map is obtained by convolution processing in the final stage K3 and used as the input to the frequency enhancement optimization module. The frequency-aware model is pre-trained; wherein, the frequency enhancement optimization module constructs a training loss based on aggregated features and enhanced frequency features, as well as identity prediction results obtained based on aggregated features and enhanced frequency features, and then uses the training loss to pre-train the frequency-aware model.

7. The frequency domain enhanced cross-modal person re-identification method according to claim 6, characterized in that, The construction of the training loss includes: For the i-th image, its aggregated features are denoted as... Enhanced frequency characteristics are denoted as The i-th image can be of any modality; using aggregated features and enhanced frequency characteristics Predict the corresponding identity prediction result and ; Calculate frequency identity loss , represented as: ； in, For the identity label of the i-th image, Calculate the expected value of the i-th image; Calculate the frequency KL divergence loss , represented as: ； in, This represents the KL divergence, where || is the separator in the KL divergence calculation, and t represents image t, which serves as a positive sample, representing other modal images with the same identity information as the i-th image in the same batch during training. The identity prediction result corresponding to image t. Using the aggregation features of image t Predicted Utilizing the enhanced frequency features of image t Predicted This indicates the average, with the subscript referring to the positive sample. Calculate frequency domain feature loss , represented as: ； in, Euclidean distance. This is the margin parameter; k represents image k, which serves as a negative sample, representing other modal images in the same batch during training that have different identity information from the i-th image. and The aggregated features and enhanced frequency features corresponding to image k are... This indicates averaging, with the subscript indicating a negative sample. The training loss is constructed by combining the above three loss parameters. , represented as: 。 8. A frequency-domain enhanced cross-modal pedestrian re-identification system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: The frequency-based data augmentation module is used to transform the images of each mode to Fourier space, exchange the amplitude components of the images of different modes in Fourier space, and transform them back to the image domain to obtain the enhanced images of each mode; wherein, the modes include: visible light mode and infrared mode; The spatial frequency attention module is used to extract feature maps from the corresponding enhanced images for each modality, and then perform spatial frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain fused feature maps. The channel frequency attention module is used to perform channel frequency attention operations in Fourier space for each mode based on the fused feature map to extract frequency domain cues and fuse them to obtain the final enhanced feature map. The frequency enhancement and optimization module is used to obtain aggregated features for each mode by feature aggregation based on the final enhanced feature map, and to obtain the corresponding enhanced frequency features by frequency calculation and enhancement processing based on the final enhanced feature map. The pedestrian re-identification unit is used for pedestrian re-identification for each modality by utilizing aggregated features and enhanced frequency features.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images