Cross-modal pedestrian re-identification method, system, device and medium enhanced in frequency domain

Abstract

The application discloses a frequency domain enhanced cross-modal pedestrian re-identification method, system, device and medium, which are corresponding solutions, and in the solutions: through frequency-based data enhancement, the illumination difference between different modalities at the image level can be reduced, through channel frequency attention and spatial frequency attention at the model level, frequency domain clues can be extracted and fused, and then through frequency domain cross-modal feature enhancement, key frequency domain details can be adaptively captured to more effectively extract identity-related contours and details while excluding irrelevant illumination and color, and on this basis, cross-modal pedestrian re-identification in a real scene can be accurately realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of frequency information mining and fusion of images, and in particular to a cross-modality pedestrian re-identification method and system based on frequency domain enhancement, a device and a medium. BACKGROUND

[0002] Person Re-Identification (ReID) aims to match the images of the same person captured by different cameras, which has been widely applied in surveillance systems. In order to achieve 24-hour person re-identification, infrared (IR) cameras are used to capture images of pedestrians at night, which promotes the research of Visible-Infrared Person Re-identification (VI-ReID). Unlike single-modality person re-identification, the large cross-modality gap between RGB (Red Green Blue) images and IR images and the intra-modality variation of IR images pose new challenges to person re-identification research. The modality gap mainly comes from the different lighting conditions of RGB images and IR images. For example, RGB images capture red, green and blue bands of light, providing rich color information, while IR images capture a single band of light. Many studies have explored this. However, it is found that light source differences are often overlooked. The light source of RGB images includes artificial lighting, sunlight and ambient reflected light, resulting in clearer details. In contrast, the infrared images captured by surveillance cameras rely only on the light source of the camera, which often results in underexposed and overexposed images. Therefore, the brightness variation caused by different light sources significantly exacerbates the intra-identity gap, posing a serious challenge to visible-infrared person re-identification.

[0003] Most existing methods operate purely in the spatial domain and do not consider frequency information, so it is difficult to alleviate the modality gap caused by changes in lighting conditions. Wang et al. in “Cross-modality paired-images generation for RGB-infrared person re-identification, AAAI 2020” and Choi et al. in “Hi-CMD: Hierarchical cross-modality disentanglement for visible-infrared person re-identification, CVPR 2022” bridge the appearance gap by generating images from one modality to another modality through a generation method. Due to the lack of cross-modality pose-aligned image pairs, the generated images are difficult to retain accurate identity information.

[0004] Other methods try to reduce the modality gap through data augmentation. Ye et al. in "Channel augmented joint learning for visible-infrared recognition. ICCV 2021" uniformly generate color-irrelevant images by randomly swapping color channels. Liu et al. in "SFANet: A spectrum-aware feature augmentation network for visible-infrared person reidentification. TNNLS 2021" completely replace RGB images with grayscale images for feature learning. Qian et al. in "Visible-infrared person reidentification via patch-mixed cross-modality learning. PR 2025" segment two images of the same person from two modalities into patches and stitch them together to form a new patch. Some other methods aim to reduce the modality gap in the feature space through carefully designed feature enhancement modules and cross-modality loss functions. In general, these spatial-domain cross-modality person re-identification methods have pushed the field in the right direction of narrowing the modality gap, but still cannot fully meet the needs of cross-modality person re-identification in real-world scenarios.

[0005] In view of this, the present application is proposed. SUMMARY

[0006] The purpose of the present application is to provide a frequency domain enhanced cross-modality person re-identification method, system, device and medium, which can adaptively capture key frequency domain details to more effectively extract identity-related contours and details, and accurately realize cross-modality person re-identification in real-world scenarios.

[0007] The purpose of the present application is achieved by the following technical solutions:

[0008] A frequency domain enhanced cross-modality person re-identification method, comprising:

[0009] Step 1, transform the images of each modality into Fourier space respectively, exchange the amplitude components of the images of different modalities in the Fourier space, and transform back to the image domain to obtain enhanced images of each modality; wherein the modalities include: visible light modality and infrared modality;

[0010] Step 2, for each modality, after extracting the feature map from the corresponding enhanced image, perform spatial frequency attention operation in the Fourier space to extract frequency domain clues and fuse them to obtain a fused feature map;

[0011] Step 3, for each modality, based on the fused feature map, channel-frequency attention operation is performed in the Fourier space to extract frequency domain clues and fuse them to obtain the final enhanced feature map;

[0012] Step 4, for each modality, based on the final enhanced feature map, aggregated features are obtained through feature aggregation, and based on the final enhanced feature map, corresponding enhanced frequency features are obtained through frequency calculation and enhancement processing;

[0013] Step 5, for each modality, the aggregated features and the enhanced frequency features are used for pedestrian re-identification.

[0014] A cross-modality pedestrian re-identification system based on frequency domain enhancement is used to implement the foregoing method, comprising:

[0015] A frequency-based data enhancement module is used to transform the images of each modality into the Fourier space respectively, exchange the amplitude components of the images of different modalities in the Fourier space, and transform back to the image domain to obtain enhanced images of each modality; wherein the modalities include: visible light modality and infrared modality;

[0016] A spatial frequency attention module is used to, for each modality, after extracting a feature map from the corresponding enhanced image, perform spatial frequency attention operation in the Fourier space to extract frequency domain clues and fuse them to obtain a fused feature map;

[0017] A channel-frequency attention module is used to, for each modality, based on the fused feature map, channel-frequency attention operation is performed in the Fourier space to extract frequency domain clues and fuse them to obtain the final enhanced feature map;

[0018] A frequency enhancement optimization module is used to, for each modality, based on the final enhanced feature map, aggregated features are obtained through feature aggregation, and based on the final enhanced feature map, corresponding enhanced frequency features are obtained through frequency calculation and enhancement processing;

[0019] A pedestrian re-identification unit is used to, for each modality, the aggregated features and the enhanced frequency features are used for pedestrian re-identification.

[0020] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the foregoing method.

[0022] A readable storage medium stores a computer program, when the computer program is executed by a processor to implement the foregoing method.

[0023] As can be seen from the technical solutions provided by the present application, by means of frequency-based data enhancement, the illumination difference between different modalities at the image level can be reduced, by means of channel frequency attention and spatial frequency attention at the model level, frequency domain clues can be extracted and fused, and then by means of frequency domain cross-modality feature enhancement, key frequency domain details can be adaptively captured, so as to more effectively extract the contours and details related to the identity while excluding irrelevant illumination and color, and on this basis, cross-modality pedestrian re-identification in a real scene can be accurately realized. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 A flowchart of a frequency domain enhanced cross-modality pedestrian re-identification method provided by the embodiment of the present application.

[0026] Figure 2 A schematic diagram of a frequency-based data enhancement module provided by the embodiment of the present application.

[0027] Figure 3 A schematic diagram of a spatial frequency attention module provided by the embodiment of the present application.

[0028] Figure 4 A schematic diagram of a channel frequency attention module provided by the embodiment of the present application.

[0029] Figure 5 A schematic diagram of a frequency enhancement optimization module provided by the embodiment of the present application.

[0030] Figure 6 A schematic diagram of a frequency domain enhanced cross-modality pedestrian re-identification system provided by the embodiment of the present application.

[0031] Figure 7 A schematic diagram of a processing device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0033] First, the terms that can be used in this paper are described as follows:

[0034] The terms "comprise", "include", "contain", "have" or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example: including a technical feature element (such as raw materials, components, ingredients, carriers, dosage forms, materials, sizes, parts, components, mechanisms, devices, steps, processes, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or articles, etc.), should be interpreted as not only including the explicitly listed technical feature elements, but also including other technical feature elements that are not explicitly listed but are known in the art.

[0035] The term "consisting of" means excluding any technical feature elements not explicitly listed. If this term is used in the claims, the term will make the claim closed, so that it does not contain technical feature elements other than the explicitly listed technical feature elements, except for conventional impurities related thereto. If the term only appears in a certain clause of the claim, it is only limited to the elements explicitly listed in that clause, and the elements described in other clauses are not excluded from the overall claim.

[0036] A frequency domain enhanced cross-modal pedestrian re-identification method, system, device and medium are described in detail below. The content not described in detail in the embodiments of the present application belongs to the prior art known to those skilled in the art. If no specific conditions are specified in the embodiments of the present application, the conventional conditions or the conditions recommended by the manufacturer are used. If no manufacturer is specified for the instruments used in the embodiments of the present application, they are all conventional products that can be obtained by market purchase.

[0037] Embodiment one

[0038] The embodiments of the present application provide a frequency domain enhanced cross-modal pedestrian re-identification method, as shown in Figure 1 The method mainly includes the following steps:

[0039] Step 1, frequency-based data enhancement.

[0040] In the embodiments of the present application, the images of each modality are respectively transformed into Fourier space, the amplitude components of the images of different modalities are exchanged in the Fourier space, and the enhanced images of each modality are obtained by transforming back to the image domain; wherein the modalities include: visible light modality and infrared modality.

[0041] In the embodiments of the present application, the preferred implementation of exchanging the amplitude components of the images of different modalities in the Fourier space is as follows:

[0042] (1) Transform the image of the current modality into Fourier space by two-dimensional Fourier transform to obtain the corresponding frequency representation split the frequency representation into amplitude components and phase components .

[0043] (2) the amplitude component of the image of another modality is denoted as , the image of the current modality is obtained by exchanging the amplitude components through the following formula the exchanged amplitude component:

[0044] ;

[0045] wherein, the image of the current modality the exchanged amplitude component, denotes the frequency coordinates, corresponding to the horizontal direction u and the vertical direction v, respectively; is an element multiplication; M is a binary mask used to control the frequency of the exchanged amplitude component, set , that is, all positions in the rectangular region of the binary mask M with the horizontal direction index from to and the vertical direction index to are set to 1, and other positions are set to 0, denotes the range factor, and H and W respectively denote the height and width of the image.

[0046] Then, the new frequency representation is obtained by combining the exchanged amplitude component and the phase component ; the new frequency representation is transformed to the image domain through inverse Fourier transform to obtain an enhanced image . If the current modality is a visible light modality, one channel is randomly selected from the enhanced image and copied multiple times to obtain the final enhanced image; if it is an infrared modality, the enhanced image is the final enhanced image.

[0047] Step 2, spatial frequency attention processing.

[0048] In the embodiments of the present application, for each modality, after extracting the feature map from the corresponding enhanced image, spatial frequency attention operation is performed in the Fourier space to extract frequency domain clues and perform fusion to obtain a fused feature map.

[0049] The preferred embodiment of this step is as follows:

[0050] (1) For the feature map X, three feature maps Q, K, V in spatial frequency attention are obtained by convolution operation, wherein Q is query, K is key, and V is value.

[0051] (2) The two-dimensional Fourier transform is performed on the feature map Q and the feature map K in the spatial dimension respectively to obtain the corresponding frequency domain representation and .

[0052] (3) The frequency domain representation is decomposed into two components and , which are represented as:

[0053] ;

[0054] wherein, is an element-wise multiplication, and are binary masks, , set , that is, the binary mask is set to 1 in the rectangular region with horizontal direction index from to and vertical direction index to , and is set to 0 in other positions, denotes a range factor, and are the height and width of the frequency domain representation.

[0055] (4) The two components and are modulated respectively to extract the frequency domain clues, and after inverse two-dimensional Fourier transform, they are fused with the feature map V to obtain the fused feature map , which is represented as:

[0056] ;

[0057] ;

[0058] ;

[0059] wherein, is the frequency domain clue obtained by modulating the component , is the frequency domain clue obtained by modulating the component , and are information obtained by inverse two-dimensional Fourier transform, denotes inverse two-dimensional Fourier transform, and denote two batch normalization layers, A is the combination of the two batch normalization layers and output the obtained feature map; W1 is a weight parameter.

[0060] Step 3, channel frequency attention processing.

[0061] In the embodiment of the present application, for each modality, based on the fused feature map, channel frequency attention operation is performed in the Fourier space to extract frequency domain clues and perform fusion to obtain the final enhanced feature map.

[0062] The preferred embodiment of the present step is as follows:

[0063] (1) Let the fused feature map be , after convolution processing, the feature map is obtained, and then the feature map is used to obtain three feature maps in channel frequency attention , , , wherein is the query, is the key, and is the value.

[0064] (2) For the feature map and the feature map , one-dimensional Fourier transform is performed in the channel dimension to obtain the corresponding frequency domain representations and .

[0065] (3) The frequency domain clues are extracted using the frequency domain representations and , and after inverse one-dimensional Fourier transform, they are fused with the feature map to obtain the final enhanced feature map , which is represented as:

[0066] ;

[0067] ;

[0068] wherein denotes the frequency domain clues extracted using the frequency domain representations and , is the inverse one-dimensional Fourier transform, is the element-wise multiplication, is a batch normalization layer, GAP is a global average pooling layer, is the feature map output by the global average pooling layer; is a weight parameter.

[0069] Step 4, frequency enhancement optimization.

[0070] In the embodiment of the present application, for each modality, based on the final enhanced feature map, the aggregated feature is obtained through feature aggregation, and the corresponding enhanced frequency feature is obtained through frequency calculation and enhancement processing based on the final enhanced feature map.

[0071] In the embodiment of the present application, the based on the final enhanced feature map, the aggregated feature is obtained through feature aggregation, and the corresponding enhanced frequency feature is obtained through frequency calculation and enhancement processing based on the final enhanced feature map includes:

[0072] For the image of the current modality , its corresponding final enhanced feature map is denoted as , the output feature map F is obtained through convolution processing.

[0073] The output feature map F is aggregated by using a global average pooling layer to obtain an aggregated feature f: ;

[0074] And the frequency representation is obtained through frequency calculation:

[0075] ;

[0076] Wherein, represents a two-dimensional Fourier transform, is the frequency representation obtained by two-dimensional Fourier transform, GAP is a global average pooling layer, is the frequency representation after global average pooling.

[0077] The second moment information is introduced to enhance the frequency representation to obtain an enhanced frequency feature , which is denoted as:

[0078] ;

[0079] Wherein, is the second moment information.

[0080] In the embodiment of the present application, the step 1 is executed by a frequency-based data enhancement module, the step 2 is executed by a spatial frequency attention module, the step 3 is executed by a channel frequency attention module, and the step 4 is executed by a frequency enhancement optimization module. The above four modules and the residual network constitute a frequency perception model; the frequency perception model is pre-trained; wherein, the training loss is constructed based on the aggregated feature and the enhanced frequency feature, and the identity prediction result obtained based on the aggregated feature and the enhanced frequency feature by the frequency enhancement optimization module, and the frequency perception model is pre-trained by using the training loss.

[0081] The above scheme related by the embodiment of the present application is generally implemented in cooperation with a related residual network, the spatial frequency attention module and the channel frequency attention module are inserted into the residual network and sequentially located at the output end of the intermediate stage K1 and the output end of the intermediate stage K2, the data enhancement module is located at the front end of the residual network, and the frequency enhancement optimization module is located at the rear end of the residual network; the enhanced image is input into the residual network, the output of the intermediate stage K1 is the aforementioned feature map X, serving as the input of the spatial frequency attention module, the fused feature map output by the spatial frequency attention module serving as the input of the intermediate stage K2, the output feature map F is output through deep convolution processing of the intermediate stage K2. and serving as the input of the channel frequency attention module; the final enhanced feature map output by the channel frequency attention module serving as the input of the last stage K3, the output feature map F is obtained through deep convolution processing of the last stage K3 and serving as the input of the frequency enhancement optimization module; wherein the intermediate stages K1 and K2 are gradually deepened.

[0082] For example, a resnet50 network (a residual network with 50 weight layers) can be selected, the residual network includes four convolution stages, the intermediate stage K1 is the second convolution stage, the intermediate stage K2 is the third convolution stage, and the last stage K3 is the fourth convolution stage.

[0083] Step 5, for each modality, the aggregated feature and the enhanced frequency feature are used for pedestrian re-identification.

[0084] In the existing pedestrian re-identification process, the features of the image (the aggregated feature and the enhanced frequency feature are spliced in the present application) are used for retrieval, and the obtained image list is the pedestrian re-identification result. Considering that the retrieval process involved in this part can be implemented according to conventional techniques, no further description is given.

[0085] In the above scheme provided by the embodiment of the present application, the frequency-based data enhancement can reduce the illumination difference between different modalities at the image level, the channel frequency attention and the spatial frequency attention at the model level can extract and fuse the frequency domain clues, and then the frequency domain cross-modality feature enhancement can adaptively capture key frequency domain details to more effectively extract the contours and details related to the identity and exclude irrelevant illumination and color. On this basis, cross-modality pedestrian re-identification in real scenes can be realized.

[0086] In order to more clearly show the technical solutions provided by the present application and the technical effects produced, the method provided by the embodiment of the present application is described in detail below with specific embodiments.

[0087] The core of this invention's frequency-domain enhanced cross-modal person re-identification method lies in its proposed frequency-aware model. This model effectively mines and fuses frequency information at the data, model, and optimization levels. It mainly comprises four parts: a frequency-based data augmentation module, a channel frequency attention module, a spatial frequency attention module, and a frequency enhancement optimization module. Images from each modality are input into the frequency-based data augmentation module to reduce illumination differences between the two modalities at the image level. Subsequently, frequency domain cues are extracted and fused through the channel and spatial frequency attention modules at the model level. Finally, the frequency enhancement optimization module performs frequency-domain cross-modal feature learning at the optimization level. All modules mutually benefit each other, jointly learning robust cross-modal features, ultimately achieving accurate cross-modal person re-identification in real-world scenarios. The following section provides a detailed introduction to these four modules.

[0088] 1. Frequency-based data enhancement module.

[0089] In this embodiment of the invention, visible light and infrared modes are primarily considered. Visible light images capture three wavelengths of light, forming an RGB image composed of red, green, and blue channels, while infrared (IR) images consist of a single channel. This significant difference in image quality increases the burden on the feature extraction network. Simple grayscale transformations can effectively minimize the visual differences between RGB and infrared images from a channel (color) perspective. Some data augmentation techniques further refine this strategy through random channel selection. Studies have found that frequent underexposure and overexposure of infrared images still lead to significant differences between them and grayscale images.

[0090] Therefore, the present invention provides Figure 2 The image shown is a frequency-based data augmentation module. This module transforms an image to simulate another modal of illumination, including variations in color and brightness. It considers that color and brightness information are primarily encoded in the amplitude components within the Fourier frequency domain. Given an image... (Can be any mode), apply Discrete Fourier Transform to the spatial dimension of each channel. Transform the image to Fourier space. The resulting frequency representation... Represented as:

[0091] ;

[0092] in, Indicates a virtual unit. These represent the frequency coordinates in the frequency domain, corresponding to the frequency components of the image in the horizontal direction u and the vertical direction v, respectively, where e is the natural constant. Let π be the mathematical constant, and H and W be the height and width of the image, respectively. For image In pixel values at the locations; and represent the real and imaginary parts of respectively. The magnitude component and the phase component can be represented as:

[0093] ;

[0094] ;

[0095] where arctan is the inverse tangent function.

[0096] The magnitude component quantizes the frequency coordinates of the magnitude, thus preserving the luminance and color information. After obtaining and , the magnitude components of the RGB and infrared images in the training batch are swapped to minimize the cross-modal lighting gap. However, swapping the entire magnitude can cause severe artifacts because it is not consistent with the phase component.

[0097] Therefore, only the low-frequency components of the magnitude are swapped, which can be represented as:

[0098] ;

[0099] where the image of the current modality the swapped magnitude component, is the magnitude value of the mid-frequency coordinates , is the element-wise multiplication; M is a binary mask used to control the frequency of the swapped magnitude component, ensuring that the swapping only occurs in the low-frequency components, set , and other positions are set to 0, denotes the range factor, which can be set to , for example.

[0100] After obtaining , the original phase is preserved to obtain the new frequency domain representation :

[0101] .

[0102] Then, the inverse Fourier transform is used to transform the frequency domain representation to the spatial domain (image domain) to obtain the enhanced image, which can be represented as:

[0103] ;

[0104] in, For the enhanced image middle Pixel value at the location, Frequency domain representation middle Frequency value at the location.

[0105] Finally, a channel is randomly selected from the enhanced RGB image and copied multiple times (e.g., three times) to make it closer to the infrared image, resulting in the final enhanced image; while the enhanced infrared image remains unchanged to preserve the color changes caused by the frequency domain swap. During training, data augmentation is applied to each image with a probability of K (e.g., setting K=0.5). In this way, the illumination patterns of the enhanced RGB and infrared images become similar, significantly reducing the difficulty of subsequent cross-modal feature learning.

[0106] 2. Spatial Frequency Attention Module.

[0107] like Figure 3 The diagram illustrates a spatial frequency attention module, which facilitates the capture of frequency domain information by performing Fourier transform operations along the spatial dimension. This spatial frequency attention module is integrated into the feature extraction network to further reduce modal discrepancies at the feature level. Specifically, given a feature map... As input, three 1×1 convolutions project the feature maps Q, K, and V. The feature maps Q and K are then transformed in the spatial dimension using a 2D (2D) Fourier transform to obtain their frequency domain representations. In order to capture high-frequency and low-frequency cues separately, Decomposed into two components and They can be represented as:

[0108] ;

[0109] in, and All are binary masks. ,set up Set all other positions to 0. The range factor, for example, can be set. To balance the learning of high-frequency and low-frequency components.

[0110] Subsequently, the high-frequency and low-frequency components are modulated to extract frequency domain cues, and then converted back to the spatial domain, which can be expressed as:

[0111] ;

[0112] .

[0113] Finally, feature fusion is performed between the frequency domain and the spatial domain to obtain the fused feature map :

[0114] .

[0115] It is considered that the meanings of the symbols involved in this part have been explained one by one in the foregoing, and therefore, no further elaboration is made.

[0116] 3. Channel frequency attention module.

[0117] In order to extract more abundant frequency domain information, the present application proposes a channel frequency attention module as shown in Figure 4 , which aims to facilitate the capture of frequency domain information by performing Fourier transform operation along the channel dimension. Intuitively, spatial frequency attention performs 2D Fourier transform on each channel independently, while channel frequency attention performs 1D (1-dimensional) Fourier transform along the channel dimension. Therefore, they explore frequency information from different angles, effectively complementing each other.

[0118] Similar to the spatial frequency attention module, given a feature map , three feature maps in channel frequency attention are obtained through convolution operation , , , where, is the query, is the key, is the value; by 1D Fourier transform , their frequency domain representations are obtained in the channel dimension, which can be formulated as:

[0119] ;

[0120] where z is the one-dimensional frequency domain coordinate, similar to the previous u, v, is the number of channel dimensions, is the value of the c coordinate of , c is the channel coordinate range from 0 to , and similarly, is the value of the c coordinate of .

[0121] After that, frequency domain clues are extracted using a method similar to spatial frequency attention, and the final enhanced feature map is obtained, which is represented as:

[0122] ;

[0123] .

[0124] Similarly, the meanings of the symbols involved in this part have been explained one by one in the foregoing, and thus will not be described again.

[0125] The feature map involved here is the feature map obtained after deep convolution processing on the fused feature map .

[0126] 4, frequency enhancement optimization module.

[0127] Similarly, the final enhanced feature map is subjected to deep convolution processing to obtain an output feature map F, and the present application obtains aggregated features by global average pooling. The existing method calculates the loss and optimizes the model in the spatial domain. The present application suggests that introducing frequency domain constraints in the model optimization process can further enhance cross-modal feature learning.

[0128] As shown in Figure 5 , it is a schematic diagram of the frequency enhancement optimization module.

[0129] For the image of the current modality , the corresponding output feature map is F, and the frequency representation is obtained by the following formula:

[0130] ;

[0131] Among them, represents a two-dimensional Fourier transform, and GAP is a global average pooling layer.

[0132] In order to extract more comprehensive frequency information, the second moment information is introduced to enhance the frequency representation, and the enhanced frequency feature is obtained, which is represented as:

[0133] .

[0134] In the embodiment of the present application, the frequency enhancement optimization module is based on the aggregated features and the enhanced frequency features, and the identity prediction results obtained based on the aggregated features and the enhanced frequency features to construct a training loss, and then the frequency-aware model is pre-trained end-to-end using the training loss.

[0135] Specifically, the core of the present application is to simultaneously impose loss constraints on the aggregated features and the enhanced frequency features. For the i-th image, the aggregated features are denoted as , the enhanced frequency features are denoted as , and the i-th image is of any modality; the corresponding identity prediction result is predicted using the aggregated features and the enhanced frequency features . With , denoted as: , , wherein and are two classifiers, corresponding to the classifier in Figure 5 , the frequency domain classifier in turn.

[0136] The present application proposes a frequency identity loss instead of the traditional identity loss, and the frequency identity loss is calculated as:

[0137] ;

[0138] wherein, is the identity label of the i-th image (known information), the mathematical expectation is obtained for the i-th image.

[0139] In order to further eliminate the modal difference, a frequency KL divergence loss is introduced, and the calculation method is as follows:

[0140] ;

[0141] wherein, denotes the KL divergence, || is the separator when calculating the KL divergence, which is used to indicate the direction of the divergence, t represents the image t, which is the positive sample, and represents the other modal images with the same identity information as the i-th image in the same batch during training, is the identity prediction result corresponding to the image t, is predicted by using the aggregated features of the image t , is predicted by using the enhanced frequency features of the image t , denotes the average, and the subscript refers to the positive sample.

[0142] Intuitively, the frequency KL divergence loss encourages the cross-modal positive samples to show similar classification probabilities.

[0143] In addition, a frequency domain feature loss is also proposed, which is used to impose a frequency domain feature constraint in the Euclidean space, and the calculation method is as follows:

[0144] ;

[0145] wherein, is the Euclidean distance, is a margin parameter (for example, ), k represents the image k, which is a negative sample, representing other modal images with different image identity information in the same batch during training, With The aggregated feature corresponding to the image k and the enhanced frequency feature, represents averaging, and the subscript refers to the negative sample.

[0146] Intuitively, the frequency domain feature loss Encourages positive samples across modalities to be close to each other, and pushes negative samples further away.

[0147] The training loss is constructed by combining the above three losses , which is:

[0148] .

[0149] Overall, the frequency enhancement optimization module encourages the model to extract discriminative clues from the frequency domain as a supplement to the spatial domain features.

[0150] As previously described, the above four modules and the residual network together constitute the frequency-aware model, and the frequency-aware model is trained using the above training loss until the model converges. The specific training process can refer to the conventional technology, and the present application will not be described in detail. In the inference stage, the aggregated feature and the enhanced frequency feature are obtained according to the process introduced in the foregoing, and are connected to be used for retrieval.

[0151] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0152] Embodiment two

[0153] The present application also provides a frequency domain enhanced cross-modal pedestrian re-identification system, which is mainly used to implement the method provided by the foregoing embodiments, as shown in the figure, the system mainly includes: Figure 6

[0154] The frequency-based data enhancement module is used to transform the images of each modality into the Fourier space respectively, exchange the amplitude components of the images of different modalities in the Fourier space, and transform back to the image domain to obtain enhanced images of each modality; wherein the modalities include: visible light modality and infrared modality;

[0155] ​The spatial frequency attention module is used to extract feature maps from the corresponding enhanced images for each modality, and then perform spatial frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain fused feature maps.

[0156] The channel frequency attention module is used to perform channel frequency attention operations in Fourier space for each mode based on the fused feature map to extract frequency domain cues and fuse them to obtain the final enhanced feature map.

[0157] The frequency enhancement and optimization module is used to obtain aggregated features for each mode by feature aggregation based on the final enhanced feature map, and to obtain the corresponding enhanced frequency features by frequency calculation and enhancement processing based on the final enhanced feature map.

[0158] The pedestrian re-identification unit is used for pedestrian re-identification for each modality by utilizing aggregated features and enhanced frequency features.

[0159] Since the technical details of the above system have been described in detail in the previous embodiments, they will not be repeated here.

[0160] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0161] Example 3

[0162] The present invention also provides a processing device, such as Figure 7 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0163] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0164] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0165] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0166] The output device can be a display terminal;

[0167] The memory can be a Random Access Memory (RAM) or a non-volatile memory such as a disk memory.

[0168] Embodiment Four

[0169] The application further provides a readable storage medium storing a computer program, which, when executed by a processor, implements the method provided in the foregoing embodiments.

[0170] The readable storage medium in the embodiments of the application as a computer readable storage medium can be arranged in the foregoing processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a Read-Only Memory (ROM), a magnetic disk or an optical disk and various media capable of storing program codes.

[0171] The above merely describes the preferred embodiments of the application, but the protection scope of the application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the application, which should be covered by the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims. The information disclosed in the background section of the specification is merely intended to deepen the understanding of the general background of the application and should not be regarded as acknowledging or implying in any form that the information constitutes the prior art known by the person skilled in the art.​​​​

Claims

1. A frequency-domain enhanced cross-modal person re-identification method, characterized in that, include: Step 1: Transform the images of each modality to Fourier space, exchange the amplitude components of the images of different modalities in Fourier space, and transform back to the image domain to obtain the enhanced images of each modality; wherein, the modalities include: visible light mode and infrared mode; the step of transforming the images of each modality to Fourier space and exchanging the amplitude components of the images of different modalities in Fourier space includes: performing a two-dimensional Fourier transform on the image of the current modality. Transform to Fourier space to obtain the corresponding frequency representation. The frequency representation is decomposed into amplitude components. With phase components ; to another modality of image The amplitude component is denoted as The image of the current mode is obtained by exchanging the amplitude components using the following formula. The amplitude components after the swap: ;in, Image of the current mode The amplitude components after the exchange These represent frequency coordinates, corresponding to the horizontal direction u and the vertical direction v, respectively. For element-wise multiplication; M is a binary mask used to control the frequency of the swapped amplitude components, set... That is: the horizontal index in the binary mask M starts from... to and vertical index to Set all positions within the rectangular area to 1, and set all other positions to 0. The range factor is represented by H and W, which represent the height and width of the image, respectively. Step 2: For each modality, after extracting the feature map from the corresponding enhanced image, perform spatial frequency attention operation in Fourier space to extract frequency domain cues and fuse them to obtain the fused feature map. Step 3: For each mode, based on the fused feature map, perform channel frequency attention operation in Fourier space to extract frequency domain cues and fuse them to obtain the final enhanced feature map. Step 4: For each mode, based on the final enhanced feature map, obtain the aggregated features through feature aggregation, and based on the final enhanced feature map, obtain the corresponding enhanced frequency features through frequency calculation and enhancement processing. Step 5: For each modality, use aggregated features and enhanced frequency features for pedestrian re-identification.

2. The frequency domain enhanced cross-modal person re-identification method according to claim 1, characterized in that, The step of performing spatial frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain a fused feature map includes: For feature map X, three feature maps Q, K, and V in spatial frequency attention are obtained through convolution operation, where Q is the query, K is the key, and V is the value; Performing two-dimensional Fourier transforms on feature maps Q and K in the spatial dimension yields their corresponding frequency domain representations. and ; Frequency domain representation Decomposed into two components and , is represented as: ； in, For element-wise multiplication, and All are binary masks. ,set up That is, a binary mask Horizontal index from to and vertical index to Set all positions within the rectangular area to 1, and set all other positions to 0. Indicates range factor, and The height and width are represented in the frequency domain; For two components and The signals are modulated separately to extract frequency domain cues, and then fused with the feature map V after inverse two-dimensional Fourier transform to obtain the fused feature map. , is represented as: ； ； ； in, For components Frequency domain clues obtained after modulation For components Frequency domain clues obtained after modulation and All of these are information obtained by performing an inverse two-dimensional Fourier transform. This represents the inverse two-dimensional Fourier transform. and This represents two batch normalization layers, where A represents the combination of the two batch normalization layers. and Output the resulting feature map; W1 is the weight parameter.

3. The frequency domain enhanced cross-modal person re-identification method according to claim 1, characterized in that, The process of performing channel frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain the final enhanced feature map includes: The fused feature map is denoted as The feature map is obtained after convolution processing. Then utilize feature maps Obtaining three feature maps in channel frequency attention , , ,in, For query, As key, Value; For feature maps With feature map Performing a one-dimensional Fourier transform along the channel dimension yields the corresponding frequency domain representation. and ; Then, frequency domain representation is used. and Frequency domain cues are extracted and subjected to inverse one-dimensional Fourier transform, then compared with the feature map. The feature maps are then fused to obtain the final enhanced feature map. , is represented as: ； ； in, Representation using the frequency domain and Extracted frequency domain clues, This is the inverse one-dimensional Fourier transform. For element-wise multiplication, GAP is a batch normalization layer, and GAP is a global average pooling layer. The feature map output by the global average pooling layer; These are the weight parameters.

4. The frequency domain enhanced cross-modal person re-identification method according to claim 1, characterized in that, The process of obtaining aggregated features through feature aggregation based on the final enhanced feature map, and obtaining corresponding enhanced frequency features through frequency calculation and enhancement processing based on the final enhanced feature map, includes: For the current modality, the corresponding final enhanced feature map is denoted as: The output feature map F is obtained through convolution processing; The output feature map F is aggregated using a global average pooling layer to obtain the aggregated feature f: ; And, the frequency representation is obtained through frequency calculation: ； in, Represents a two-dimensional Fourier transform. The frequency representation is obtained from the two-dimensional Fourier transform, and GAP is a global average pooling layer. This represents the frequency after global average pooling. Introducing second-order moment information to enhance frequency representation and obtain enhanced frequency features , is represented as: ； in, This represents the second-order moment information.

5. A frequency-domain enhanced cross-modal person re-identification method according to any one of claims 1 to 4, characterized in that, Step 1 is executed through a frequency-based data augmentation module, step 2 through a spatial frequency attention module, step 3 through a channel frequency attention module, and step 4 through a frequency enhancement optimization module. These four modules, together with the residual network, form a frequency-aware model. The spatial frequency attention module and the channel frequency attention module are inserted into the residual network, the data augmentation module is located at the front end of the residual network, and the frequency enhancement optimization module is located at the back end of the residual network. The enhanced image is input to the residual network, and the feature map X output by the intermediate stage K1 of the residual network serves as the input to the spatial frequency attention module. The fused feature map output by the spatial frequency attention module... The input to intermediate stage K2 is used for convolution processing, and the output feature map is... As input to the channel frequency attention module; the final enhanced feature map output by the channel frequency attention module. As the input to the final stage K3, the feature map is obtained by convolution processing in the final stage K3 and used as the input to the frequency enhancement optimization module. The frequency-aware model is pre-trained; wherein, the frequency enhancement optimization module constructs a training loss based on aggregated features and enhanced frequency features, as well as identity prediction results obtained based on aggregated features and enhanced frequency features, and then uses the training loss to pre-train the frequency-aware model.

6. The frequency domain enhanced cross-modal person re-identification method according to claim 5, characterized in that, The construction of the training loss includes: For the i-th image, its aggregated features are denoted as... Enhanced frequency characteristics are denoted as The i-th image can be of any modality; using aggregated features and enhanced frequency characteristics Predict the corresponding identity prediction result and ; Calculate frequency identity loss , is represented as: ； in, For the identity label of the i-th image, Calculate the expected value of the i-th image; Calculate the frequency KL divergence loss , is represented as: ； in, This represents the KL divergence, where || is the separator in the KL divergence calculation, and t represents image t, which serves as a positive sample, representing other modal images with the same identity information as the i-th image in the same batch during training. The identity prediction result corresponding to image t. Using the aggregation features of image t Predicted results Utilizing the enhanced frequency features of image t Predicted results This indicates the average, with the subscript referring to the positive sample. Calculate frequency domain feature loss , is represented as: ； in, Euclidean distance. This is the margin parameter; k represents image k, which serves as a negative sample, representing other modal images in the same batch during training that have different identity information from the i-th image. and The aggregated features and enhanced frequency features corresponding to image k are... This indicates averaging, with the subscript indicating a negative sample. The training loss is constructed by combining the above three loss parameters. , is represented as: 。 7. A frequency-domain enhanced cross-modal person re-identification system, characterized in that, To implement the method according to any one of claims 1 to 6, comprising: The frequency-based data augmentation module is used to transform the images of each mode to Fourier space, exchange the amplitude components of the images of different modes in Fourier space, and transform them back to the image domain to obtain the enhanced images of each mode; wherein, the modes include: visible light mode and infrared mode; The spatial frequency attention module is used to extract feature maps from the corresponding enhanced images for each modality, and then perform spatial frequency attention operations in Fourier space to extract frequency domain cues and fuse them to obtain fused feature maps. The channel frequency attention module is used to perform channel frequency attention operations in Fourier space for each mode based on the fused feature map to extract frequency domain cues and fuse them to obtain the final enhanced feature map. The frequency enhancement and optimization module is used to obtain aggregated features for each mode by feature aggregation based on the final enhanced feature map, and to obtain the corresponding enhanced frequency features by frequency calculation and enhancement processing based on the final enhanced feature map. The pedestrian re-identification unit is used for pedestrian re-identification for each modality by utilizing aggregated features and enhanced frequency features.

8. A processing apparatus, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images