Cross-modal pedestrian re-identification method and system in coal mine low-illumination environment

Abstract

The invention discloses a cross-modal pedestrian re-identification method and system in a coal mine low-illumination environment. The system comprises an edge computing device and a multi-modal camera, and a feature diversity module, a sub-band filtering module, a multi-dimensional interaction module and a diversity center loss function module which are connected in sequence are deployed on the edge computing device; the method comprises the following steps: constructing a feature diversity module to carry out feature expansion and splicing of spatial dimension and channel dimension on a visible light image and an infrared image respectively; a sub-band filtering module is constructed to couple modal specificity residual information existing in spatial domain feature extraction of the shared parameter network; on the basis of a multi-dimensional interaction module, fine-grained feature complementation is realized by constructing a unified cross-modal and intra-modal interaction normal form; and cross-modal feature alignment and discrimination enhancement are realized through a diversity center loss function. According to the method and the system, fine-grained alignment of cross-modal pedestrian features can be realized, and the cross-modal key region matching precision is improved.

Description

Technical Field

[0001] The present invention relates to a cross-modal pedestrian re-identification method and system in a low-illumination environment of a coal mine, belonging to the technical field of pedestrian re-identification. Background Art

[0002] To effectively monitor production activities and supervise personnel behavior, a large number of camera systems are used in underground coal mine operations, enabling real-time transmission of video footage and identification and alarming of personnel violations. However, the narrow and long mine tunnels are often subject to harsh environments such as dim light, uneven illumination, dust, water mist, and single-character personnel. Images often suffer from problems such as overexposure, underexposure, noise, and blur, which seriously affect image quality. This makes traditional pedestrian recognition methods face multiple challenges in mine tunnel environments. At the same time, related personnel recognition research is mainly based on independent viewpoint cameras, which cannot implement cross-viewing area monitoring, and also limits the efficient and safe mining and operation of coal mines.

[0003] Person re-identification (Re-ID) is a key technical task in the field of image retrieval. Its core lies in matching pedestrian images across non-overlapping camera perspectives through feature similarity. This technology is widely used in scenarios such as multimedia, video surveillance, and security systems. Although existing Re-ID methods perform well under normal lighting conditions, in the complex low-light environment of coal mines, it is difficult for visible light cameras to capture detailed appearance features, resulting in a significant decrease in algorithm performance. To address this problem, infrared cameras are introduced into the monitoring system to provide supplementary information for cross-modal matching. However, the inherent spectral differences between visible light and infrared images lead to significant modal differences, resulting in the following defects in existing visible light-infrared cross-modal pedestrian re-identification:

[0004] (1) An effective feature decoupling mechanism has not been established, making it impossible to deeply explore the inherent differences between visible light and infrared images in spectral response and imaging mechanism. This makes it difficult for the shared network to fully utilize modality-specific clues, resulting in a decrease in the comprehensiveness of feature capture.

[0005] (2) The existing attention mechanism relies on basic matrix operations such as element-wise addition or dot multiplication, which results in cross-modal feature interaction only generating fixed-mode fusion output and lacking dynamic adaptability; and the coarse-grained feature aggregation process does not establish a cross-modal discriminative region association mechanism, and cannot effectively separate key identity features from noise areas. Summary of the Invention

[0006] The purpose of the present invention is to provide a cross-modal pedestrian re-identification method and system in a low-light environment in a coal mine. The method and system can achieve fine-grained alignment of cross-modal pedestrian features and improve the accuracy of cross-modal key area matching.

[0007] To achieve the above objectives, the present invention provides a cross-modal pedestrian re-identification method in a low-light environment in a coal mine, comprising the following steps:

[0008] S1. By constructing a feature diversity module, the spatial dimension and channel dimension features of visible light images and infrared images are expanded and spliced respectively.

[0009] S2, by building a sub-band filtering module to couple the modality-specific residual information existing in the shared parameter network in the spatial domain feature extraction;

[0010] S3, based on the multi-dimensional interaction module, achieves fine-grained feature complementarity by building a unified cross-modal and intra-modal interaction paradigm;

[0011] S4. Achieve cross-modal feature alignment and discriminability enhancement through diversity center loss function.

[0012] Furthermore, the specific process of S1 is as follows:

[0013] S1.1, the visible light image x V Expanded to spatial level image x Vg With channel-level image x Vc ; Transform the single channel infrared image x I Expanded to spatial level image x Ig With channel-level image x Ic ;

[0014] S1.2. Feature extraction G of the shared parameter network is used to perform feature mapping on the expanded multi-dimensional data and perform channel splicing operations as follows:

[0015] f 0 =[f Vg ,f Vc ,f Ig ,f Ic ]=G(x Vg ,x Vc ,x Ig ,x Ic );

[0016] Among them, f 0 Represents the joint feature tensor output by the network layer 0, f Vg , f Vc , f Ig , f Ic They are the characteristic components of visible light space dimension, visible light channel dimension, infrared space dimension and infrared channel dimension respectively.

[0017] Furthermore, the specific process of S2 is as follows:

[0018] S2.1. Use discrete horizontal wavelet transform (DHWT) to decompose the input features in the frequency domain. For infrared mode, the input feature map F r In the horizontal direction, they are respectively passed through a low-pass filter With high-pass filter Perform convolution operation to generate low-frequency components With high frequency components Its mathematical expression is:

[0019]

[0020] S2.2, the low-frequency component obtained in S2.1 With high frequency components Perform vertical convolution and two-dimensional downsampling operations with a step size of 2 and decompose into four directional sub-bands:

[0021]

[0022] Among them, the low-frequency sub-band Capturing global structural information, high-frequency sub-band Encodes the detailed texture features in the horizontal, vertical and diagonal directions respectively;

[0023] S2.3. Reconstruct the decomposed subbands through inverse wavelet transform (IDHWT). First, perform zero interpolation upsampling on the four subbands with a step size of 2 to generate upsampled features. where ab∈{ll,lh,hl,hh}):

[0024]

[0025] Then, a synthesis filter is applied in the horizontal direction and Blending low-frequency and high-frequency components:

[0026]

[0027] Then pass through the vertical synthesis filter and Generate reconstruction features

[0028]

[0029] Finally, the same decomposition-reconstruction process is performed on the visible light mode to obtain And generate fusion features through channel splicing operation:

[0030]

[0031] Wavelet transforms decompose coupled contour, edge, and texture features in the spatial domain into independent frequency bands, eliminating modality-specific interference, such as high-frequency texture noise in visible light and low-frequency thermal distribution deviations in infrared. Frequency-domain modulation of the reconstructed features enhances cross-modal shared semantics, such as human posture, significantly improving feature discriminability. This module achieves cross-frequency semantic-structural joint modeling through sub-band decomposition, modulation, and cross-modal fusion.

[0032] Furthermore, the specific process of S3 is as follows:

[0033] S3.1. Construct an attention generator AG to generate cross-modal or unimodal multi-dimensional feature weights. The AG includes a two-dimensional convolution layer, a batch normalization (BN) layer, a QuickGLUE activation function, a 1×1 convolution layer, and a Sigmoid function. Its expression is as follows:

[0034] AG(x i )=σ(Conv(GLUE(BN(Conv(x i )))));

[0035] Among them, σ represents the Sigmoid function, which is used to transform the input features into Perform weight mapping to generate query vector Key Vector and value vector Its expression is:

[0036]

[0037] Where m represents visible light or infrared mode, k represents spatial dimension or channel dimension, where the spatial dimension is 1 and the channel dimension is 2;

[0038] S3.2. For one-way interaction from visible light modality to infrared modality, the query vector of the visible light modality is Bond vector with infrared mode After addition, the input is AG to generate the attention map, which is normalized by Softmax and then combined with the value vector of the infrared mode. Perform matrix multiplication and connect with the weighted residual of the original features. The expression is as follows:

[0039]

[0040] Where ⊙ represents the matrix multiplication operation and γ is a learnable parameter;

[0041] For the reverse interaction from infrared modality to visible light modality, the query vector of infrared modality is The bond vector with the visible light mode After addition, input AG to generate attention map, which is normalized by Softmax and then combined with the value vector of visible light mode. Perform matrix multiplication and connect with the weighted residual of the original features. The expression is as follows:

[0042]

[0043] S3.3. For the interaction of different dimensions in the visible light modality, query the channel dimension vector with the spatial dimension key vector Add to generate the attention map and add it to the spatial dimension value vector Perform residual connection:

[0044]

[0045] Concatenate the interacted features in the original input order to generate a fused tensor

[0046]

[0047] For the interaction of different dimensions within the infrared modality, the channel dimension query vector is also with the spatial dimension key vector Add to generate the attention map and add it to the spatial dimension value vector Perform residual connection:

[0048]

[0049] Concatenate the interacted features in the original input order to generate a fused tensor

[0050]

[0051] Through a unified cross-modal and intra-modal interaction architecture, the potential correlations between different modalities or dimensions are revealed while preserving the consistency of the original information, achieving fine-grained feature complementarity and enhanced discriminability.

[0052] Furthermore, the specific process of the diversity center loss function of S4 to achieve cross-modal feature alignment and discriminability enhancement is as follows:

[0053] S4.1. Diversification center loss l dc This is achieved by calculating fine-grained feature centers, including the feature centers of each modality in the spatial and channel dimensions:

[0054]

[0055] Among them, p represents the pedestrian identity number, K is the total number of sample features of the same pedestrian, Indicates the features of different modalities and dimensions of the p-th pedestrian identity, C Mm Represents the center of the corresponding K features, M represents the visible light modality V or infrared modality I, and m represents the spatial dimension g or channel dimension c;

[0056] S4.2. Characterize the feature distribution of each dimension using Kullback-Leibler divergence, normalize the features and project them onto the unit hypersphere:

[0057]

[0058] Where ‖C‖ is the L2 norm of the eigenvector C, and τ is the temperature coefficient;

[0059] Calculate the intra-modal and inter-modal KL divergence constraint feature distribution:

[0060]

[0061] in, For different modalities, M represents the probability value of the i-th feature of the visible light modality Vis or infrared modality ir in different dimensions, m represents the spatial dimension g or channel dimension c, represents the sum of KL divergences between different dimensions of the same modality, represents the sum of KL divergences between the same dimensions of different modalities, l dc The loss of the sampling center;

[0062] S4.3. Calculate the center loss l of each modal feature c , by narrowing the cross-modal center distance constraint feature distribution and combining the triplet loss l based on Euclidean distance t , the expression is as follows:

[0063]

[0064] Among them, N represents the total number of pedestrians, y i represents the i-th pedestrian label, c yi Indicates the feature center of the same pedestrian label, x a is the anchor sample, x o is a positive sample of the same type, x n is a negative sample of different categories, α is a boundary parameter, represents the square of the Euclidean norm, ‖·‖ + Indicates taking the absolute value after calculating using the Euclidean norm;

[0065] S4.4, the joint loss function is composed of the center correlation loss l q With identity loss l id composition:

[0066] l q=l dc +l c +l t ;

[0067] l=l q +l id .

[0068] The loss function achieves cross-modal feature alignment and discriminability enhancement through the joint constraints of the global embedding space and the structured subspace. c Pulling the cross-modal overall feature center closer to model identity consistency, while the diversified center loss l dc The cross-modal center distribution is pushed further in the subspace to preserve the modality specificity, while the centers of the features of different dimensions in the same modality are brought closer to enhance the structural consistency; the triplet loss l t Cross-modal feature matching is further optimized through distance constraints between anchor samples and positive and negative samples.

[0069] Furthermore, the shared parameter network is a cross-modal shared parameter neural network structure based on ResNet-50. The network structure consists of five stages: Stage 0 to Stage 4. Stage 0 is the independent modality feature adaptation stage, which does not use parameter sharing and is used to extract the underlying modality-specific features of visible light and infrared images, enhancing the perception of each input. Stages 1 to 4 are constructed as unified parameter-shared convolutional layers based on the ResNet-50 backbone network architecture, including residual modules and cross-layer connections, to extract high-level semantic features and achieve cross-modal representation alignment. During training, input images of different modalities are initially processed through their respective convolutional channels in Stage 0 and then input into the shared network of Stages 1–4 to obtain a unified deep feature representation. This balances the preservation of modality differences with the consistent extraction of high-level semantic features, demonstrating good scalability and cross-modal matching capabilities.

[0070] A cross-modal pedestrian re-identification system in a low-light environment in a coal mine includes an edge computing device and a multimodal camera. The edge computing device is equipped with a feature diversity module, a subband filtering module, a multi-dimensional interaction module, and a diversity center loss function module connected in sequence.

[0071] The multimodal camera is used to capture streaming images and transmit the captured streaming images to the edge computing device;

[0072] The edge computing device is used to receive the stream image and process the image through the deployed feature diversity module, sub-band filtering module, multi-dimensional interaction module and diversity center loss function module;

[0073] The feature diversity module is used to enhance the features of the input visible light and infrared images, copying and expanding the infrared single-channel data into three channels, constructing a tensor format consistent with the RGB input, extracting spatial information in the spatial dimension through a standard convolution kernel, and obtaining channel information in the channel dimension through channel normalization;

[0074] The sub-band filtering module uses discrete wavelet transform to map features to the frequency domain space, performs modal-specific noise suppression on high-frequency sub-bands and low-frequency sub-bands through learnable band-selective filters, and reconstructs them to the spatial domain based on inverse transform to preserve the integrity of the information structure;

[0075] The multi-dimensional interaction module is used to perform a channel-space joint attention mechanism within the modality to enhance the fine-grained features of a single modality, while constructing a cross-modal spatial interaction matrix between modalities to achieve bidirectional interaction of specific information of visible light and infrared modalities to enhance feature comprehensiveness;

[0076] The diversity center loss function module is used to calculate the KL divergence in the structured subspace: pushing the feature center distribution of different modalities in the same subspace away to preserve modality specificity, while pulling the feature centers of different dimensions of the same modality closer to enhance structural consistency.

[0077] Furthermore, it also includes a display deployed on the edge computing device, which is used to issue image alarms and voice alarms when abnormal behavior occurs.

[0078] The present invention decomposes visible light and infrared features into high-frequency and low-frequency subbands based on subband filtering and a diversity feature interaction network. It uses a learnable band-selective filter to suppress modality-specific noise and reconstructs spatial domain features through an inverse wavelet transform to preserve the integrity of the information structure. Through a multi-granularity interaction mechanism, spectrally weighted spatial-channel attention is used within the modality to enhance fine-grained identity features. A cross-scale cross-attention approach is constructed between modalities to achieve dynamic alignment of high-frequency textures and low-frequency features, improving the accuracy of cross-modal key region matching. Using a diversity center loss function, the KL divergence is calculated within a structured subspace: the feature centers of different modalities in the same subspace are pushed further away to preserve modality specificity, while the feature centers of different dimensions of the same modality are brought closer together to enhance structural consistency. Combined with the global center loss and the triplet loss, the present invention effectively balances modality invariance and discriminability. The present invention achieves fine-grained alignment of cross-modal pedestrian features and improves the accuracy of cross-modal key region matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a schematic diagram of the module structure of the identification method of the present invention;

[0080] Figure 2 It is a flowchart of the identification method of the present invention;

[0081] Figure 3 Schematic diagram of the working process of the sub-band filtering module of the present invention;

[0082] Figure 4 Schematic diagram of the workflow of the multi-dimensional interaction module of the present invention;

[0083] Figure 5 Schematic diagram of the workflow of the diversity center loss module of the present invention;

[0084] Figure 6 It is a structural diagram of the identification system of the present invention. DETAILED DESCRIPTION

[0085] The present invention will be further described below with reference to the accompanying drawings.

[0086] like Figure 1 As shown in FIG, a cross-modal pedestrian re-identification method in a low-light environment of a coal mine includes the following steps:

[0087] S1. By constructing a feature diversity module, the spatial dimension and channel dimension features of visible light images and infrared images are expanded and spliced respectively.

[0088] S2, by building a sub-band filtering module to couple the modality-specific residual information existing in the shared parameter network in the spatial domain feature extraction;

[0089] S3, based on the multi-dimensional interaction module, achieves fine-grained feature complementarity by building a unified cross-modal and intra-modal interaction paradigm;

[0090] S4. Achieve cross-modal feature alignment and discriminability enhancement through diversity center loss function.

[0091] like Figure 2 As shown, the specific process of S1 is:

[0092] S1.1, the visible light image x V Expanded to spatial level image x Vg With channel-level image x Vc ; Transform the single channel infrared image x I Expanded to spatial level image x Ig With channel-level image x Ic ;

[0093] S1.2. Feature extraction G of the shared parameter network is used to perform feature mapping on the expanded multi-dimensional data and perform channel splicing operations as follows:

[0094] f 0 =[f Vg ,f Vc ,f Ig ,fIc ]=G(x Vg ,x Vc ,x Ig ,x Ic );

[0095] Among them, f 0 Represents the joint feature tensor output by the network layer 0, f Vg , f Vc , f Ig , f Ic They are the characteristic components of visible light space dimension, visible light channel dimension, infrared space dimension and infrared channel dimension respectively.

[0096] like Figure 3 As shown, the specific process of S2 is:

[0097] S2.1. Use discrete horizontal wavelet transform (DHWT) to decompose the input features in the frequency domain. For infrared mode, the input feature map F r In the horizontal direction, they are respectively passed through a low-pass filter With high-pass filter Perform convolution operation to generate low-frequency components With high frequency components Its mathematical expression is:

[0098]

[0099] S2.2, the low-frequency component obtained in S2.1 With high frequency components Perform vertical convolution and two-dimensional downsampling operations with a step size of 2 and decompose into four directional sub-bands:

[0100]

[0101] Among them, the low-frequency sub-band Capturing global structural information, high-frequency sub-band Encodes the detailed texture features in the horizontal, vertical and diagonal directions respectively;

[0102] S2.3. Reconstruct the decomposed subbands through inverse wavelet transform (IDHWT). First, perform zero interpolation upsampling on the four subbands with a step size of 2 to generate upsampled features. where ab∈{ll,lh,hl,hh}):

[0103]

[0104] Then, a synthesis filter is applied in the horizontal direction and Blending low-frequency and high-frequency components:

[0105]

[0106] Then pass through the vertical synthesis filter and Generate reconstruction features

[0107]

[0108] Finally, the same decomposition-reconstruction process is performed on the visible light mode to obtain And generate fusion features through channel splicing operation:

[0109]

[0110] Wavelet transforms decompose coupled contour, edge, and texture features in the spatial domain into independent frequency bands, eliminating modality-specific interference, such as high-frequency texture noise in visible light and low-frequency thermal distribution deviations in infrared. Frequency-domain modulation of the reconstructed features enhances cross-modal shared semantics, such as human posture, significantly improving feature discriminability. This module achieves cross-frequency semantic-structural joint modeling through sub-band decomposition, modulation, and cross-modal fusion.

[0111] like Figure 4 As shown, the specific process of S3 is:

[0112] S3.1. Construct an attention generator AG to generate cross-modal or unimodal multi-dimensional feature weights. The AG includes a two-dimensional convolution layer, a batch normalization (BN) layer, a QuickGLUE activation function, a 1×1 convolution layer, and a Sigmoid function. Its expression is as follows:

[0113] AG(x i )=σ(Conv(GLUE(BN(Conv(x i )))));

[0114] Among them, σ represents the Sigmoid function, which is used to transform the input features into Perform weight mapping to generate query vector Key Vector and value vector Its expression is:

[0115]

[0116] Where m represents visible light or infrared mode, k represents spatial dimension or channel dimension, where the spatial dimension is 1 and the channel dimension is 2;

[0117] S3.2. For one-way interaction from visible light modality to infrared modality, the query vector of the visible light modality is Bond vector with infrared mode After addition, the input is AG to generate the attention map, which is normalized by Softmax and then combined with the value vector of the infrared mode. Perform matrix multiplication and connect with the weighted residual of the original features. The expression is as follows:

[0118]

[0119] Where ⊙ represents the matrix multiplication operation and γ is a learnable parameter;

[0120] For the reverse interaction from infrared modality to visible light modality, the query vector of infrared modality is The bond vector with the visible light mode After addition, input AG to generate attention map, which is normalized by Softmax and then combined with the value vector of visible light mode. Perform matrix multiplication and connect with the weighted residual of the original features. The expression is as follows:

[0121]

[0122]

[0123] S3.3. For the interaction of different dimensions in the visible light modality, query the channel dimension vector with the spatial dimension key vector Add to generate the attention map and add it to the spatial dimension value vector Perform residual connection:

[0124]

[0125] For the interaction of different dimensions within the infrared modality, the channel dimension query vector is also with the spatial dimension key vector Add to generate the attention map and add it to the spatial dimension value vector Perform residual connection:

[0126]

[0127] Concatenate the interacted features in the original input order to generate a fused tensor

[0128]

[0129] Through a unified cross-modal and intra-modal interaction architecture, the potential correlations between different modalities or dimensions are revealed while preserving the consistency of the original information, achieving fine-grained feature complementarity and enhanced discriminability.

[0130] like Figure 5As shown in Figure 2, the specific process of the S4 diversity center loss function to achieve cross-modal feature alignment and discriminability enhancement is as follows:

[0131] S4.1. Diversification center loss l dc This is achieved by calculating fine-grained feature centers, including the feature centers of each modality in the spatial and channel dimensions:

[0132]

[0133] Among them, p represents the pedestrian identity number, K is the total number of sample features of the same pedestrian, Indicates the features of different modalities and dimensions of the p-th pedestrian identity, C mm Represents the center of the corresponding K features, M represents the visible light modality V or the infrared modality I, and m represents the spatial dimension g or the channel dimension c;

[0134] S4.2. Characterize the feature distribution of each dimension using Kullback-Leibler divergence, normalize the features and project them onto the unit hypersphere:

[0135]

[0136] Where ‖C‖ is the L2 norm of the eigenvector C, and τ is the temperature coefficient;

[0137] Calculate the intra-modal and inter-modal KL divergence constraint feature distribution:

[0138]

[0139] in, For different modalities, M represents the probability value of the i-th feature of the visible light modality Vis or infrared modality ir in different dimensions, m represents the spatial dimension g or channel dimension c, represents the sum of KL divergences between different dimensions of the same modality, represents the sum of KL divergences between the same dimensions of different modalities, l dc The loss of the sampling center;

[0140] S4.3. Calculate the center loss l of each modal feature c , by narrowing the cross-modal center distance constraint feature distribution and combining the triplet loss l based on Euclidean distance t , the expression is as follows:

[0141]

[0142] Among them, N represents the total number of pedestrians, y i represents the i-th pedestrian label, Indicates the feature center of the same pedestrian label, x ais the anchor sample, x p is a positive sample of the same type, x n is a negative sample of different categories, α is a boundary parameter, represents the square of the Euclidean norm, ‖·‖ + Indicates taking the absolute value after calculating using the Euclidean norm;

[0143] S4.4, the joint loss function is composed of the center correlation loss l q With identity loss l id composition:

[0144] l q =l dc +l c +l t ;

[0145] l=l q +l id .

[0146] like Figure 6 As shown, a cross-modal pedestrian re-identification system in a low-light environment in a coal mine includes an edge computing device and a multimodal camera. The edge computing device is deployed with a feature diversity module, a sub-band filtering module, a multi-dimensional interaction module, and a diversity center loss function module connected in sequence;

[0147] The multimodal camera is used to capture streaming images and transmit the captured streaming images to the edge computing device;

[0148] The edge computing device is used to receive the stream image and process the image through the deployed feature diversity module, sub-band filtering module, multi-dimensional interaction module and diversity center loss function module;

[0149] The feature diversity module is used to enhance the features of the input visible light and infrared images, copying and expanding the infrared single-channel data into three channels, constructing a tensor format consistent with the RGB input, extracting spatial information in the spatial dimension through a standard convolution kernel, and obtaining channel information in the channel dimension through channel normalization;

[0150] The sub-band filtering module uses discrete wavelet transform to map features to the frequency domain space, performs modal-specific noise suppression on high-frequency sub-bands and low-frequency sub-bands through learnable band-selective filters, and reconstructs them to the spatial domain based on inverse transform to preserve the integrity of the information structure;

[0151] The multi-dimensional interaction module is used to perform a channel-space joint attention mechanism within the modality to enhance the fine-grained features of a single modality, while constructing a cross-modal spatial interaction matrix between modalities to achieve bidirectional interaction of specific information of visible light and infrared modalities to enhance feature comprehensiveness;

[0152] The diversity center loss function module is used to calculate the KL divergence in the structured subspace: pushing the feature center distribution of different modalities in the same subspace away to preserve modality specificity, while pulling the feature centers of different dimensions of the same modality closer to enhance structural consistency.

[0153] Furthermore, it also includes a display deployed on the edge computing device, which is used to issue image alarms and voice alarms when abnormal behavior occurs.

[0154] Example: A ResNet50 network was used as the backbone, pre-trained on ImageNet, and trained and validated on the SYSU-MM01, RegDB, and LLCM datasets, as shown in Tables 1, 2, and 3, respectively. The model was implemented in the PyTorch library and trained on an NVIDIA 4090D GPU. Each image was resized to a resolution of 384×192, and data augmentation was applied via zero padding, multi-level transformations, and random erasure. During training, a mini-batch of four RGB images and four IR images was randomly sampled from six identities, and an SGD optimizer with momentum of 0.9 and weight decay of 5×10⁻4 was used. The learning rate was reduced by a factor of 10 at epochs 30, 90, and 120, for a total of 150 epochs.

[0155] Table 1 Comparison methods of SYSU-MM01 dataset

[0156]

[0157] As shown in Table 1, experimental results on the large-scale SYSU-MM01 dataset demonstrate that the proposed method demonstrates superior performance in both Rank-k and mAP metrics. In the All-Search mode, the proposed method achieves 74.34% Rank-1 accuracy and 70.18% mAP; in the Indoor-Search mode, it achieves 80.91% Rank-1 accuracy and 83.94% mAP, both of which are state-of-the-art compared to existing methods. For example, CSVI utilizes a cross-modal center weight generation module and a segmentation decoder to extract more modality-shared information and implements the interaction mechanism between RGB and infrared images through implicit modeling, resulting in excellent performance. Meanwhile, TMD imposes orthogonal constraints between extracted modality-related and identity-related features, reducing instance-level modality differences and improving feature decoupling capabilities, making the extraction of modality-independent identity features more effective. Compared to CSVI, the proposed method improves both Rank-1 accuracy and mAP by 4% in the All-Search mode; compared to TMD, the improvement is 6%. This is because while both CSVI and TMD focus on the interaction mechanism between modalities, and in particular TMD attempts to decouple different types of features through loss function constraints, neither focuses on the refined modeling of interaction information. In contrast, the present invention focuses on discriminative information in the frequency domain, allowing modal interaction to be performed at a finer granularity.

[0158] Table 2 Comparison methods of RegDB dataset

[0159]

[0160] As shown in Table 2, on the smaller-scale RegDB dataset, the experimental results show that the method proposed in the present invention outperforms the existing technologies in recent years. In the VI retrieval mode, the present invention achieved a Rank-1 of 90.7 and a mAP of 86.1; in the IV retrieval mode, the Rank-1 was 90.6 and the mAP also reached 86.1. Although the performance of the method proposed in the present invention is slightly lower than that of MRCN and CSVI in the VI retrieval mode, it performs better in the other three indicators, showing stronger overall performance. For example, MRCN proposed a modal recovery module (MRM) and a modal compensation module (MCM), which respectively extract modal invariant features and modal specific features from the eliminated information, and at the same time normalize the cross-modal features and compensate for the related features, thereby achieving relatively excellent results. In contrast, the present invention simultaneously considers the diverse interactions between and within modalities, making the extraction of mutual information more comprehensive, thereby more effectively capturing useful information.

[0161] Table 3 Comparison of methods on LLCM dataset

[0162]

[0163] As shown in Table 3, the present invention also achieves good performance on the LLCM dataset.

Claims

1. A cross-modal pedestrian re-identification method in a low-light environment in a coal mine, characterized by: The steps include: S1. By constructing a feature diversity module, the spatial dimension and channel dimension features of visible light images and infrared images are expanded and spliced respectively. S2, by building a sub-band filtering module to couple the modality-specific residual information existing in the shared parameter network in the spatial domain feature extraction; S3, based on the multi-dimensional interaction module, achieves fine-grained feature complementarity by building a unified cross-modal and intra-modal interaction paradigm; S4. Achieve cross-modal feature alignment and discriminability enhancement through diversity center loss function.

2. The cross-modal pedestrian re-identification method in a coal mine low-light environment according to claim 1 is characterized in that: The specific process of S1 is as follows: S1.1, the visible light image x V Expanded to spatial level image x Vg With channel-level image x Vc ; The single-channel infrared image x I Expanded to spatial level image x Ig With channel-level image x Ic ; S1.

2. Feature extraction G of the shared parameter network is used to perform feature mapping on the expanded multi-dimensional data and perform channel splicing operations as follows: f 0 =[f Vg ,f Vc ,f Ig ,f Ic ]=G(x Vg ,x Vc ,x Ig ,x Ic ); Among them, f 0 Represents the joint feature tensor output by the network layer 0, f Vg , f Vc , f Ig , f Ic They are the characteristic components of visible light space dimension, visible light channel dimension, infrared space dimension and infrared channel dimension respectively.

3. The cross-modal pedestrian re-identification method in a coal mine low-light environment according to claim 1 is characterized in that: The specific process of S2 is: S2.

1. Use discrete horizontal wavelet transform (DHWT) to decompose the input features in the frequency domain. For infrared mode, the input feature map F r In the horizontal direction, they are respectively passed through a low-pass filter With high-pass filter Perform convolution operation to generate low-frequency components With high frequency components Its mathematical expression is: S2.2, the low-frequency component obtained in S2.1 With high frequency components Perform vertical convolution and two-dimensional downsampling operations with a step size of 2 and decompose into four directional sub-bands: Among them, the low-frequency sub-band Capturing global structural information, high-frequency sub-band Encodes detailed texture features in horizontal, vertical and diagonal directions respectively; S2.

3. Reconstruct the decomposed subbands through inverse wavelet transform (IDHWT). First, perform zero interpolation upsampling on the four subbands with a step size of 2 to generate upsampled features. where ab∈{ll,lh,hl,hh}): Then, a synthesis filter is applied in the horizontal direction and Blending low-frequency and high-frequency components: Then pass through the vertical synthesis filter and Generate reconstruction features Finally, the same decomposition-reconstruction process is performed on the visible light mode to obtain And generate fusion features through channel splicing operation:

4. The cross-modal pedestrian re-identification method in a coal mine low-light environment according to claim 1 is characterized in that: The specific process of S3 is as follows: S3.

1. Construct an attention generator AG to generate cross-modal or unimodal multi-dimensional feature weights. The AG includes a two-dimensional convolution layer, a batch normalization (BN) layer, a QuickGLUE activation function, a 1×1 convolution layer, and a Sigmoid function. Its expression is as follows: AG(x i )=σ(Conv(GLUE(BN(Conv(x i ))))); Among them, σ represents the Sigmoid function, which is used to transform the input features into Perform weight mapping to generate query vector Key Vector and value vector Its expression is: Where m represents visible light or infrared mode, k represents spatial dimension or channel dimension, where the spatial dimension is 1 and the channel dimension is 2; S3.

2. For one-way interaction from visible light modality to infrared modality, the query vector of the visible light modality is Bond vector with infrared mode After addition, the input is AG to generate the attention map, which is normalized by Softmax and then combined with the value vector of the infrared mode. Perform matrix multiplication and connect with the weighted residual of the original features. The expression is as follows: Where ⊙ represents the matrix multiplication operation and γ is a learnable parameter; For the reverse interaction from infrared modality to visible light modality, the query vector of infrared modality is The bond vector with the visible light mode After addition, input AG to generate attention map, which is normalized by Softmax and then combined with the value vector of visible light mode. Perform matrix multiplication and connect with the weighted residual of the original features. The expression is as follows: S3.

3. For the interaction of different dimensions in the visible light modality, query the channel dimension vector with the spatial dimension key vector Add to generate the attention map and add it to the spatial dimension value vector Perform residual connection: For the interaction of different dimensions within the infrared modality, the channel dimension query vector is also with the spatial dimension key vector Add to generate the attention map and add it to the spatial dimension value vector Perform residual connection: Concatenate the interacted features in the original input order to generate a fused tensor 5. The cross-modal pedestrian re-identification method in a coal mine low-light environment according to claim 1 is characterized in that: The specific process of the S4 diversity center loss function to achieve cross-modal feature alignment and discriminability enhancement is as follows: S4.

1. Diversification center loss l dc This is achieved by calculating fine-grained feature centers, including the feature centers of each modality in the spatial and channel dimensions: Among them, p represents the pedestrian identity number, K is the total number of sample features of the same pedestrian, Indicates the features of different modalities and dimensions of the p-th pedestrian identity, C Mm Represents the center of the corresponding K features, M represents the visible light modality V or infrared modality I, and m represents the spatial dimension g or channel dimension c; S4.

2. Characterize the feature distribution of each dimension using Kullback-Leibler divergence, normalize the features and project them onto the unit hypersphere: Where ‖C‖ is the L2 norm of the eigenvector C, and τ is the temperature coefficient; Calculate the intra-modal and inter-modal KL divergence constraint feature distribution: in, For different modalities, M represents the probability value of the i-th feature of the visible light modality Vis or infrared modality ir in different dimensions, m represents the spatial dimension g or channel dimension c, represents the sum of KL divergences between different dimensions of the same modality, represents the sum of KL divergences between the same dimensions of different modalities, l dc The loss of the sampling center; S4.

3. Calculate the center loss l of each modal feature c , by narrowing the cross-modal center distance constraint feature distribution and combining the triplet loss l based on Euclidean distance t , the expression is as follows: Among them, N represents the total number of pedestrians, y i represents the i-th pedestrian label, Indicates the feature center of the same pedestrian label, x a is the anchor sample, x p is a positive sample of the same type, x n is a negative sample of different categories, α is a boundary parameter, represents the square of the Euclidean norm, ‖·‖ + Indicates taking the absolute value after calculating using the Euclidean norm; S4.4, the joint loss function is composed of the center correlation loss l q With identity loss l id composition: l q ＝l dc +l c +l t ； l＝l q +l id 。 6. The cross-modal pedestrian re-identification method in a coal mine low-light environment according to claim 2 is characterized in that: The shared parameter network is a cross-modal shared parameter neural network structure with Resnet50 as the backbone, and the network structure includes five stages from Stage 0 to Stage 4. Among them, Stage 0 is the independent modality feature adaptation stage, which does not adopt the parameter sharing mechanism and is used to extract the underlying modality-specific features of visible light and infrared images and enhance the perception ability of each input. Stages 1 to 4 are constructed as a unified parameter-sharing convolutional layer, which includes residual modules and cross-layer connections based on the ResNet-50 backbone network architecture, and is used to extract high-level semantic features and achieve cross-modal representation alignment. During the training process, input images of different modalities are initially processed through their respective convolution channels in Stage 0, and then input into the shared network of Stages 1 to 4 to obtain a unified deep feature representation.

7. A cross-modal pedestrian re-identification system in a low-light environment in a coal mine, comprising an edge computing device and a multi-modal camera, characterized in that: The edge computing device is deployed with a feature diversity module, a sub-band filtering module, a multi-dimensional interaction module and a diversity center loss function module connected in sequence; The multimodal camera is used to capture streaming images and transmit the captured streaming images to the edge computing device; The edge computing device is used to receive the stream image and process the image through the deployed feature diversity module, sub-band filtering module, multi-dimensional interaction module and diversity center loss function module; The feature diversity module is used to enhance the features of the input visible light and infrared images, copying and expanding the infrared single-channel data into three channels, constructing a tensor format consistent with the RGB input, extracting spatial information in the spatial dimension through a standard convolution kernel, and obtaining channel information in the channel dimension through channel normalization; The sub-band filtering module uses discrete wavelet transform to map features to the frequency domain space, performs modal-specific noise suppression on high-frequency sub-bands and low-frequency sub-bands through learnable band-selective filters, and reconstructs them to the spatial domain based on inverse transform to preserve the integrity of the information structure; The multi-dimensional interaction module is used to perform a channel-space joint attention mechanism within the modality to enhance the fine-grained features of a single modality, while constructing a cross-modal spatial interaction matrix between modalities to achieve bidirectional interaction of specific information of visible light and infrared modalities to enhance feature comprehensiveness; The diversity center loss function module is used to calculate the KL divergence in the structured subspace: pushing the feature center distribution of different modalities in the same subspace away to preserve modality specificity, while pulling the feature centers of different dimensions of the same modality closer to enhance structural consistency.

8. The cross-modal pedestrian re-identification system in a low-light environment in a coal mine according to claim 7, further comprises a display deployed on an edge computing device for providing image and voice alarms when abnormal behavior occurs.

Images