A method and system for cross-modal pedestrian re-identification in low-light environments of coal mines

Abstract

A method and system for cross-modal pedestrian re-identification in low-light environments of coal mines is disclosed. The system includes an edge computing device and a multimodal camera. The edge computing device deploys a feature diversity module, a sub-band filtering module, a multi-dimensional interaction module, and a diversity center loss function module, all connected sequentially. The method includes: expanding and stitching features in the spatial and channel dimensions of visible light and infrared images respectively by constructing the feature diversity module; coupling modality-specific residual information in the spatial domain feature extraction of the shared parameter network by constructing the sub-band filtering module; achieving fine-grained feature complementarity by constructing a unified cross-modal and intra-modal interaction paradigm based on the multi-dimensional interaction module; and achieving cross-modal feature alignment and discriminative enhancement through the diversity center loss function. This method and system can achieve fine-grained alignment of cross-modal pedestrian features and improve the matching accuracy of key cross-modal regions.

Description

Technical Field

[0001] This invention relates to a cross-modal pedestrian re-identification method and system in a low-light environment in a coal mine, belonging to the field of pedestrian re-identification technology. Background Technology

[0002] To effectively monitor production activities and supervise personnel behavior, numerous video surveillance systems are deployed in underground coal mines, enabling real-time video transmission and the identification and alarming of personnel violations. However, the narrow tunnels of mines are often hampered by harsh environments such as low light, uneven illumination, dust, water mist, and the homogeneity of personnel characteristics. This results in images frequently suffering from overexposure, underexposure, noise, and blurring, severely impacting image quality and posing multiple challenges to traditional pedestrian recognition methods in mine tunnel environments. Furthermore, current personnel recognition research primarily relies on single-viewpoint cameras, hindering cross-view monitoring and limiting efficient and safe mining operations.

[0003] Pedestrian re-identification (Re-ID) is a key technical task in the field of image retrieval, its core being the matching of pedestrian images across non-overlapping camera viewpoints through feature similarity. This technology is widely used in multimedia, video surveillance, and security systems. Although existing Re-ID methods perform well under normal lighting conditions, in the complex low-light environment of coal mines, visible light cameras struggle to capture detailed appearance features, leading to a significant drop in algorithm performance. To address this issue, monitoring systems introduce infrared cameras to provide supplementary information for cross-modal matching. However, the inherent spectral differences between visible light and infrared images result in significant modal differences, leading to the following shortcomings in existing visible light-infrared cross-modal pedestrian re-identification methods:

[0004] (1) Without an effective feature decoupling mechanism, it is impossible to deeply explore the inherent differences between visible light and infrared images in spectral response and imaging mechanism, which makes it difficult for the shared network to fully utilize modality-specific cues, resulting in a decrease in the comprehensiveness of feature capture;

[0005] (2) Existing attention mechanisms rely on basic matrix operations such as element-level addition or dot multiplication, which results in cross-modal feature interaction only generating fixed-pattern fusion outputs and lacking dynamic adaptability; and the coarse-grained feature aggregation process does not establish a cross-modal discriminative region association mechanism, which cannot effectively separate key identity features from noisy regions. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for cross-modal pedestrian re-identification in low-light environments in coal mines. This method and system can achieve fine-grained alignment of cross-modal pedestrian features and improve the matching accuracy of cross-modal key areas.

[0007] To achieve the above objectives, the present invention provides a cross-modal pedestrian re-identification method under low-light conditions in coal mines, comprising the following steps:

[0008] S1. By constructing a feature diversity module, the visible light image and infrared image are respectively extended and stitched in the spatial dimension and channel dimension.

[0009] S2. The modality-specific residual information of the shared parameter network in spatial domain feature extraction is coupled by constructing a sub-band filtering module;

[0010] S3. Based on multi-dimensional interaction modules, fine-grained feature complementarity is achieved by constructing a unified cross-modal and intra-modal interaction paradigm.

[0011] S4. Cross-modal feature alignment and discriminative enhancement are achieved through the diversity center loss function.

[0012] Furthermore, the specific process of S1 is as follows:

[0013] S1.1, Transform the visible light image x V Expanded to a spatial image x Vg With channel-level image x Vc ; to convert a single-channel infrared image x I Expanded to a spatial image x Ig With channel-level image x Ic ;

[0014] S1.2. The feature extractor G of the shared parameter network performs feature mapping on the expanded multi-dimensional data and performs channel concatenation as follows:

[0015] f 0 =[f Vg ,f Vc ,f Ig ,f Ic ]=G(x Vg ,x Vc ,x Ig ,x Ic );

[0016] Among them, f 0 f represents the joint feature tensor output by layer 0 of the network. Vg f Vc f Ig f Ic These are the feature components of the visible light spatial dimension, the visible light channel dimension, the infrared spatial dimension, and the infrared channel dimension, respectively.

[0017] Furthermore, the specific process of S2 is as follows:

[0018] S2.1. The discrete horizontal wavelet transform (DHWT) is used to decompose the input features in the frequency domain. For the infrared mode, the input feature map F r Low-pass filters are used in the horizontal direction. With high-pass filter Perform convolution operations to generate low-frequency components. With high frequency components Its mathematical expression is:

[0019]

[0020] S2.2, the low-frequency component obtained in S2.1 With high frequency components Perform vertical convolution and 2D downsampling operations with a stride of 2, decomposing the data into four directional subbands:

[0021]

[0022] Among them, the low-frequency sub-band Capture global structural information, high-frequency subband These encode detailed texture features in the horizontal, vertical, and diagonal directions, respectively.

[0023] S2.3. Reconstruct the decomposed subbands using inverse wavelet transform (IDHWT). First, perform zero-interpolation upsampling on the four subbands with a step size of 2 to generate upsampled features. Where ab∈{ll,lh,hl,hh}):

[0024]

[0025] Then, a synthesis filter is used in the horizontal direction. and Integrating low-frequency and high-frequency components:

[0026]

[0027] Then, through a vertical synthesis filter and Generate reconstructed features

[0028]

[0029] Finally, the same decomposition-reconstruction process is performed on the visible light modes to obtain... And generate fused features through channel splicing operations:

[0030]

[0031] Wavelet transform is used to decompose coupled contour, edge, and texture features in the spatial domain into independent frequency bands, eliminating mode-specific interference, such as the discrepancy between high-frequency texture noise in visible light and low-frequency thermal distribution in infrared light. The reconstructed features are then frequency-domain modulated to enhance cross-modal common semantics, such as human posture, significantly improving feature discriminability. This module achieves cross-frequency level semantic-structural joint modeling through sub-band decomposition, modulation, and cross-modal fusion.

[0032] Furthermore, the specific process of S3 is as follows:

[0033] S3.1 Construct an attention generator (AG) to generate multi-dimensional feature weights for cross-modal or single-modal applications. The AG includes a two-dimensional convolutional layer, a batch normalized (BN) layer, a QuickGLUE activation function, a 1×1 convolutional layer, and a sigmoid function, with the following expression:

[0034] AG(x i )=σ(Conv(GLUE(BN(Conv(x i )))));

[0035] Where σ represents the Sigmoid function, which is applied to the input features through AG. Perform weight mapping to generate query vectors key vector and value vector Its expression is:

[0036]

[0037] Where m represents the visible light or infrared mode, and k represents the spatial dimension or channel dimension, where the spatial dimension is 1 and the channel dimension is 2;

[0038] S3.2 For unidirectional interaction from visible light mode to infrared mode, the query vector of the visible light mode... bond vector with infrared mode After addition, the input AG is used to generate an attention map, which is then normalized by Softmax and compared with the value vector of the infrared mode. Perform matrix multiplication and concatenate it with the weighted residuals of the original features, as shown in the following expression:

[0039]

[0040] Where ⊙ represents matrix multiplication, and γ is a learnable parameter;

[0041] For the reverse interaction from infrared mode to visible mode, the query vector of the infrared mode is... Bond vectors with visible light modes After addition, the input AG is used to generate an attention map, which is then normalized by Softmax and compared with the value vector of the visible light mode. Perform matrix multiplication and concatenate it with the weighted residuals of the original features, as shown in the following expression:

[0042]

[0043] S3.3. For interactions in different dimensions within the visible light mode, query the channel dimension vector. With spatial dimension key vector The attention map is generated by adding the values ​​and then compared with the spatial dimension value vector. Perform residual join:

[0044]

[0045] The interactive features are concatenated in the original input order to generate a fusion tensor.

[0046]

[0047] For interactions across different dimensions within the infrared modality, the channel dimension query vector is also used. With spatial dimension key vector The attention map is generated by adding the values ​​and then compared with the spatial dimension value vector. Perform residual join:

[0048]

[0049] The interactive features are concatenated in the original input order to generate a fusion tensor.

[0050]

[0051] By adopting a unified cross-modal and intra-modal interaction architecture, the potential correlations between different modalities or dimensions are revealed while preserving the consistency of the original information, thereby achieving fine-grained feature complementarity and discriminative enhancement.

[0052] Furthermore, the specific process by which the diversity center loss function of S4 achieves cross-modal feature alignment and discriminative enhancement is as follows:

[0053] S4.1, Diversified Center Loss dc This is achieved by calculating fine-grained feature centers, including feature centers for each modality in both spatial and channel dimensions:

[0054]

[0055] Where p represents the pedestrian's identification number, and K represents the total number of sample features for the same pedestrian. C represents the different modalities and dimensions of the p-th pedestrian's identity. Mm The center of the corresponding K features is represented by M, which represents the visible light mode V or the infrared mode I, and m represents the spatial dimension g or the channel dimension c.

[0056] S4.2. Characterize the feature distribution of each dimension using Kullback-Leibler divergence, normalize the features, and project them onto the unit hypersphere:

[0057]

[0058] Where ||C|| is the L2 norm of the eigenvector C, and τ is the temperature coefficient;

[0059] Calculate the characteristic distribution of intramodal and intermodal KL divergence constraints:

[0060]

[0061] in, For different modes, M represents the probability value of the i-th feature of the visible light mode Vis or the infrared mode ir in different dimensions, and m represents the spatial dimension g or the channel dimension c. This represents the sum of KL divergences between different dimensions of the same mode. The sum of KL divergences between different modalities of the same dimension, l dc For the loss of the sample center;

[0062] S4.3 Calculate the feature center loss for each mode. c By narrowing the cross-modal center distance constraint feature distribution and combining it with Euclidean distance-based triplet loss, t The expression is as follows:

[0063]

[0064] Where N represents the total number of pedestrians, y i c represents the label of the i-th pedestrian. yi x represents the feature center of the same pedestrian label a For anchor samples, x o For positive samples of the same type, x n For negative samples of different classes, α is the boundary parameter. Denotes the square of the Euclidean norm, ‖·‖ + This indicates that the absolute value is taken after calculation using the Euclidean norm.

[0065] S4.4, Joint loss function consists of central correlation loss l q With identity recognition loss l id composition:

[0066] l q=l dc +l c +l t ;

[0067] l = l q +l id .

[0068] The loss function achieves cross-modal feature alignment and discriminative enhancement through joint constraints of the global embedding space and the structured subspace. Global center loss l c To model identity consistency, the cross-modal global feature center is brought closer together, while the diversity center loss is used. dc Within the subspace, the distribution of cross-modal centers is pushed further away to preserve modality specificity, while feature centers of different dimensions within the same modality are brought closer to enhance structural consistency; triplet loss l t Cross-modal feature matching is further optimized by using distance constraints between anchor samples and positive and negative samples.

[0069] Furthermore, the shared parameter network is a cross-modal shared parameter neural network structure with ResNet50 as the backbone. This network structure comprises five stages, from Stage 0 to Stage 4. Stage 0 is an independent modality feature adaptation stage, which does not employ a parameter sharing mechanism and is used to extract low-level modality-specific features from visible light and infrared images to enhance the perception capabilities of their respective inputs. Stages 1 to 4 are constructed as unified parameter-shared convolutional layers, based on the ResNet-50 backbone network architecture, including residual modules and cross-layer connections, used to extract high-level semantic features and achieve cross-modal representation alignment. During training, input images from different modalities undergo preliminary processing in Stage 0 through their respective convolutional channels, and are then input into the shared network of Stages 1–4 to obtain unified deep feature representations. This approach balances the preservation of modal differences with the consistent extraction of high-order semantic features, exhibiting good scalability and cross-modal matching capabilities.

[0070] A cross-modal pedestrian re-identification system for low-light environments in coal mines includes an edge computing device and a multimodal camera. The edge computing device is equipped with a feature diversity module, a sub-band filtering module, a multi-dimensional interaction module, and a diversity center loss function module that are connected in sequence.

[0071] The multimodal camera is used to capture streaming images and transmit the captured images to edge computing devices;

[0072] The edge computing device is used to receive streaming images and process the images through the deployed feature diversity module, sub-band filtering module, multi-dimensional interaction module, and diversity center loss function module.

[0073] The feature diversity module is used to enhance the features of the input visible light and infrared images, copy and expand the single-channel infrared data into three channels, construct a tensor format consistent with the RGB input, extract spatial information through standard convolution kernels in the spatial dimension, and obtain channel information through channel normalization in the channel dimension.

[0074] The subband filtering module uses discrete wavelet transform to map features to the frequency domain space, and uses a learnable frequency band selection filter to suppress mode-specific noise in the high-frequency and low-frequency subbands. It then reconstructs the information structure to the spatial domain based on inverse transform to preserve the integrity of the information structure.

[0075] The multi-dimensional interaction module is used to execute a channel-space joint attention mechanism within a modality to enhance fine-grained features of a single modality, while constructing a cross-modal spatial interaction matrix between modalities to achieve bidirectional interaction of specific information of visible light and infrared modalities to enhance feature comprehensiveness.

[0076] The aforementioned diversity center loss function module is used to calculate KL divergence within a structured subspace: it pushes apart the feature center distributions of different modalities in the same subspace to preserve modality specificity, while bringing feature centers of different dimensions of the same modality closer together to enhance structural consistency.

[0077] Furthermore, it also includes displays deployed on edge computing devices for visual and voice alerts in the event of abnormal behavior.

[0078] This invention decomposes visible light and infrared features into high-frequency and low-frequency subbands based on subband filtering and a diverse feature interaction network. It utilizes a learnable bandgap selector filter to suppress modality-specific noise and reconstructs spatial domain features through inverse wavelet transform to preserve information structure integrity. Through a multi-granularity interaction mechanism, it strengthens fine-grained identity features within a modality using spectral-weighted spatial-channel attention, and constructs cross-scale cross-attention between modalities to achieve dynamic alignment of high-frequency textures and low-frequency features, improving the matching accuracy of key cross-modal regions. Using a diversity center loss function, it calculates KL divergence within a structured subspace: pushing away the feature center distribution of different modalities within the same subspace to preserve modality specificity, while simultaneously bringing feature centers of different dimensions within the same modality closer to enhance structural consistency. Combining global center loss and triplet loss effectively balances modality invariance and discriminativity. This invention achieves fine-grained alignment of cross-modal pedestrian features and improves the matching accuracy of key cross-modal regions. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of the module structure of the identification method of the present invention;

[0080] Figure 2 This is a flowchart of the identification method of the present invention;

[0081] Figure 3 This is a schematic diagram of the working process of the sub-band filtering module of the present invention;

[0082] Figure 4 This is a schematic diagram of the workflow of the multi-dimensional interaction module of the present invention;

[0083] Figure 5 This is a schematic diagram of the workflow of the diversity center loss module of the present invention;

[0084] Figure 6 This is a schematic diagram of the identification system of the present invention. Detailed Implementation

[0085] The invention will now be further described with reference to the accompanying drawings.

[0086] like Figure 1 As shown, a cross-modal pedestrian re-identification method under low-light conditions in coal mines includes the following steps:

[0087] S1. By constructing a feature diversity module, the visible light image and infrared image are respectively extended and stitched in the spatial dimension and channel dimension.

[0088] S2. The modality-specific residual information of the shared parameter network in spatial domain feature extraction is coupled by constructing a sub-band filtering module;

[0089] S3. Based on multi-dimensional interaction modules, fine-grained feature complementarity is achieved by constructing a unified cross-modal and intra-modal interaction paradigm.

[0090] S4. Cross-modal feature alignment and discriminative enhancement are achieved through the diversity center loss function.

[0091] like Figure 2 As shown, the specific process of S1 is as follows:

[0092] S1.1, Transform the visible light image x V Expanded to a spatial image x Vg With channel-level image x Vc ; to convert a single-channel infrared image x I Expanded to a spatial image x Ig With channel-level image x Ic ;

[0093] S1.2. The feature extractor G of the shared parameter network performs feature mapping on the expanded multi-dimensional data and performs channel concatenation as follows:

[0094] f 0 =[f Vg ,f Vc ,f Ig ,fIc ]=G(x Vg ,x Vc ,x Ig ,x Ic );

[0095] Among them, f 0 f represents the joint feature tensor output by layer 0 of the network. Vg f Vc f Ig f Ic These are the feature components of the visible light spatial dimension, the visible light channel dimension, the infrared spatial dimension, and the infrared channel dimension, respectively.

[0096] like Figure 3 As shown, the specific process of S2 is as follows:

[0097] S2.1. The discrete horizontal wavelet transform (DHWT) is used to decompose the input features in the frequency domain. For the infrared mode, the input feature map F r Low-pass filters are used in the horizontal direction. With high-pass filter Perform convolution operations to generate low-frequency components. With high frequency components Its mathematical expression is:

[0098]

[0099] S2.2, the low-frequency component obtained in S2.1 With high frequency components Perform vertical convolution and 2D downsampling operations with a stride of 2, decomposing the data into four directional subbands:

[0100]

[0101] Among them, the low-frequency sub-band Capture global structural information, high-frequency subband These encode detailed texture features in the horizontal, vertical, and diagonal directions, respectively.

[0102] S2.3. Reconstruct the decomposed subbands using inverse wavelet transform (IDHWT). First, perform zero-interpolation upsampling on the four subbands with a step size of 2 to generate upsampled features. Where ab∈{ll,lh,hl,hh}):

[0103]

[0104] Then, a synthesis filter is used in the horizontal direction. and Integrating low-frequency and high-frequency components:

[0105]

[0106] Then, through a vertical synthesis filter and Generate reconstructed features

[0107]

[0108] Finally, the same decomposition-reconstruction process is performed on the visible light modes to obtain... And generate fused features through channel splicing operations:

[0109]

[0110] Wavelet transform is used to decompose coupled contour, edge, and texture features in the spatial domain into independent frequency bands, eliminating mode-specific interference, such as the discrepancy between high-frequency texture noise in visible light and low-frequency thermal distribution in infrared light. The reconstructed features are then frequency-domain modulated to enhance cross-modal common semantics, such as human posture, significantly improving feature discriminability. This module achieves cross-frequency level semantic-structural joint modeling through sub-band decomposition, modulation, and cross-modal fusion.

[0111] like Figure 4 As shown, the specific process of S3 is as follows:

[0112] S3.1 Construct an attention generator (AG) to generate multi-dimensional feature weights for cross-modal or single-modal applications. The AG includes a two-dimensional convolutional layer, a batch normalized (BN) layer, a QuickGLUE activation function, a 1×1 convolutional layer, and a sigmoid function, with the following expression:

[0113] AG(x i )=σ(Conv(GLUE(BN(Conv(x i )))));

[0114] Where σ represents the Sigmoid function, which is applied to the input features through AG. Perform weight mapping to generate query vectors key vector and value vector Its expression is:

[0115]

[0116] Where m represents the visible light or infrared mode, and k represents the spatial dimension or channel dimension, where the spatial dimension is 1 and the channel dimension is 2;

[0117] S3.2 For unidirectional interaction from visible light mode to infrared mode, the query vector of the visible light mode... bond vector with infrared mode After addition, the input AG is used to generate an attention map, which is then normalized by Softmax and compared with the value vector of the infrared mode. Perform matrix multiplication and concatenate it with the weighted residuals of the original features, as shown in the following expression:

[0118]

[0119] Where ⊙ represents matrix multiplication, and γ is a learnable parameter;

[0120] For the reverse interaction from infrared mode to visible mode, the query vector of the infrared mode is... Bond vectors with visible light modes After addition, the input AG is used to generate an attention map, which is then normalized by Softmax and compared with the value vector of the visible light mode. Perform matrix multiplication and concatenate it with the weighted residuals of the original features, as shown in the following expression:

[0121]

[0122]

[0123] S3.3. For interactions in different dimensions within the visible light mode, query the channel dimension vector. With spatial dimension key vector The attention map is generated by adding the values ​​and then compared with the spatial dimension value vector. Perform residual join:

[0124]

[0125] For interactions across different dimensions within the infrared modality, the channel dimension query vector is also used. With spatial dimension key vector The attention map is generated by adding the values ​​and then compared with the spatial dimension value vector. Perform residual join:

[0126]

[0127] The interactive features are concatenated in the original input order to generate a fusion tensor.

[0128]

[0129] By adopting a unified cross-modal and intra-modal interaction architecture, the potential correlations between different modalities or dimensions are revealed while preserving the consistency of the original information, thereby achieving fine-grained feature complementarity and discriminative enhancement.

[0130] like Figure 5As shown, the specific process by which the diversity center loss function of S4 achieves cross-modal feature alignment and discriminative enhancement is as follows:

[0131] S4.1, Diversified Center Loss dc This is achieved by calculating fine-grained feature centers, including feature centers for each modality in both spatial and channel dimensions:

[0132]

[0133] Where p represents the pedestrian's identification number, and K represents the total number of sample features for the same pedestrian. C represents the different modalities and dimensions of the p-th pedestrian's identity. mm The center of the corresponding K features is represented by M, which represents the visible light mode V or the infrared mode I, and m represents the spatial dimension g or the channel dimension c.

[0134] S4.2. Characterize the feature distribution of each dimension using Kullback-Leibler divergence, normalize the features, and project them onto the unit hypersphere:

[0135]

[0136] Where ||C|| is the L2 norm of the eigenvector C, and τ is the temperature coefficient;

[0137] Calculate the characteristic distribution of intramodal and intermodal KL divergence constraints:

[0138]

[0139] in, For different modes, M represents the probability value of the i-th feature of the visible light mode Vis or the infrared mode ir in different dimensions, and m represents the spatial dimension g or the channel dimension c. This represents the sum of KL divergences between different dimensions of the same mode. The sum of KL divergences between different modalities of the same dimension, l dc For the loss of the sample center;

[0140] S4.3 Calculate the feature center loss for each mode. c By narrowing the cross-modal center distance constraint feature distribution and combining it with Euclidean distance-based triplet loss, t The expression is as follows:

[0141]

[0142] Where N represents the total number of pedestrians, y i This represents the label of the i-th pedestrian. x represents the feature center of the same pedestrian label aFor anchor samples, x p For positive samples of the same type, x n For negative samples of different classes, α is the boundary parameter. Denotes the square of the Euclidean norm, ‖·‖ + This indicates that the absolute value is taken after calculation using the Euclidean norm.

[0143] S4.4, Joint loss function consists of central correlation loss l q With identity recognition loss l id composition:

[0144] l q =l dc +l c +l t ;

[0145] l = l q +l id .

[0146] like Figure 6 As shown, a cross-modal pedestrian re-identification system in a low-light environment in a coal mine includes an edge computing device and a multimodal camera. The edge computing device is equipped with a feature diversity module, a sub-band filtering module, a multi-dimensional interaction module, and a diversity center loss function module that are connected in sequence.

[0147] The multimodal camera is used to capture streaming images and transmit the captured images to edge computing devices;

[0148] The edge computing device is used to receive streaming images and process the images through the deployed feature diversity module, sub-band filtering module, multi-dimensional interaction module, and diversity center loss function module.

[0149] The feature diversity module is used to enhance the features of the input visible light and infrared images, copy and expand the single-channel infrared data into three channels, construct a tensor format consistent with the RGB input, extract spatial information through standard convolution kernels in the spatial dimension, and obtain channel information through channel normalization in the channel dimension.

[0150] The subband filtering module uses discrete wavelet transform to map features to the frequency domain space, and uses a learnable frequency band selection filter to suppress mode-specific noise in the high-frequency and low-frequency subbands. It then reconstructs the information structure to the spatial domain based on inverse transform to preserve the integrity of the information structure.

[0151] The multi-dimensional interaction module is used to execute a channel-space joint attention mechanism within a modality to enhance fine-grained features of a single modality, while constructing a cross-modal spatial interaction matrix between modalities to achieve bidirectional interaction of specific information of visible light and infrared modalities to enhance feature comprehensiveness.

[0152] The aforementioned diversity center loss function module is used to calculate KL divergence within a structured subspace: it pushes apart the feature center distributions of different modalities in the same subspace to preserve modality specificity, while bringing feature centers of different dimensions of the same modality closer together to enhance structural consistency.

[0153] Furthermore, it also includes displays deployed on edge computing devices for visual and voice alerts in the event of abnormal behavior.

[0154] Example: ResNet50 was used as the backbone, pre-trained on ImageNet, and trained and validated on the SYSU-MM01, RegDB, and LLCM datasets, as shown in Tables 1, 2, and 3, respectively. The model implementation was based on the PyTorch library and trained on an NVIDIA 4090D GPU. Each image was resized to a resolution of 384×192 and data augmentation was applied through zero-padding, multi-level transformation, and random erasure. During training, four RGB images and four IR images were randomly selected from six identities to construct a mini-batch, and an SGD optimizer with momentum of 0.9 and weight decay of 5×10⁻⁴ was used. The learning rate was reduced by a factor of 10 at iterations 30, 90, and 120, for a total of 150 iterations.

[0155] Table 1 Comparison Methods of the SYSU-MM01 Dataset

[0156]

[0157] As shown in Table 1, experimental results on the large-scale SYSU-MM01 dataset demonstrate that the proposed method exhibits superior performance in both Rank-k and mAP metrics. In All-Search mode, the method achieves 74.34% Rank-1 accuracy and 70.18% mAP; in Indoor-Search mode, it achieves 80.91% Rank-1 accuracy and 83.94% mAP, both of which are best-in-class performances compared to existing technologies. For example, CSVI employs a cross-modal center weight generation module and a segmentation decoder to extract more modality-shared information and implements an interaction mechanism between RGB and infrared images through implicit modeling, thus achieving excellent performance. On the other hand, TMD reduces instance-level modality differences by applying orthogonal constraints between extracted modality-related features and identity-related features, thereby improving feature decoupling capabilities and making modality-independent identity feature extraction more effective. Compared to CSVI, the method improves Rank-1 accuracy and mAP by 4% in All-Search mode and by 6% compared to TMD. This is because, although both CSVI and TMD focus on intermodal interaction mechanisms, especially TMD which attempts to decouple different types of features through loss function constraints, they do not focus on fine-grained modeling of interaction information. In contrast, this invention focuses on discriminative information in the frequency domain, thereby enabling modal interactions to be performed at a more granular level.

[0158] Table 2 Comparison Methods of RegDB Datasets

[0159]

[0160] As shown in Table 2, experimental results on the smaller RegDB dataset demonstrate that the proposed method outperforms existing technologies in recent years. In the VI retrieval mode, the proposed method achieves a Rank-1 of 90.7 and an mAP of 86.1; in the IV retrieval mode, the Rank-1 is 90.6, and the mAP also reaches 86.1. Although the proposed method performs slightly worse than MRCN and CSVI in the VI retrieval mode, it performs better in the other three metrics, showing stronger overall performance. For example, MRCN proposes a Modality Recovery Module (MRM) and a Modality Compensation Module (MCM) to extract modality-invariant and modality-specific features from the removed information, respectively, while normalizing cross-modal features and compensating for related features, thus achieving superior results. In contrast, the proposed method considers diverse interactions between and within modalities, making the extraction of mutual information more comprehensive and thus more effectively capturing useful information.

[0161] Table 3 Comparison Methods of the LLCM Dataset

[0162]

[0163] As shown in Table 3, the present invention also achieved good performance on the LLCM dataset.

Claims

1. A method for cross-modal pedestrian re-identification in a low-light environment in a coal mine, characterized in that, Includes the following steps: S1. By constructing a feature diversity module, the visible light image and infrared image are respectively extended and stitched in the spatial dimension and channel dimension. S2. The modality-specific residual information of the shared parameter network in spatial domain feature extraction is coupled by constructing a sub-band filtering module; S3. Based on multi-dimensional interaction modules, fine-grained feature complementarity is achieved by constructing a unified cross-modal and intra-modal interaction paradigm. S4. Cross-modal feature alignment and discriminative enhancement are achieved through the diversity center loss function; The specific process of S3 is as follows: S3.1 Construct an attention generator (AG) to generate multi-dimensional feature weights for cross-modal or single-modal applications. The AG includes a two-dimensional convolutional layer, a batch normalized (BN) layer, a QuickGLUE activation function, a 1×1 convolutional layer, and a sigmoid function, with the following expression: ； in, This represents the Sigmoid function, which is used to analyze the input features through AG. Perform weight mapping to generate query vectors Key vector and value vector Its expression is: ； Where m represents the visible light or infrared mode, and k represents the spatial dimension or channel dimension, where the spatial dimension is 1 and the channel dimension is 2; S3.2 For unidirectional interaction from visible light mode to infrared mode, the query vector of the visible light mode... bond vector with infrared mode After addition, the input AG is used to generate an attention map, which is then normalized by Softmax and compared with the value vector of the infrared mode. Perform matrix multiplication and concatenate it with the weighted residuals of the original features, as shown in the following expression: ； ； ； Where ⊙ represents matrix multiplication, and γ is a learnable parameter; For the reverse interaction from infrared mode to visible mode, the query vector of the infrared mode is... Bond vectors with visible light modes After addition, the input AG is used to generate an attention map, which is then normalized by Softmax and compared with the value vector of the visible light mode. Perform matrix multiplication and concatenate it with the weighted residuals of the original features, as shown in the following expression: ； ； ； S3.

3. For interactions in different dimensions within the visible light mode, query the channel dimension vector. With spatial dimension key vector The attention map is generated by adding the values ​​and then compared with the spatial dimension value vector. Perform residual join: ； ； ； For interactions across different dimensions within the infrared modality, the channel dimension query vector is also used. With spatial dimension key vector The attention map is generated by adding the values ​​and then compared with the spatial dimension value vector. Perform residual join: ； ； ； The interactive features are concatenated in the original input order to generate a fusion tensor. : 。 2. The cross-modal pedestrian re-identification method under low-light conditions in coal mines according to claim 1, characterized in that, The specific process of S1 is as follows: S1.1, Visible light image Expanded into spatial-level images With channel-level images ; Single-channel infrared image Expanded into spatial-level images With channel-level images ; S1.

2. The feature extractor G of the shared parameter network performs feature mapping on the expanded multi-dimensional data and performs channel concatenation as follows: ； in, This represents the joint feature tensor output by layer 0 of the network. , , , These are the feature components of the visible light spatial dimension, the visible light channel dimension, the infrared spatial dimension, and the infrared channel dimension, respectively.

3. The cross-modal pedestrian re-identification method under low-light conditions in coal mines according to claim 1, characterized in that, The specific process of S2 is as follows: S2.

1. The Discrete Horizontal Wavelet Transform (DHWT) is used to decompose the input features in the frequency domain. For the infrared mode, the input feature map... Low-pass filters are used in the horizontal direction. With high-pass filter Perform convolution operations to generate low-frequency components. With high frequency components Its mathematical expression is: = × ； = × ； S2.2, the low-frequency component obtained in S2.1 With high frequency components Perform vertical convolution and 2D downsampling operations with a stride of 2, decomposing the data into four directional subbands: = ( × )↓2； = ( × )↓2； = ( × )↓2； = ( × )↓2； Among them, the low-frequency sub-band Capture global structural information, high-frequency subband , , These encode detailed texture features in the horizontal, vertical, and diagonal directions, respectively. S2.

3. Reconstruct the decomposed subbands using inverse wavelet transform (IDHWT). First, perform zero-interpolation upsampling on the four subbands with a step size of 2 to generate upsampled features. (where ab∈{ll, lh, hl, hh}). = ↑2； Then, a synthesis filter is used in the horizontal direction. and Integrating low-frequency and high-frequency components: = ； = ； Then, through a vertical synthesis filter and Generate reconstructed features : = ； Finally, the same decomposition-reconstruction process is performed on the visible light modes to obtain... And generate fused features through channel splicing operation: = Concat( , )。 4. The cross-modal pedestrian re-identification method under low-light conditions in coal mines according to claim 1, characterized in that, The specific process by which the diversity center loss function of S4 achieves cross-modal feature alignment and discriminative enhancement is as follows: S4.1, Diversified Center Loss This is achieved by calculating fine-grained feature centers, including feature centers for each modality in both spatial and channel dimensions: ； ； Where p represents the pedestrian's identification number, and K represents the total number of sample features for the same pedestrian. These represent the different modalities and dimensions of the p-th pedestrian's identity. The center of the corresponding K features is represented by M, which represents the visible light mode V or the infrared mode I, and m represents the spatial dimension g or the channel dimension c. S4.

2. Characterize the feature distribution of each dimension using Kullback-Leibler divergence, normalize the features, and project them onto the unit hypersphere: ； in, Let τ be the L2 norm of the eigenvector C, and τ be the temperature coefficient. Calculate the characteristic distribution of intramodal and intermodal KL divergence constraints: ； ； ； in, For different modes, M represents the probability value of the i-th feature of the visible light mode Vis or the infrared mode ir in different dimensions, and m represents the spatial dimension g or the channel dimension c. This represents the sum of KL divergences between different dimensions of the same mode. This represents the sum of KL divergences between different modalities of the same dimension. Loss due to diversification; S4.3 Calculate the feature center loss for each modality. By narrowing the cross-modal center distance constraint feature distribution and combining it with Euclidean distance-based triplet loss, The expression is as follows: ； ； Where N represents the total number of pedestrians, This represents the label of the i-th pedestrian. The feature center representing the same pedestrian label, For anchor samples, Positive samples of the same type For negative samples of different classes, For boundary parameters, Represents the square of the Euclidean norm. This indicates that the absolute value is taken after calculation using the Euclidean norm. S4.4, Joint loss function consists of central correlation loss With identity recognition loss composition: ； 。 5. The cross-modal pedestrian re-identification method under low-light conditions in coal mines according to claim 2, characterized in that, The shared parameter network is a cross-modal shared parameter neural network structure with ResNet50 as the backbone. The network structure includes five stages from Stage 0 to Stage 4. Stage 0 is the independent modality feature adaptation stage, which does not adopt the parameter sharing mechanism and is used to extract low-level modality-specific features of visible light and infrared images to enhance the perception capability of their respective inputs. Stages 1 to 4 are constructed as unified parameter-shared convolutional layers, based on the ResNet-50 backbone network architecture, including residual modules and cross-layer connections, to extract high-level semantic features and achieve cross-modal representation alignment. During training, input images of different modalities are initially processed in Stage 0 through their respective convolutional channels, and then input into the shared network of Stages 1 to 4 to obtain a unified deep feature representation.

6. A system for cross-modal pedestrian re-identification in low-light environments in coal mines as described in any one of claims 1 to 5, comprising an edge computing device and a multimodal camera, characterized in that, The edge computing device is equipped with a feature diversity module, a sub-band filtering module, a multi-dimensional interaction module, and a diversity center loss function module that are connected in sequence. The multimodal camera is used to capture streaming images and transmit them to edge computing devices; The edge computing device is used to receive streaming images and process the images through the deployed feature diversity module, sub-band filtering module, multi-dimensional interaction module, and diversity center loss function module. The feature diversity module is used to enhance the features of the input visible light and infrared images, copy and expand the single-channel infrared data into three channels, construct a tensor format consistent with the RGB input, extract spatial information through standard convolution kernels in the spatial dimension, and obtain channel information through channel normalization in the channel dimension. The subband filtering module uses discrete wavelet transform to map features to the frequency domain space, and uses a learnable frequency band selection filter to suppress mode-specific noise in the high-frequency and low-frequency subbands. It then reconstructs the information structure to the spatial domain based on inverse transform to preserve the integrity of the information structure. The multi-dimensional interaction module is used to execute a channel-space joint attention mechanism within a modality to enhance fine-grained features of a single modality, while constructing a cross-modal spatial interaction matrix between modalities to achieve bidirectional interaction of specific information of visible light and infrared modalities to enhance feature comprehensiveness. The aforementioned diversity center loss function module is used to calculate KL divergence within a structured subspace: it pushes apart the feature center distributions of different modalities in the same subspace to preserve modality specificity, while bringing feature centers of different dimensions of the same modality closer together to enhance structural consistency.

7. The cross-modal pedestrian re-identification system under low illumination conditions in coal mines according to claim 6, characterized in that, It also includes displays deployed on edge computing devices for visual and voice alerts in the event of abnormal behavior.

Images