A cross-modal pedestrian re-identification method and system for deep earth emergency rescue

Abstract

The application discloses a cross-modal pedestrian re-identification method and system for deep earth emergency rescue, utilizes a double-flow neural network to extract multi-layer features of two modes, performs multi-granularity semantic enhancement on the features through a self-adaptive multi-granularity enhancement module, introduces a cross-modal fusion attention module, fuses cross-modal adjacency relations based on a graph structure, and adopts a multi-head graph attention mechanism to improve robustness and generalization capability; in combination with a cross-modal aggregation loss and a joint loss optimization strategy, the discriminability of cross-modal features is effectively improved, a near-real-time digital twin model of an underground space is constructed, a two-dimensional cross-modal Re-ID result is coupled with an incremental three-dimensional reconstruction based on Gaussian splashing, and three-dimensional positioning of an identified person in a digital twin scene is realized. The application can improve pedestrian feature expression capability, reduce modal differences, can expand two-dimensional video information to three-dimensional position information of a person for a rescuer, and significantly improve rescue search positioning efficiency and intuitiveness of command decision.

Description

Technical Field

[0001] This invention belongs to the field of safety monitoring and emergency rescue technology in deep-earth operation environments, specifically relating to a cross-modal pedestrian re-identification method and system for deep-earth emergency rescue. Background Technology

[0002] In deep-earth working environments such as mines, personnel safety monitoring and emergency rescue are core aspects of ensuring safe production. However, deep-earth environments are often accompanied by dangerous factors such as low light, high humidity, and toxic gases. Gas leaks, collapses, water inrushes, fires, or other sudden disasters can easily pose a serious threat to the lives of underground personnel. As mining and engineering progress to deeper levels, underground environmental conditions become increasingly complex, with narrow spaces, winding rescue routes, and poor visibility, significantly increasing the difficulty of personnel search and rescue after an accident. Therefore, how to achieve rapid identification, precise location, continuous tracking, and visualization of personnel in complex deep-earth environments to support rapid decision-making and effective rescue by the command center is a core technological requirement for current deep-earth safety monitoring and emergency rescue.

[0003] Existing underground safety monitoring and early warning systems primarily employ a combination of fixed roadway sensor networks and contact-based personnel monitoring devices. Sensor nodes collect environmental parameters such as methane and carbon monoxide, and data is transmitted to the surface monitoring center via aggregation nodes. Dingqiao Company uses smartwatches to achieve real-time monitoring of workers' health data, including heart rate, blood oxygen, and steps, and combines UWB / BLE technology to achieve centimeter-level underground positioning. However, in practical applications, it has been found that AI health monitoring equipment is time-consuming, has low employee satisfaction, and is ineffective.

[0004] To compensate for the shortcomings of single monitoring methods, cross-modal pedestrian re-identification (VI-ReID) technology is introduced. This technology refers to the task of accurately matching the identity of the same pedestrian between visible light images and infrared light images. Its core objective is to achieve accurate matching and retrieval of the same miner under different lighting conditions and different camera perspectives in a hybrid monitoring network consisting of one or more visible light cameras and one or more infrared cameras (such as thermal imagers). However, existing cross-modal research often only focuses on global or local features and lacks a systematic solution for multi-granularity fusion. At the same time, existing cross-modal re-identification research and applications are still mainly limited to the two-dimensional image domain, lacking the ability to map the recognition results to the underground space for three-dimensional positioning and visualization.

[0005] In summary, the following key issues still exist:

[0006] (1) Existing networks are not good at expressing the characteristics of underground personnel. Current cross-modal ReID research often focuses on one of the global features or local features, and lacks a systematic fusion scheme for multi-level and multi-granular features (such as global tool color / outline, fine-grained safety helmet markings, etc.). In the scenario where underground clothing is highly homogeneous, richer multi-granular features are needed to improve the discrimination ability.

[0007] (2) Insufficient cross-modal feature alignment results in severe information loss or interference from strong light in dark / low-light areas of visible light images; while infrared images can penetrate darkness, they lose color, fine textures (such as tooling numbers and reflective strip details) and facial features (due to dust and thermal imaging characteristics). The dusty environment further blurs the imaging quality of the two modalities, making cross-modal matching extremely difficult. Therefore, a stronger cross-modal fusion mechanism is needed to reduce the VIS-IR gap.

[0008] (3) In rescue scenarios, the two-dimensional recognition results obtained by cross-modal re-identification technology are difficult to accurately reflect the spatial positional relationship between the trapped personnel and the surrounding geometric environment. Due to the lack of a mechanism to accurately associate two-dimensional pixel coordinates with real three-dimensional spatial coordinates, the recognition results cannot intuitively present the spatial distribution of the trapped personnel in a unified "personnel-location" manner in the digital twin system, making it difficult to provide intuitive and effective geometric basis for path planning and resource scheduling. Summary of the Invention

[0009] The purpose of this invention is to provide a cross-modal pedestrian re-identification method and system for deep-earth emergency rescue, which can improve the ability to express pedestrian features, reduce modal differences, and enable rescuers to expand from two-dimensional video information to three-dimensional location information of people, significantly improving the efficiency of rescue search and positioning and the intuitiveness of command and decision-making.

[0010] To achieve the above objectives, this invention provides a cross-modal pedestrian re-identification method for deep-earth emergency rescue, comprising the following steps:

[0011] S1. Data preparation: Obtain pedestrian image datasets in deep earth environments, divide them into visible light image sets and infrared image sets according to modality, and preprocess the visible light image sets and infrared image sets.

[0012] S2. Construct a network model and build a two-stream network structure. Each stream uses a deep convolutional network to extract pedestrian features from the visible light image set and the infrared image set. Each stream contains some shared parameters and some independent parameters, taking into account both modal common features and modal specific features.

[0013] S3. Construct an adaptive multi-granularity enhancement module AMGE and intersperse AMGE in the network model to perform adaptive multi-granularity enhancement on single modality features.

[0014] S4. Construct a cross-modal fusion attention module (CMFA), interspersed in the network model, to interactively fuse the features extracted from visible light flow and infrared light flow, and output a joint feature vector.

[0015] S5. Calculate the loss function, introduce triplet loss for metric learning, and introduce cross-modal fusion loss to improve the stability of cross-modal alignment and aggregation;

[0016] S6. Jointly train the network model, weightedly combine the loss functions in S5 into a total loss function, and optimize the network parameters through backpropagation;

[0017] S7. Feature extraction and matching: After the joint training in S6 is completed, for a given visible light pedestrian query image and infrared image database, the dual-stream network is input to generate its feature vector, and the similarity between the features of the query image and each image in the database is calculated. The matching result is returned to complete the re-identification task.

[0018] S8. Construct a digital twin model, project the identified target personnel's location onto the digital twin 3D model through camera pose estimation and depth mapping, forming an intuitive visual label of "personnel-location". In the digital twin environment, update the spatial location and trajectory of trapped or waiting-to-be-rescued personnel in real time, and achieve seamless coupling of identity recognition and spatial positioning.

[0019] As a further aspect of the present invention: the AMGE processing step includes:

[0020] Multiple parallel convolutional branches with different receptive fields are applied to the input features to generate multiple scale embeddings. The feature maps output by each convolutional branch are first processed by several sets of convolutions, normalization and activation functions, and then the channels are compressed by 1×1 projection to obtain scale-specific feature representations. Then, global average pooling is performed on each branch in the spatial dimension and fed into a small fully connected network to generate branch scores. The outputs of each branch are then weighted and fused to obtain the AMGE output.

[0021] As a further aspect of the present invention: the CMFA processing steps include:

[0022] The AMGE outputs of each stage of visible light and infrared images are used as node features. A cross-modal graph is constructed by setting neighbor relationships. Then, query, key, and value vectors are calculated for each node. The neighbor nodes are weighted and aggregated using a multi-head graph attention mechanism to update the node feature representation. The attention weights are adaptively generated based on the similarity between node features. Finally, cross-modal fusion features are obtained, realizing information fusion within and across modalities.

[0023] As a further aspect of the present invention: the triplet loss in S5 is used to bring positive sample pairs of the same identity closer together and negative sample pairs of different identities further apart; the cross-modal fusion loss is used to promote the joint discriminative power of visible light and infrared modal features.

[0024] As a further aspect of the present invention: constructing a digital twin model, estimating keyframe poses using the SLAM method, and performing dense depth estimation, includes the following steps:

[0025] S8.1. Based on the two-dimensional bounding box detection results returned by the re-identification module, take the center pixel coordinates as the initial location point of the person;

[0026] S8.2. Based on the available depth information types, different strategies are used to recover the 3D camera coordinates corresponding to the pixel;

[0027] S8.3. Transform the 3D camera coordinates obtained in S8.2 into world coordinates using the pose of the current frame;

[0028] S8.4. Evaluate the credibility of the positioning results comprehensively, and define the mapping confidence as a weighted fusion of multi-source confidence;

[0029] S8.5 Store the personnel identifier, world coordinates, confidence level, and timestamp into the scene database and mark them as three-dimensional icons in the digital twin model; if the personnel already have historical trajectories, then perform trajectory association and updates based on spatiotemporal proximity and feature embedding similarity to form a continuous and smooth personnel movement trajectory.

[0030] As a further aspect of the present invention: in S1, a handheld terminal that integrates human-machine-environment information is used to acquire a dataset of pedestrian images in a deep underground environment.

[0031] To achieve the above-mentioned objectives, the present invention also provides a cross-modal pedestrian re-identification system for deep-earth emergency rescue, comprising:

[0032] The data acquisition module is used to acquire visible light and infrared images of pedestrians;

[0033] A dual-stream feature extraction module is connected to the data acquisition module to establish a dual-stream network structure for extracting multi-stage backbone features from visible light images and infrared images respectively.

[0034] The adaptive multi-granularity enhancement module AMGE is connected to the dual-stream feature extraction module to generate multi-granularity adaptive embeddings for features at each stage of the dual-stream network structure to enhance the features.

[0035] The Cross-Modal Fusion Attention Module (CMFA) receives the AMGE outputs from each stage, constructs a cross-modal graph, performs graph attention fusion, and outputs the fused feature representation.

[0036] The discriminant output module receives the fused features from the CMFA and outputs the final pedestrian recognition result.

[0037] The loss calculation module, connected to the discriminant output module, is used to calculate the total loss based on multiple loss functions during the training phase and to feed back and optimize the two-stream network structure.

[0038] The spatial mapping and association module, connected to the discrimination output module, is used to convert pedestrian recognition results into three-dimensional coordinates and insert personnel tags into the digital twin to construct a near real-time digital twin model of underground space.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] The adaptive multi-granularity enhancement module AMGE proposed in this invention differs structurally from traditional methods. Traditional methods often employ single-scale or fixed-weight fusion schemes, relying solely on fixed weights or manual priors, failing to dynamically adjust according to sample scale or modal differences, and easily overlooking key details or global semantics. AMGE uses multi-dilation rate convolutional branches to extract features from different receptive fields in parallel, covering multi-level information from local texture to global semantics. Global pooling and a lightweight network generate scores for each branch, and the Gumbel-Softmax gating mechanism enables sample-level adaptive scale selection, dynamically highlighting the scale information most discriminative for the current data. Simultaneously, branch complementarity regularization is introduced, making the embeddings generated by each branch complementary and diverse, effectively improving cross-modal discriminative capabilities.

[0041] The proposed Cross-Modal Fusion Attention Module (CMFA) treats the deep embeddings of visible light and infrared images at different levels as graph nodes. By constructing a complete cross-modal adjacency matrix, it tightly connects nodes of the same identity across different modalities, as well as their intra-modal neighbors. CMFA employs a multi-head graph attention mechanism, weighting and aggregating the neighborhood features and self-features of each node to achieve deep interaction with the global context across modalities. Unlike traditional local attention or single-modal graph convolution, CMFA can adaptively weight multi-modal and multi-level features simultaneously. This suppresses the negative impact of noisy nodes and enhances complementary information between different modalities of the same identity, thus significantly smoothing the cross-modal distribution differences in the feature space during training.

[0042] This invention deeply integrates visible light / infrared cross-modal re-identification with digital twin 3D models, overcoming the limitations of traditional methods that rely solely on 2D identification or single 3D modeling. By employing an improved aggregation loss weighted by modal confidence, it significantly enhances the robustness of identification under low-light and complex conditions. SLAM and incremental dense reconstruction are used to accurately map 2D identification results to a 3D scene, enabling intuitive visualization and trajectory playback of "personnel-location". The system can provide precise spatial location of personnel, 3D escape path planning, and rescue scheduling suggestions based on spatial semantics, thereby significantly improving the positioning accuracy and scheduling efficiency of underground search and rescue, reducing false alarms and missed alarms, and providing reliable digital twin support for post-event analysis and drills. Attached Figure Description

[0043] Figure 1 This is a flowchart of a cross-modal pedestrian re-identification method for deep-earth emergency rescue according to the present invention;

[0044] Figure 2 This is a schematic diagram of the network structure of a cross-modal pedestrian re-identification method for deep-earth emergency rescue according to the present invention;

[0045] Figure 3 This is a schematic diagram of the adaptive multi-granularity enhancement module AMGE in this invention;

[0046] Figure 4 This is a schematic diagram of the cross-modal fusion attention module CMFA in this invention. Detailed Implementation

[0047] The invention will now be further described with reference to the accompanying drawings.

[0048] like Figure 1 As shown, a cross-modal pedestrian re-identification method for deep-earth emergency rescue includes the following steps:

[0049] S1. Data preparation: Obtain pedestrian image datasets in deep underground environments, divide them into visible light image sets and infrared image sets according to modality, and preprocess the visible light image sets and infrared image sets.

[0050] S2. Construct a network model, such as Figure 2 As shown, a two-stream network structure is constructed. Each stream uses a deep convolutional network to extract pedestrian features from the visible light image set and the infrared image set. Each stream contains some shared parameters and some independent parameters, taking into account both modality common features and modality specific features.

[0051] S3. Construct an Adaptive Multi-granularity Enhancement Module (AMGE) and intersperse AMGE in the network model to perform adaptive multi-granularity enhancement on single modality features.

[0052] Furthermore, the AMGE processing steps include:

[0053] Multiple parallel convolutional branches with different receptive fields are applied to the input features to generate embeddings at multiple scales. Each branch uses a different dilation rate to capture spatial information at different scales. For example, a smaller dilation rate can extract detailed features, while a larger dilation rate helps to obtain structural features over a wider range. This multi-scale design can significantly enhance the robustness of features under conditions such as low light in downhole drilling, blurred infrared imaging, or inconsistent target scaling. The feature maps output by each convolutional branch are first processed by several sets of convolutions, normalization, and activation functions, and then the channels are compressed by 1×1 projection to obtain scale-specific feature representations. Then, global average pooling is performed on each branch in the spatial dimension and fed into a small fully connected network to generate branch scores. The outputs of each branch are then weighted and fused to obtain the AMGE output. To make the embeddings generated by each branch diverse and complementary, a differentiable gating mechanism based on Gumbel-Softmax and complementary constraints between branches are introduced.

[0054] Specifically, such as Figure 3 As shown, for the backbone number Intermediate features of the layer ,use Parallel branch operator Extraction with the first Characteristics of the sensory field : ;

[0055] Compressing and activating the projection of the channel yields... This is to facilitate subsequent global descriptions and branch-weighted fusion: ;

[0056] In equation (2), For 1×1 convolution, It is the ReLU activation function. For batch normalization.

[0057] For each compressed branch feature Global average pooling (GAP) is used to obtain its global description vector. The GAP vectors are mapped to scalar scores using a small multilayer perceptron (MLP). This score characterizes the first The relative importance or fit of each scale branch to the current sample: ;

[0058] Furthermore, Gumbel noise is injected into the branch scores during training. This allows scale selection to be both learnable and approximate discrete selection: ;

[0059] In equation (4), These are independent, uniformly sampled variables.

[0060] Furthermore, the normalized weights of each branch are calculated. The weighted branch outputs are then fused to obtain the fused multi-scale features. This fusion integrates information from different receptive fields at each sample in an adaptive scale, preserving both local details and contextual semantics. ;

[0061] In equation (5), For temperature parameters, K This represents the number of parallel convolution branches.

[0062] Furthermore, to prevent branches at different scales from learning highly redundant representations, complementary constraints are introduced. This promotes the diversity of scale branches, thereby improving the effectiveness of information fusion. ;

[0063] In equation (6), p and q are branch indices used for summing complementary losses.

[0064] This fusion enables the network to adaptively focus on combinations of features at different scales. Through the above structure and mechanism, AMGE generates richer and more expressive embedded features while ensuring branch diversity, thereby alleviating the problem of weak representational power caused by single-scale features.

[0065] S4. Construct a Cross-Modality Feature Aggregation Module (CMFA) and interweave it into the network model to interactively fuse the features extracted from visible light flow and infrared light flow, and output a joint feature vector.

[0066] Further, the CMFA processing steps include:

[0067] The AMGE outputs of each stage of visible light and infrared images are used as node features, and a cross-modal graph is constructed by setting neighbor relationships. Then, query, key, and value vectors are calculated for each node, and a multi-head graph attention mechanism is used to weighted aggregate neighbor nodes to update the node feature representation. The attention weights are adaptively generated based on the similarity between node features, ultimately obtaining cross-modal fusion features, achieving information fusion within and across modalities. Simultaneously, deep feature similarity is combined with prior terms from infrared statistical distributions and confidence scores calculated by traditional operators, ensuring that infrared information is utilized during the fusion process and improving recognition robustness under nighttime or thermal imaging conditions. Through graph modeling, CMFA can not only achieve semantic fusion of upper and lower layer features within the same modality, but also establish one-to-one or many-to-one semantic correspondences between different modalities, capturing the potential consistency of identity information across different modalities.

[0068] Specifically, such as Figure 4 As shown, for each feature block from AMGE First, global average pooling is used to obtain the description vector, and then a linear projection matrix is ​​used. Mapped to fixed-dimensional node representation For any node, obtain the query, key, and value vectors through a linear transformation: ;

[0069] Calculate the normalized temperature / grayscale histogram And obtain the node pairs using the normalized inner product. Infrared prior similarity It characterizes the similarity of the thermal energy distribution between the two regions, providing supplementary information for cross-modal matching from a physical perspective;

[0070] ;

[0071] In equations (7, 8), , , For learnable matrices, An infrared patch for each node.

[0072] Furthermore, the traditional confidence score in attention is calculated, and for each node, the GAP description of several traditional image operators is calculated and concatenated. The confidence level is then obtained through a small MLP and a sigmoid mapping. ;

[0073] ;

[0074] In equation (9), The MLP function is used to map traditional operator features to confidence scores, and σ(⋅) is the Sigmoid normalization function.

[0075] Furthermore, the attention weights are calculated by linearly combining the three parts. And for all candidate aggregation nodes Normalization is performed to obtain standardized attention weights. : ;

[0076] In equation (10), Let be the dimensions of the node vectors and the attention projection. , represents the weighting coefficients for infrared prior similarity and traditional confidence.

[0077] Furthermore, according to normalized weights For the Value vector Perform a weighted summation, and then output the projection matrix. Mapping to the node representation space yields the fused node representation. : .

[0078] Through the aforementioned mechanism, each node can adaptively aggregate information from intramodal and cross-modal neighbors, eliminating the negative impact of samples with significant visual differences. The multi-head attention variant of the CMFA module further enhances expressive power, with multiple attention heads operating in parallel before being concatenated or fused to capture various associated patterns. After CMFA processing, semantic information from different modalities (VIS / IR) and scales is fused, while also considering infrared and traditional priors, enabling the network to learn stronger cross-modal discriminative features.

[0079] S5. Calculate the loss function, introduce triplet loss for metric learning, and introduce cross-modal fusion loss to improve the stability of cross-modal alignment and aggregation.

[0080] Triplet loss is used to bring positive sample pairs of the same identity closer together and negative sample pairs of different identities further apart. An input triplet consists of a pair of positive sample pairs and a pair of negative sample pairs, with the three images named as the anchor image (a), the positive image (p), and the negative image (negative). n Let image a and image p be a positive sample pair, and image a and image n be a negative sample pair. Define the Euclidean distance metric function. The triplet loss can then be expressed as: ;

[0081] In equation (12), P This indicates the number of pedestrians selected in the training batch. For boundary hyperparameters, For anchor point samples, Positive samples with the same identity. These are negative samples from different identities. This represents the feature vector extracted by the two-stream network.

[0082] Cross-Modality Aggregation Loss is used to enhance the joint discriminative power of visible and infrared modal features. Specifically, it involves first constructing cross-modality fusion features, and then applying classification and similarity constraints to these features. The fusion loss can be expressed as: ;

[0083] In equation (13), , , These are the weighting coefficients; To aggregate the convergence term, each sample is pulled toward the fusion center. To improve cross-modal consistency; L represents the number of identities included in a training batch; M represents the modality set; These are sample-level weights, used to reduce the impact of low-quality samples; The defined fusion center; For identity In modality The One sample; For modal alignment, each modal center is constrained to be close to the fusion center; Ensure that there is at least one connection between the integration centers with different identities. The distance between classes is used to maintain separability; P is the set of center pairs to be considered.

[0084] This loss directly pulls the sample towards the fusion center. Instead of relying solely on modality centers to cluster together, this approach helps compress the representations of different modalities into a unified cluster, thereby improving the recall and accuracy of cross-modal retrieval.

[0085] S6. Jointly train the network model, weighted and combined the loss functions in S5 into a total loss function, and optimize the network parameters through backpropagation.

[0086] Specifically, the above loss functions are weighted and combined according to a preset ratio to construct the total loss function, and the network parameters are simultaneously optimized through backpropagation. The formula for the total loss function is as follows: ;

[0087] In equation (14), Cross-entropy loss (ID loss) is used for identity classification. , , These are weighting coefficients to ensure a balance in the contribution of different losses to model training.

[0088] S7. Feature Extraction and Matching: After the joint training in S6 is completed, for a given visible light pedestrian query image and infrared image database, the dual-stream network is input to generate its feature vector. The similarity between the features of the query image and each image in the database is calculated, and the matching result is returned to complete the re-identification task.

[0089] S8. Construct a digital twin model, project the identified target personnel's location onto the digital twin 3D model through camera pose estimation and depth mapping, forming an intuitive visual label of "personnel-location". In the digital twin environment, update the spatial location and trajectory of trapped or waiting-to-be-rescued personnel in real time, and achieve seamless coupling of identity recognition and spatial positioning.

[0090] Furthermore, the digital twin model is constructed using the SLAM method to estimate the keyframe pose and perform dense depth estimation, including the following steps:

[0091] S8.1: Based on the 2D bounding box detection results returned by the re-identification module, take its center pixel coordinates as the initial location point of the person: ;

[0092] In equation (15), , , , These are the pixel coordinates of the top-left and bottom-right corners of the bounding box, respectively. , Let the coordinates of its center point be given.

[0093] S8.2: Based on the available depth information type, different strategies are used to recover the 3D camera coordinates corresponding to the pixel. If the current keyframe provides a dense depth map... , then ( , Take a local window centered on the target area and calculate the median depth to suppress noise. ;

[0094] In equation (16), z Let K be the depth value we want to find, and K be the camera intrinsic parameter matrix. This indicates the three-dimensional position in the camera coordinate system.

[0095] If the system output is a point cloud Then select all 3D points whose projection falls within the bounding box area, and calculate their centroids as... If no depth information is available, a coarse estimate is made by combining the device's geometric priors with the scene structure (such as ground assumptions or typical floor heights), at which point the confidence level will be significantly reduced.

[0096] S8.3: Transform the 3D camera coordinates obtained in S8.2 to world coordinates using the current frame pose. : ;

[0097] In equation (17), Let be the transformation matrix from the camera frame of reference to the world frame of reference. The coordinates of the personnel in the world coordinate system.

[0098] S8.4: Comprehensively evaluate the reliability of the localization results and define the mapping confidence level. Weighted fusion of multi-source confidence scores: ;

[0099] In equation (18), To re-identify confidence; To estimate the quality confidence level for depth estimation; The pose uncertainty; Visibility rating; These are the weighting coefficients corresponding to each confidence level.

[0100] S8.5: Include personnel ID and world coordinates Confidence level The timestamp is stored in the scene database and marked as a 3D icon in the digital twin model. If the person already has a historical trajectory, the trajectory is associated and updated based on spatiotemporal proximity and feature embedding similarity, thereby forming a continuous and smooth trajectory of the person's movement.

[0101] Furthermore, S1 uses a handheld terminal that integrates human-machine-environment information to acquire a dataset of pedestrian images in the deep earth environment. The handheld terminal is preferably KJD3.7(A). This terminal integrates human-machine-environment information and adopts a Gaussian splash-based SLAM method to achieve real-time reconstruction of single-camera video and photo-level realistic rendering. It has functions such as disaster environment image recognition and perception, intelligent path planning and rescue navigation, as well as emergency rescue knowledge base and real-time decision support.

[0102] The terminal is equipped with a high-resolution camera and various environmental sensors, such as gravity sensors, geomagnetic sensors, gyroscopes, rotation vector sensors, and ambient light sensors, to capture the situation on-site downhole.

[0103] In the event of sudden accidents such as landslides, water inrushes, or excessive gas levels, the terminal uses deep learning algorithms to analyze on-site images to identify the type and extent of the disaster; simultaneously, it combines collected sensor data to assess the condition of trapped personnel and rescue workers. Based on the underground 3D terrain model and real-time environmental information, the terminal performs path planning, calculating and guiding rescue personnel to the target area along the optimal route.

[0104] The terminal has a built-in rich emergency rescue knowledge base, storing rescue plans and expert strategies for various accidents. It can match the corresponding plan according to the current accident type and on-site situation, and provide real-time decision support suggestions for commanders.

[0105] Furthermore, the SLAM method based on Gaussian splashing first uses Gaussian splashing to construct a 3D scene representation and divides the scene into multiple sub-maps, and initializes each sub-map according to the input frame;

[0106] We optimize these submaps using an optimization function to achieve the optimal solution for each submap, thereby achieving the overall optimal solution. The optimization objective function is: ;

[0107] In equation (19), Representing the 1-norm loss, D(p) and C(p) represent the depth and color of pixel p, respectively. 0.5 is the weight of the color error. S(p) is the confidence level calculated based on the opacity of pixel p, which is to judge the degree of Gaussian reconstruction of light by pixel p. S(p) can be calculated by formula.

[0108] Then, it is calculated whether the difference between the current frame and the most recent keyframe reaches a certain threshold. If it exceeds the threshold, the current frame is identified as a keyframe. Gaussian sampling is performed on the keyframe, and the sampled new Gaussian is added to the current sub-map.

[0109] Next, pose tracking is used to ensure that the rendering depth and color of the current frame are as consistent as possible with the actual image. Finally, geometric and color losses are calculated to optimize the model parameters, ultimately achieving real-time reconstruction and photorealistic rendering of single-camera video. The objective function obtained by combining pose loss, geometric loss, and color loss is: ;

[0110] A comprehensive safety early warning model based on multi-source heterogeneous data fusion constructs a spatiotemporal hybrid index model and employs data integration and quality assessment techniques. The system performs risk quantification assessments on collected environmental and physiological data from four dimensions: human, machine, environment, and management. It establishes a data credibility function and performs weighted calculations and corrections on various types of sensor data.

[0111] In equation (20), the target loss used is: ;

[0112] In equation (21), D and These are the true depth map and the reconstructed depth map, respectively.

[0113] For color loss, a weighted combination of channel mean loss and SSIM (structural similarity) loss is used. :

[0114]

[0115] ;

[0116] In equation (22), It is the average value of the p channel of the rendered image. It is the average value of the q channels of the original image. It's rendering an image. It is the original image. , , These are the weighting coefficients for each loss.

[0117] A cross-modal pedestrian re-identification system for deep-earth emergency rescue includes:

[0118] The data acquisition module is used to acquire visible light and infrared images of pedestrians;

[0119] A dual-stream feature extraction module is connected to the data acquisition module to establish a dual-stream network structure for extracting multi-stage backbone features from visible light images and infrared images respectively.

[0120] The adaptive multi-granularity enhancement module AMGE is connected to the dual-stream feature extraction module to generate multi-granularity adaptive embeddings for features at each stage of the dual-stream network structure to enhance the features.

[0121] The Cross-Modal Fusion Attention Module (CMFA) receives the AMGE outputs from each stage, constructs a cross-modal graph, performs graph attention fusion, and outputs the fused feature representation.

[0122] The discriminant output module receives the fused features from the CMFA and outputs the final pedestrian recognition result.

[0123] The loss calculation module, connected to the discriminant output module, is used to calculate the total loss based on multiple loss functions during the training phase and to feed back and optimize the two-stream network structure.

[0124] The spatial mapping and association module, connected to the discrimination output module, is used to convert pedestrian recognition results into three-dimensional coordinates and insert personnel tags into the digital twin to construct a near real-time digital twin model of underground space.

[0125] This invention first utilizes a two-stream neural network to extract multi-layer features from two modalities. Then, an adaptive multi-granularity enhancement module performs multi-granularity semantic enhancement on the features, improving feature perception capabilities under low light and complex conditions. Furthermore, a cross-modal fusion attention module is introduced, fusing cross-modal adjacency relationships based on a graph structure and employing a multi-head graph attention mechanism to improve robustness and generalization ability. Combining cross-modal aggregation loss and joint loss optimization strategies, the discriminative power of cross-modal features is effectively improved. This invention combines the 3D reconstruction and UWB navigation functions of a mining intelligent handheld terminal to construct a near-real-time digital twin model of underground space. It couples the 2D cross-modal Re-ID results with incremental 3D reconstruction based on Gaussian splashing, enabling 3D positioning of identified personnel in the digital twin scene and providing them with spatial location and escape route planning, thus enhancing personnel safety monitoring and emergency rescue capabilities in deep-earth environments.

Claims

1. A cross-modal pedestrian re-identification method for deep-earth emergency rescue, characterized in that, Includes the following steps: S1. Data preparation: Obtain pedestrian image datasets in deep earth environments, divide them into visible light image sets and infrared image sets according to modality, and preprocess the visible light image sets and infrared image sets. S2. Construct a network model and build a two-stream network structure. Each stream uses a deep convolutional network to extract pedestrian features from the visible light image set and the infrared image set. Each stream contains some shared parameters and some independent parameters, taking into account both modal common features and modal specific features. S3. Construct an adaptive multi-granularity enhancement module AMGE and intersperse AMGE in the network model to perform adaptive multi-granularity enhancement on single modality features. AMGE processing steps include: Multiple parallel convolutional branches with different receptive fields are applied to the input features to generate multiple scale embeddings. The feature maps output by each convolutional branch are first processed by several sets of convolutions, normalization and activation functions, and then the channels are compressed by 1×1 projection to obtain scale-specific feature representations. Then, global average pooling is performed on each branch in the spatial dimension and fed into a small fully connected network to generate branch scores. The outputs of each branch are then weighted and fused to obtain the AMGE output. S4. Construct a cross-modal fusion attention module (CMFA), interspersed in the network model, to interactively fuse the features extracted from visible light flow and infrared light flow, and output a joint feature vector. CMFA processing steps include: The AMGE outputs of each stage of visible light and infrared images are used as node features. A cross-modal graph is constructed by setting neighbor relationships. Then, query, key, and value vectors are calculated for each node. The neighbor nodes are weighted and aggregated using a multi-head graph attention mechanism to update the node feature representation. The attention weights are adaptively generated based on the similarity between node features. Finally, cross-modal fusion features are obtained, realizing information fusion within and across modalities. Specifically, for each feature block from AMGE First, global average pooling is used to obtain the description vector, and then a linear projection matrix is ​​used. Mapped to fixed-dimensional node representation For any node, obtain the query, key, and value vectors through a linear transformation: ； Calculate the normalized temperature / grayscale histogram And obtain the node pairs using the normalized inner product. Infrared prior similarity It characterizes the similarity of the thermal energy distribution between the two regions, providing supplementary information for cross-modal matching from a physical perspective; ； In equations (7, 8), , , For learnable matrices, Infrared patch for each node; Furthermore, the traditional confidence score in attention is calculated, and for each node, the GAP description of several traditional image operators is calculated and concatenated. The confidence level is then obtained through a small MLP and a sigmoid mapping. ; ； In equation (9), The MLP function maps traditional operator features to confidence scores, where σ(⋅) is the Sigmoid normalization function; Furthermore, the attention weights are calculated by linearly combining the three parts. and for all candidate aggregation nodes Normalization is performed to obtain standardized attention weights. : ； In equation (10), Let be the dimensions of the node vectors and the attention projection. , The weighting coefficients for infrared prior similarity and traditional confidence are: Furthermore, according to normalized weights For the Value vector Perform a weighted summation, and then output the projection matrix. Mapping to the node representation space yields the fused node representation. : ； S5. Calculate the loss function, introduce triplet loss for metric learning, and introduce cross-modal fusion loss to improve the stability of cross-modal alignment and aggregation; S6. Jointly train the network model, weightedly combine the loss functions in S5 into a total loss function, and optimize the network parameters through backpropagation; S7. Feature extraction and matching: After the joint training in S6 is completed, for a given visible light pedestrian query image and infrared image database, the dual-stream network is input to generate its feature vector. The similarity between the features of the visible light pedestrian query image and each image in the infrared image database is calculated, and the matching result is returned to complete the re-identification task. S8. Construct a digital twin model, project the identified target personnel's location onto the digital twin 3D model through camera pose estimation and depth mapping, forming an intuitive visual label of "personnel-location". In the digital twin environment, update the spatial location and trajectory of trapped or waiting-to-be-rescued personnel in real time, and achieve seamless coupling of identity recognition and spatial positioning.

2. The cross-modal pedestrian re-identification method for deep-earth emergency rescue according to claim 1, characterized in that, In S5, the triplet loss is used to bring positive sample pairs of the same identity closer together and negative sample pairs of different identities further apart; the cross-modal fusion loss is used to enhance the joint discriminative power of visible light and infrared modal features.

3. The cross-modal pedestrian re-identification method for deep-earth emergency rescue according to claim 1, characterized in that, The construction of a digital twin model employs the SLAM method to estimate keyframe poses and perform dense depth estimation, including the following steps: S8.

1. Based on the two-dimensional bounding box detection results returned by the re-identification module, take the center pixel coordinates as the initial location point of the person; S8.

2. Based on the available depth information types, different strategies are used to recover the 3D camera coordinates corresponding to the pixel; S8.

3. Transform the 3D camera coordinates obtained in S8.2 into world coordinates using the pose of the current frame; S8.

4. Evaluate the credibility of the positioning results comprehensively, and define the mapping confidence as a weighted fusion of multi-source confidence; S8.5 Store the personnel identifier, world coordinates, confidence level, and timestamp into the scene database and mark them as three-dimensional icons in the digital twin model; if the personnel already have historical trajectories, then perform trajectory association and updates based on spatiotemporal proximity and feature embedding similarity to form a continuous and smooth personnel movement trajectory.

4. The cross-modal pedestrian re-identification method for deep-earth emergency rescue according to claim 1, characterized in that, S1 uses a handheld terminal that integrates human-machine-environment information to acquire a dataset of pedestrian images in a deep underground environment.

5. A cross-modal pedestrian re-identification system for deep-earth emergency rescue, characterized in that, The method for cross-modal pedestrian re-identification for deep-earth emergency rescue as described in any one of claims 1-4 includes: The data acquisition module is used to acquire visible light and infrared images of pedestrians; A dual-stream feature extraction module is connected to the data acquisition module to establish a dual-stream network structure for extracting multi-stage backbone features from visible light images and infrared images respectively. The adaptive multi-granularity enhancement module AMGE is connected to the dual-stream feature extraction module to generate multi-granularity adaptive embeddings for features at each stage of the dual-stream network structure to enhance the features. The Cross-Modal Fusion Attention Module (CMFA) receives the AMGE outputs from each stage, constructs a cross-modal graph, performs graph attention fusion, and outputs the fused feature representation. The discriminant output module receives the fused features from the CMFA and outputs the final pedestrian recognition result. The loss calculation module, connected to the discriminant output module, is used to calculate the total loss based on multiple loss functions during the training phase and feed it back to optimize the two-stream network structure. The spatial mapping and association module, connected to the discrimination output module, is used to convert pedestrian recognition results into three-dimensional coordinates and insert personnel tags into the digital twin to construct a near real-time digital twin model of underground space.

Images