Three-dimensional positioning method for roadway caving scene anchor rod based on dynamic multi-scale cavity attention network

Abstract

This application relates to the field of tunnel safety technology and provides a method for three-dimensional anchor bolt positioning based on a dynamic multi-scale cavity attention network. This method utilizes multi-modal sensors to generate three-dimensional point cloud data of the anchor bolt, overcoming the limitations of a single sensor. Through a spatiotemporal attention mechanism, it dynamically captures the salient regions of the buried anchor bolt in time and space, removing noise points from the three-dimensional point cloud data and enhancing the geometric feature representation of the anchor bolt. Simultaneously, it embeds the anchor bolt's geometry and material properties as physical constraints into the point cloud optimization process, significantly improving the quality of the anchor bolt's point cloud and ensuring its integrity and accuracy. This enables the anchor bolt to maintain stable performance in unstructured environments, exhibits strong noise resistance, stably extracts the spatial pose features of the anchor bolt, and evaluates the stress distribution and stability of the anchor bolt based on the three-dimensional reconstructed point cloud model. This achieves full-process coverage from dynamic perception to hazard prediction, providing a scientific basis for tunnel rescue.

Description

Technical Field

[0001] This application relates to the field of tunnel safety technology, and in particular to a three-dimensional positioning method for anchor bolts in tunnel collapse scenarios based on a dynamic multi-scale void attention network. Background Technology

[0002] In the mining of mineral resources at depths of several kilometers, the complex environment of high ground stress, high gas pressure, and high ground temperature can easily lead to sudden roadway collapses due to factors such as stress concentration in the roof and floor of the coal seam, roadway deformation, damage to the surrounding rock, and exceeding the bearing capacity of the support structure, resulting in severe loss of life and property. To ensure the safe construction of coal mine roadways, high-strength steel bars such as anchor bolts and anchor cables are commonly used as support materials to improve the stability and bearing capacity of the roadways. However, once a roadway collapse accident occurs, the deformation, breakage, and collapse of anchor bolts can increase the burial depth, obstruct rescue channels, and form complex spatial barriers, posing a significant obstacle to rescue efforts. Summary of the Invention

[0003] The purpose of this application is to provide a three-dimensional positioning method for anchor bolts in roadway collapse scenarios based on a dynamic multi-scale void attention network, so as to solve or alleviate the problems existing in the above-mentioned prior art.

[0004] To achieve the above objectives, this application provides the following technical solution: This application provides a method for three-dimensional localization of anchor bolts in a roadway collapse scene based on a dynamic multi-scale void attention network, including: extracting features from multimodal data of anchor bolts in a roadway collapse scene based on a dynamic perception feature extraction mechanism, and fusing the multimodal features obtained by feature extraction based on a spatiotemporal attention mechanism to generate fused features of anchor bolts in a roadway collapse scene. By using a spatiotemporal attention mechanism to significantly enhance the fusion features, preliminary three-dimensional point cloud data of anchor bolts in a roadway collapse scenario are obtained. Surface reconstruction is performed based on optimized 3D point cloud data of anchor bolts, and texture mapping is performed on the continuous 3D anchor bolt model generated by surface reconstruction to obtain an enhanced 3D anchor bolt model; wherein, the optimized 3D point cloud data is obtained by optimizing the initial 3D point cloud data through the physical constraints of the anchor bolts; Deformation detection was performed on the enhanced 3D anchor bolt model to determine the fracture area of ​​the anchor bolt.

[0005] Preferably, based on a dynamic perception-based feature extraction mechanism, features are extracted from the multimodal data of anchor bolts in a roadway collapse scenario. Then, based on a spatiotemporal attention mechanism, the extracted multimodal features are fused to generate fused features of the anchor bolts in the roadway collapse scenario, including: Dynamic multi-scale dilated convolution operation is performed on the multimodal data of anchor bolts in the roadway collapse scenario obtained based on multimodal sensors to extract the multimodal features of anchor bolts in the roadway collapse scenario. The multimodal features are then sequentially aligned with time series and dynamically fused to obtain the fused features of anchor bolts in the roadway collapse scenario. Specifically, based on the saliency distribution of the input feature map, the porosity of the dilated convolution kernel is dynamically adjusted to obtain the dynamic porosity of the corresponding dilated convolution kernel; and based on the dynamic porosity of the dilated convolution kernel and the saliency value of the input feature map, the multimodal features of the anchor bolt in the roadway collapse scenario are determined.

[0006] Preferably, the multimodal data includes: ultrasonic data, millimeter-wave radar data, and binocular vision data; The ultrasonic data were analyzed by multi-scale weighted window and short-time Fourier transform respectively to extract the multi-scale energy of the anchor bolt in the roadway collapse scenario. The extracted multi-scale energy was then fused to obtain the first edge feature of the anchor bolt in the roadway collapse scenario. Doppler features of anchor bolts in a roadway collapse scenario are extracted from millimeter-wave radar data, and enhanced Doppler features are obtained by dynamic multi-scale dilated convolution. Image inpainting is performed on multi-view binocular vision data based on nonlocal mean filtering to extract the second edge features and surface texture features of anchor bolts in the roadway collapse scene.

[0007] Preferably, according to the formula:

[0008] Determine anchor bolt sampling points in the scenario of tunnel collapse. Use the first Weighted energy eigenvalues ​​obtained by weighting windows at each scale In the formula, For the first The window length of a scale-weighted window. For the first The scale-weighted window at the th ... Position weight, This is the position index within the scale-weighted window. ; For the ultrasonic echo signal at the 1st The amplitude of each sampling point; For sampling points The weighted energy feature values ​​obtained from weighted windows at various scales are fused to obtain the anchor bolt sampling points in the roadway collapse scenario. eigenvectors at location Among them, the feature vector The first boundary features of the anchor bolt are characterized in the scenario of tunnel collapse; The phase information of the ultrasonic echo signal is extracted by short-time Fourier transform, and the phase transition point is detected to obtain the second boundary features of the anchor bolt in the roadway collapse scenario. The first and second boundary features of the anchor rod in the roadway collapse scenario are fused to obtain the first edge feature of the anchor rod in the roadway collapse scenario.

[0009] Preferably, according to the formula:

[0010] Determine the first Dynamic void ratio of each voided convolution kernel In the formula, Input feature map Middle position The significance value at the location, The maximum and minimum dilatation rates during dynamic multi-scale dilated convolution are respectively determined. According to the formula:

[0011] Determine the location of anchor bolts in a tunnel collapse scenario From the first Local feature values ​​extracted by dilated convolution kernels In the formula, For the first The global void ratio of each dilated convolution kernel. , For the first The weights of each dilated convolution kernel, Input feature map At dynamic convolution sampling points Eigenvalues ​​at; This is a two-dimensional coordinate index for the dilated convolution kernel, representing the position of the weights within the kernel. Input feature map The two-dimensional spatial coordinates represent the center position of the current convolution operation. ， is the receptive field radius of the dilated convolution kernel; It is a positive integer; Anchor bolt location in a tunnel collapse scenario From the first Local feature values ​​extracted by dilated convolution kernels Perform aggregation operation to obtain the first... Each dilated convolution kernel convolves the input feature map. Global modal features corresponding to convolution operations .

[0012] Preferably, according to the formula:

[0013] The obtained global modal features are dynamically fused to obtain the fused features of anchor bolts in the roadway collapse scenario. ; In the formula, For the first Each dilated convolution kernel convolves the input feature map. Global modal features corresponding to the convolution operation; Global modal features Dynamic weights, This represents the number of types of dilated convolution kernels. All are positive integers.

[0014] Preferably, a saliency map of the anchor bolts is generated by extracting features from the multimodal data of the anchor bolts in the roadway collapse scenario; wherein, the saliency map represents the positional saliency of the anchor bolts in the roadway collapse scenario; Based on the saliency map of the anchor bolts, the fusion features are analyzed. Significance enhancement was performed to obtain the enhanced features of the anchor bolts in the scenario of tunnel collapse. ; Based on enhanced features Determine the three-dimensional spatial coordinates of the anchor bolt in a tunnel collapse scenario. To generate preliminary 3D point cloud data of anchor bolts in a roadway collapse scenario.

[0015] Preferably, based on the geometry and material properties of the anchor bolt, the preliminary three-dimensional point cloud data is globally optimized to obtain optimized three-dimensional point cloud data, and the surface of the optimized three-dimensional point cloud data is reconstructed using the Marching Cubes algorithm to generate a continuous three-dimensional anchor bolt model. The surface texture features obtained from binocular vision data in multimodal data are mapped onto the 3D anchor model to obtain the enhanced 3D anchor model.

[0016] Preferably, the deformation amplitude of the anchor bolt in the roadway collapse scenario is determined based on the geometric deviation between the enhanced 3D anchor bolt model and the standard anchor bolt model, so as to generate a deformation saliency map of the anchor bolt in the roadway collapse scenario; Based on the temperature distribution matrix extracted from the infrared thermogram of the anchor bolt in the roadway collapse scenario, the local gradient of the temperature distribution of the anchor bolt in the roadway collapse scenario is calculated to generate a temperature saliency map of the anchor bolt in the roadway collapse scenario. The deformation significance map and temperature significance map of the anchor bolt in the roadway collapse scenario are weighted and fused to generate a comprehensive significance map of the anchor bolt in the roadway collapse scenario. The comprehensive saliency map of the anchor bolts in the roadway collapse scenario is mapped onto the surface mesh points of the enhanced 3D anchor bolt model, and the saliency marker areas exceeding the preset saliency threshold are identified as the fracture areas of the anchor bolts.

[0017] Preferably, based on the enhanced three-dimensional anchor bolt model, the stress distribution of the anchor bolt under the roadway collapse scenario is determined by the finite element method, and the mechanical stability of the anchor bolt under the roadway collapse scenario is evaluated according to the material characteristics of the anchor bolt. By analyzing the load and stress relationship of anchor bolts in a roadway collapse scenario, the potential failure area of ​​anchor bolts in such a scenario is predicted.

[0018] Beneficial effects: The method for 3D localization of anchor bolts in a roadway collapse scene based on a dynamic multi-scale void attention network provided in this application embodiment extracts features from the multimodal data of anchor bolts in a roadway collapse scene based on a dynamic perception feature extraction mechanism. Then, based on a spatiotemporal attention mechanism, the extracted multimodal features are fused to generate fused features of the anchor bolts in the roadway collapse scene. The fused features are then significantly enhanced using a spatiotemporal attention mechanism to obtain preliminary 3D point cloud data of the anchor bolts in the roadway collapse scene. The preliminary 3D point cloud data is then optimized using the physical constraints of the anchor bolts to obtain optimized 3D point cloud data. Surface reconstruction is then performed based on the optimized 3D point cloud data. Texture mapping is then applied to the continuous 3D anchor bolt model generated by the surface reconstruction to obtain an enhanced 3D anchor bolt model. Finally, deformation detection is performed on the enhanced 3D anchor bolt model to determine the fracture area of ​​the anchor bolt. Therefore, by using multimodal data of anchor bolts to compensate for the insufficient data collection in single detection methods, the location and shape of buried and collapsed anchor bolts can be reconstructed in three dimensions. This enables the analysis of the three-dimensional distribution of anchor bolts at multiple scales in the collapse scenario, accurately identifying the location and three-dimensional shape of anchor bolts in coal mines with roadway collapse accidents, and providing a reference for rescue decision-making. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. Wherein: Figure 1 This is a flowchart illustrating a method for three-dimensional anchor bolt localization in a roadway collapse scenario based on a dynamic multi-scale void attention network, according to some embodiments of this application. Figure 2 This is a logical schematic diagram of a three-dimensional anchor bolt localization method based on a dynamic multi-scale void attention network for a roadway collapse scenario provided according to some embodiments of this application; Figure 3 This is a schematic diagram illustrating the generation of an enhanced three-dimensional anchor bolt model according to some embodiments of this application. Detailed Implementation

[0020] The present application will now be described in detail with reference to the accompanying drawings and embodiments. Various examples are provided by way of explanation and not by way of limitation. In fact, those skilled in the art will understand that modifications and variations can be made to the present application without departing from the scope or spirit of the present application. For example, a feature shown or described as part of one embodiment may be used in another embodiment to produce yet another embodiment. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention should fall within the scope of protection of the embodiments of the present invention.

[0021] Regarding the deformation, fracture, and collapse of anchor bolts in mine roadway collapse accidents, current research primarily focuses on anchor bolt support to prevent roadway collapses, with less attention paid to post-collapse scenario analysis. After a roadway collapse disaster occurs in coal mines, the location of the collapsed anchor bolt is often determined through acoustic emission systems, borehole verification, and manual tapping. However, due to severe signal attenuation caused by the complex environment, accurate location is impossible, leading to missed opportunities for optimal rescue. Existing technologies are mostly concentrated on early warning systems for coal mine roadway collapses; even when it comes to coal mine disaster rescue, they largely focus on rescue robots, sensor-based obstacle detection and avoidance.

[0022] On the one hand, existing rescue robots generally use lasers or multi-sensor fusion to achieve environmental perception, but they do not specifically identify key support structures (such as anchor bolts) after a landslide. In landslide scenarios, anchor bolts are often obscured by rubble and coal dust, and traditional point cloud filtering and SLAM algorithms are easily affected by noise, leading to the failure of anchor bolt feature extraction and thus failing to provide accurate spatial reference for rescue path planning. While binocular vision and ultrasonic sensors can avoid obstacles, their resolution for small obstacles (such as broken anchor bolts) in unstructured environments after a landslide is very low.

[0023] On the other hand, in traditional convolutional network-based methods, the receptive field of traditional convolutional networks is fixed, making it difficult to dynamically capture the local features of anchor bolts under multi-scale occlusion. In landslide scenarios, anchor bolts are often in a twisted or broken state, and their three-dimensional pose directly affects the stability assessment of rescue channels. Existing rescue equipment based on two-dimensional maps or simple three-dimensional reconstruction lacks accurate modeling of the spatial pose of anchor bolts, making it impossible to achieve high-precision positioning.

[0024] In addition, in the detection of anchor bolts after a collapse, traditional electromagnetic positioning technology is affected by the shielding of concrete, with a detection depth of less than 3 meters, and is easily affected by environmental noise; acoustic detection suffers from signal aliasing in dense anchor bolt areas, and cannot support the three-dimensional reconstruction of anchor bolts; single-mode positioning methods are affected by signal attenuation under complex geological conditions, resulting in an extremely high failure rate.

[0025] Based on this, this embodiment proposes a three-dimensional anchor bolt localization method for roadway collapse scenarios based on a dynamic multi-scale void attention network. Using ultrasonic waves, millimeter-wave radar, and binocular vision, combined with a dynamic multi-scale void attention network, a three-dimensional dynamic perception network structure is established to locate anchor bolts in complex mine roadway collapse scenarios. This achieves three-dimensional distribution analysis of multi-scale anchor bolts in collapse scenarios, helping coal mines experiencing roadway collapse accidents to accurately identify the location and three-dimensional shape of anchor bolts, providing a reference for rescue decision-making.

[0026] like Figures 1 to 3 As shown, the method for 3D localization of anchor bolts in a tunnel collapse scenario based on a dynamic multi-scale void attention network includes: Step S101: Based on the feature extraction mechanism of dynamic perception, feature extraction is performed on the multimodal data of anchor bolts in the roadway collapse scenario. Based on the spatiotemporal attention mechanism, feature fusion is performed on the multimodal features obtained by feature extraction to generate fused features of anchor bolts in the roadway collapse scenario.

[0027] In tunnel collapse scenarios, anchor bolts are typically covered by 1-3 cm of gravel, coal dust, etc., obscuring approximately 70%-90% of their surface, significantly increasing the difficulty of target feature extraction. Within the covered area, the signal reflection intensity of single sensors such as vision, ultrasound, and millimeter-wave radar decreases by 50%-80%, making it difficult to accurately capture the detailed features of the anchor bolts. Furthermore, anchor bolts in collapse environments have complex shapes (they may be broken, twisted, or partially buried), and their feature distribution exhibits significant multi-scale characteristics (the size of the break point (local detail) is approximately 5-10 mm, while the overall anchor bolt (global outline) length can reach 1.5-1 meter). Traditional convolutional networks, limited by a fixed receptive field, struggle to simultaneously capture millimeter-level details and meter-level multi-scale features. In complex noisy environments, coal dust, gravel, etc., cause the signal-to-noise ratio of visual images to drop to 3-5 dB, and ultrasonic signals generate strong interference due to multipath effects, with noise errors accounting for as much as 40%-60%. Existing attention mechanisms (such as channel attention) fail to incorporate spatiotemporal information, easily losing target details in complex environments.

[0028] This application establishes a multimodal sensor collaborative sensing framework incorporating different types of sensors to achieve multimodal data acquisition of anchor bolts in coal mine collapse scenarios. An ultrasonic array sensor, millimeter-wave radar, and a binocular vision system combining visible and infrared light are used to collaboratively detect anchor bolts in coal mine collapse scenarios, compensating for insufficient data acquisition and cross-validating data from multiple sources. Specifically, the visible and infrared binocular vision system non-destructively identifies the location and surface features of collapsed anchor bolts, clearly defining the collapse area. Simultaneously, to overcome the limitations of binocular vision, which relies on lighting conditions and is difficult to identify due to the harsh environment after a coal mine collapse, an ultrasonic array sensor and millimeter-wave radar are combined for detection. The location of the anchor bolt is determined by the emission and absorption of sound waves from the ultrasonic array sensor, unaffected by ambient light interference. Utilizing the strong penetrating power of millimeter-wave radar waves, sub-millimeter high resolution accurately distinguishes collapsed anchor bolts from concrete, minerals, and other materials, identifying anchor bolts buried beneath other materials. This effectively solves the problem that the resolution of a single ultrasonic sensor is only at the millimeter level, making it impossible to accurately identify and extract the precise edges of collapsed anchor bolts.

[0029] Specifically, the ultrasonic array sensor enables data acquisition to withstand light interference in harsh environments such as dust and water vapor; the echo time difference and Doppler frequency shift are used to detect the anchor bolt metal results, and beamforming is formed by the array to achieve multi-angle detection; the phase difference and intensity difference of multiple sensor echo signals are used to monitor the anchor bolt metal results, helping to determine the approximate area of ​​the anchor bolt.

[0030] Millimeter-wave radar emits electromagnetic waves that penetrate metallic media to detect embedded anchor bolts. The high reflectivity of metal (reflectivity greater than 0.9) enhances edge echo signals, and high-resolution (sub-millimeter) imaging extracts the anchor bolt's geometric contours, distinguishing adjacent metal components. In a binocular vision system, a visible light camera identifies underground obstacles and captures visible features such as anchor bolt surface texture; an infrared thermal imager monitors abnormal areas of the anchor bolt structure based on temperature distribution, and pixel-level fusion of the infrared thermal image and the visible light image enhances target recognition and extraction of the anchor bolt.

[0031] This method utilizes binocular vision to identify significant underground obstacles (visible light portion) after a coal mine collapse, and infrared imaging to identify abnormal areas (fractures) and approximate collapse locations of anchor bolts. Ultrasonic detection is combined to determine the area of ​​buried and collapsed anchor bolts in the harsh environment of a collapse scenario. Millimeter-wave radar is then used to further clarify the location and shape of the buried and collapsed anchor bolts, enabling three-dimensional reconstruction of the anchor bolt morphology.

[0032] In this embodiment, multi-scale dilated convolution operations are performed on the multi-modal data of anchor bolts in a roadway collapse scenario obtained from multi-modal sensors to extract multi-modal features of the anchor bolts in the roadway collapse scenario. These multi-modal features are then sequentially time-series aligned and dynamically fused to obtain the fused features of the anchor bolts in the roadway collapse scenario. Specifically, anchor bolt features reflecting local geometric characteristics and depth abrupt changes (such as surface cracks and fracture edges) are extracted based on the echo energy and phase abrupt change points of the data collected by the ultrasonic array sensor. Doppler features (micro-Doppler features and example Doppler images) are extracted from the data collected by the millimeter-wave radar. Edge features reflecting visible optical boundaries and surface texture abrupt change regions (such as anchor bolt tips and bends) and thermal radiation anomaly regions are extracted from the data collected by the binocular vision system. Thus, through a dynamic perception feature extraction mechanism, key features of each modality of data are extracted, maximizing the preservation of the saliency and fine features of the anchor bolt target.

[0033] In this application, before extracting the multimodal features of anchor bolts in a roadway collapse scenario, the collected multimodal data is first preprocessed. Specifically, wavelet transform is used to remove multipath interference from ultrasonic data, clutter suppression is performed on millimeter-wave radar signal data based on Constant 0 Alarm Rate (CFAR), and binocular vision data undergoes filtering, denoising, and contrast enhancement and detail restoration using a generative adversarial network. Simultaneously, environmental compensation sensors are used to correct the collected multimodal data. During the ultrasonic data processing, a real-time temperature sensor measures the downhole temperature. According to the formula:

[0034] speed of sound waves Make corrections.

[0035] During the processing of data acquired by millimeter-wave radar, the following formula is used:

[0036] The dielectric constant in millimeter-wave radar signals is corrected; where, The dielectric constant before correction in millimeter-wave radar signals. The corrected dielectric constant in millimeter-wave radar signals. This is a correction factor for humidity levels in underground coal mines. This represents the percentage of humidity in underground coal mines.

[0037] After preprocessing the collected multimodal data, feature extraction is performed on the multimodal data of anchor bolts in a roadway collapse scenario based on a dynamic perception feature extraction mechanism. In a specific example, ultrasonic data is analyzed using multi-scale weighted windows and short-time Fourier transform to extract the multi-scale energy of anchor bolts in the roadway collapse scenario. Specifically, the ultrasonic echo signal is analyzed using multi-scale weighted windows to extract the energy characteristics of the significant reflection area of ​​the anchor bolt. The acquired ultrasonic echo signal is filtered and denoised, then framed using multi-scale windows, with each window applying a different weighting function to calculate the energy characteristics within each window. Here, according to the formula:

[0038] Determine anchor bolt sampling points in the scenario of tunnel collapse. Use the first Weighted energy eigenvalues ​​obtained by weighting windows at each scale In the formula, For the first The window length of a scale-weighted window. For the first The scale-weighted window at the th ... Position weight, This is the position index within the scale-weighted window. ; For the ultrasonic echo signal at the 1st The amplitude of each sampling point.

[0039] Then, feature fusion is performed on the extracted multi-scale energy to obtain the geometric boundary information of the anchor bolts in the tunnel collapse scenario, namely the first edge features (local geometric features and depth abrupt change points of the anchor bolts, such as deep cracks, fracture edges, etc.). Specifically, for the sampling points... The weighted energy feature values ​​obtained from weighted windows at various scales are fused to obtain the anchor bolt sampling points in the roadway collapse scenario. The feature vector represents the coarse boundary features (i.e., the first boundary features) of the anchor bolt in the scenario of tunnel collapse. .Right now:

[0040] Through feature vectors The weighted energy distribution of ultrasonic signals at different scales is described to reflect the depth variation, geometric characteristics, and edge strength of the anchor bolt surface. Simultaneously, the phase information of the ultrasonic echo signal is extracted using short-time Fourier transform, and phase transition points are detected to obtain the fine boundary features (i.e., second boundary features) of the anchor bolt in a tunnel collapse scenario. Specifically, according to the formula:

[0041] Detecting phase transition points This accurately captures the fine boundary features of the anchor bolt. In the formula, These represent the time-frequency distribution of the ultrasonic echo signal in the time-frequency domain. ,time The phase information is used to significantly enhance the signal strength in the anchor bolt reflection area, suppress background noise, and accurately identify anchor bolt fracture edges and complex shapes.

[0042] In this embodiment, through feature vectors The spatial domain information of the anchor bolts is processed to characterize their geometric properties and overall edge strength, reflecting areas with significant depth variations. Short-time Fourier transform is used to process the time-frequency domain information (phase and frequency information) of the anchor bolts, detecting subtle abrupt changes on the anchor bolt surface and reflecting areas with minor surface changes and significant phase variations. Finally, feature fusion is performed on the first and second boundary features of the anchor bolts in a tunnel collapse scenario, and the feature vectors are analyzed. By complementing the information from the short-time Fourier transform features, the first edge features of the anchor bolt in the tunnel collapse scenario are obtained. This provides both the depth variation and overall geometric information of the anchor bolt, describing its coarse-grained edge features, and the surface abrupt change points and phase information of the anchor bolt surface, capturing the fine boundary of the anchor bolt and generating more complete and accurate anchor bolt edge features.

[0043] In a specific example, millimeter-wave radar transmits frequency-modulated continuous waves or pulse signals and receives the echo signals reflected from the anchor bolts. The Doppler characteristics of the anchor bolts are then extracted from the echo signals. Specifically, frequency analysis of the echo signals based on Fast Fourier Transform (FFT) is used to obtain the distance information of the anchor bolts; the velocity of the anchor bolts is analyzed using the Doppler effect; and the azimuth angle of the anchor bolts is analyzed using a multi-input multi-output (MIMO) antenna array based on phase difference.

[0044] Specifically, according to the formula:

[0045] Distance characteristics of anchor bolts Extraction is performed; where, At the speed of light, The difference frequency of the echo signal from the millimeter-wave radar. This represents the frequency modulation slope of the millimeter-wave radar.

[0046] According to the formula:

[0047] Velocity characteristics of anchor bolts Extraction is performed; where, The radar wavelength of millimeter-wave radar. This is the Doppler frequency shift.

[0048] According to the formula:

[0049] azimuth of the anchor bolt Extraction is performed; where, This refers to the phase difference between the received signals from adjacent antennas in a multi-input multi-output antenna array. Phase difference The antenna spacing between adjacent antennas.

[0050] Then, for the Doppler features of anchor bolts extracted from millimeter-wave radar data in the tunnel collapse scenario, the extracted Doppler features are enhanced through dynamic multi-scale dilated convolution to obtain enhanced Doppler features of anchor bolts in the tunnel collapse scenario. During the dynamic multi-scale dilated convolution process, the receptive field of feature extraction is significantly expanded by dynamically adjusting the convolution dilation rate, capturing the spatial details of anchor bolts under multi-scale occlusion.

[0051] In this embodiment, the hole rate of the convolution is adjusted as needed based on the saliency distribution of the extracted Doppler features of the anchor bolt to extract the anchor bolt adaptability features. Specifically, the hole rate of the dilated convolution kernel is dynamically adjusted according to the saliency distribution of the input feature map to obtain the dynamic hole rate of the corresponding dilated convolution kernel; and the multimodal features of the anchor bolt under the tunnel collapse scenario are determined based on the dynamic hole rate of the dilated convolution kernel and the saliency value of the input feature map.

[0052] According to the formula:

[0053] Determine the first Dynamic void ratio of each voided convolution kernel In the formula, Input feature map for dynamic multi-scale dilated convolution (Location in the extracted Doppler features) The significance value at the location, The maximum and minimum void ratios during dynamic multi-scale dilated convolution are respectively determined.

[0054] According to the formula:

[0055] Determine the location of anchor bolts in a tunnel collapse scenario From the first Local feature values ​​extracted by dilated convolution kernels In the formula, For the first The global void ratio of each dilated convolution kernel. , For the first The weights of each dilated convolution kernel, Input feature map At dynamic convolution sampling points Eigenvalues ​​at; This is a two-dimensional coordinate index for the dilated convolution kernel, representing the position of the weights within the kernel. Input feature map The two-dimensional spatial coordinates represent the center position of the current convolution operation. ， is the receptive field radius of the dilated convolution kernel; It is a positive integer.

[0056] Finally, according to the formula:

[0057] Anchor bolt location in a tunnel collapse scenario From the first Local feature values ​​extracted by dilated convolution kernels Perform aggregation operations (such as weighted summation) to obtain the first... Each dilated convolution kernel convolves the input feature map. Global modal features corresponding to convolution operations Therefore, by dynamically adjusting the void ratio of the convolution kernel, effective modeling of the anchor bolt's features at different scales can be achieved, solving the problem that traditional convolution with a fixed receptive field cannot simultaneously capture the local details and global distribution of the anchor bolt.

[0058] In a specific example, a mobile binocular vision system equipped with a visible light camera and an infrared thermal imager is used to acquire surface images of the anchor bar during movement using a multi-view scanning strategy. Image inpainting is then performed based on non-local mean filtering to restore texture details in areas occluded by the anchor bar. Specifically, the Sobel operator is used to extract edge features of the anchor bar, and a ResNet network is used to extract surface texture features. Then, the temperature distribution acquired by the infrared thermal imager and the visible light texture features are fused at the pixel level, and anomaly regions of the anchor bar are dynamically captured using a spatiotemporal attention mechanism.

[0059] After extracting the multimodal features of anchor bolts in a roadway collapse scenario, a unified coordinate system for ultrasonic, millimeter-wave radar, and binocular vision was calibrated using laser point cloud data. Simultaneous localization and SLAM mapping were then used to construct a real-time calibration sensor position, eliminating interference from vibrations in the data acquisition equipment. This ensured that the coordinate system of the multimodal features was synchronized with time, allowing for sequential time-series alignment of the multimodal features and ensuring consistency of features across different models. Furthermore, according to the formula:

[0060] The obtained global modal features are dynamically fused to obtain the fused features of anchor bolts in the roadway collapse scenario. In the formula, For the first Each dilated convolution kernel convolves the input feature map. Global modal features corresponding to the convolution operation; Global modal features Dynamic weights, This represents the number of types of dilated convolution kernels. All are positive integers.

[0061] Step S102: The saliency of the fused features is enhanced by a spatiotemporal attention mechanism to obtain the preliminary three-dimensional point cloud data of the anchor bolts in the roadway collapse scene.

[0062] In a scenario of tunnel collapse, the anchor bolts that collapse and become buried may experience positional shifts at different time steps due to equipment vibration, sensor movement, or changes during data acquisition. Or morphological changes (such as twisting or breaking); the surface of the anchor bolt exhibits a complex three-dimensional geometric shape, and the outline of the anchor bolt in the visual data may be buried or partially obscured (the obscured area can account for 50% to 90%).

[0063] In response to the dynamic and multi-scale spatial distribution characteristics of anchor bolts in the aforementioned coal mine roadway collapse scenario, this application introduces a spatiotemporal and spatial joint modeling capability through a spatiotemporal attention mechanism. By dynamically adjusting the weights of time series and spatial features, it solves the problem that traditional attention mechanisms are unable to cope with target loss in complex environments.

[0064] Specifically, dynamic multi-scale convolution is used to process multimodal data in parallel with different hole rates. The multi-scale features are then weighted and fused to generate fused features of anchor bolts in a roadway collapse scenario, capturing the multi-scale spatial details of the anchor bolts and effectively separating them from background noise. A spatiotemporal joint attention mechanism is used to combine time-series and spatial features to enhance the saliency of anchor bolts in complex scenarios.

[0065] In a specific example, a saliency map of the anchor bolt is generated by extracting features from multimodal data of the anchor bolt in a tunnel collapse scenario. This includes geometric saliency based on ultrasonic data, motion saliency based on millimeter-wave radar data, and texture saliency based on binocular vision data. Geometric saliency based on ultrasonic data provides saliency information on the depth and surface set variations of the anchor bolt; motion saliency based on millimeter-wave radar data provides Doppler feature differences between the anchor bolt and the background; and texture saliency based on binocular vision data provides optical boundary and texture information of the anchor bolt surface.

[0066] Here, the saliency map is generated using an attention mechanism model or a feature weighting model. In this process, features are extracted from the multimodal data of the anchor bolts to obtain an initial feature map. This initial feature map is then weighted to generate a saliency weight for each location. Finally, the saliency weights are normalized to generate the anchor bolt saliency map. The saliency map describes the positional saliency of the anchor bolts in a tunnel collapse scenario. Each saliency in the saliency map represents the importance of the corresponding location in the input feature map; salient regions have higher weights (e.g., areas with prominent anchor bolt features), while non-salient regions have lower weights.

[0067] Then, based on the saliency map of the anchor bolts, the fusion features are... Significance enhancement was performed to obtain the enhanced features of the anchor bolts in the scenario of tunnel collapse. Specifically, according to the formula:

[0068] Fusion features Significance enhancement is performed. In the formula, To generate a saliency map for anchor bolts based on the multimodal features obtained from feature extraction. This indicates an element-wise weighted operation. Therefore, through a saliency enhancement mechanism combined with multi-scale feature maps from dilated convolution, the anchor bolt target region is further highlighted, and background interference is suppressed.

[0069] In this embodiment, the fused features are processed through a temporal attention module and a spatial attention module, respectively. Temporal and spatial attention enhancements are implemented. The temporal attention module models time-series features using a bidirectional short-term memory network, extracting dynamic features of the anchor at different time steps. This involves fusing features. The temporal attention features are input into the temporal attention model, that is, the multimodal sensor is applied to the temporal attention model at continuous time steps. Internally acquired time feature sequences ( For multimodal sensors in the first Multimodal features from each time step are input into the temporal attention model for temporal attention enhancement. Then, information from preceding and following time steps is obtained through a bidirectional LSTM to generate temporal attention weights for each time step. Specifically, according to

[0070] Generate temporal attention weights for each time step In the formula, This is a trainable temporal attention weight matrix. For the first The hidden state of the time step.

[0071] Next, based on time attention weights According to the formula:

[0072] For input features Weighting is performed to obtain time-attention enhancement features. .

[0073] In this embodiment, the spatial attention module enhances the salient region of the anchor in the spatial dimension by calculating the salience weight of each pixel in the feature map. This involves fusing features... Spatial attention features are input into the spatial attention model, i.e., the dimension is... ( :high, Feature map (width) (fusion features) The spatial attention features are input into the spatial attention model for spatial attention enhancement; then, convolutional layers are used to learn the feature maps. The significance distribution, specifically, according to

[0074] Determine the feature map pixel position Significance weight at position ; is a trainable spatial attention weight matrix.

[0075] Next, based on pixel position Significance weight at position According to the formula:

[0076] For feature maps Perform pixel-wise weighting to generate spatially enhanced attention features. .

[0077] Next, the temporal attention enhancement features generated by the temporal attention module and the spatial attention module are analyzed. Spatial Enhancement Attention Features According to the formula:

[0078] Weighted fusion is performed to obtain the enhanced features of anchor bolts after spatiotemporal weighted fusion in the scenario of tunnel collapse. In the formula, The fusion coefficient of temporal attention and spatial attention is determined adaptively through dynamic learning of the network.

[0079] Point cloud data is a collection of discrete points in three-dimensional space, used to describe the shape and spatial structure of anchor bolts. In this application, feature enhancement is used... Extract salient features of the anchor bolts (robust visual identifiers that can be reliably detected in complex tunnel environments, specifically geometric and physical features such as the anchor bolt's appearance, distribution, and fracture morphology) to obtain the three-dimensional spatial coordinates of the anchor bolts in a tunnel collapse scenario. This is used to generate preliminary 3D point cloud data of anchor bolts in a tunnel collapse scenario. Here, the position of one or more discrete points of the anchor bolt in 3D space is described by 3D spatial coordinates, with the coordinates of each point represented by... The combination of all three-dimensional spatial coordinates is used to form the preliminary three-dimensional point cloud data of the anchor rod.

[0080] Point cloud generation is the foundation of anchor bolt 3D reconstruction. In this embodiment, preliminary 3D point cloud data is generated by integrating measurement results from multimodal sensors. Specifically, millimeter-wave radar data, ultrasonic data, and binocular vision data are integrated to obtain discrete 3D spatial point clouds of the anchor bolt. Each discrete point contains corresponding 3D spatial coordinates. It may also contain information such as color and reflection intensity, and supplement the deficiencies in cross-view feature association and motion estimation in the discrete three-dimensional spatial point cloud of anchor bolts through inference calculation using three-dimensional positioning functions (such as geometric constraints, optimization algorithms, etc.).

[0081] Here, millimeter-wave radar data is used to generate millimeter-wave point clouds from range-Doppler images. A preliminary 3D point cloud is extracted using radar echo intensity and the metallic reflection characteristics of the anchor bolt (reflection coefficient > 0.9). Here, according to the formula:

[0082] Determine millimeter-wave radar point cloud data In the formula, For the target object (anchor bolt buried by the collapse of the tunnel) The distance from each measurement point to the millimeter-wave radar; The target object is the first The horizontal and vertical angles at each measurement point; For radar signals in millimeter-wave radar and target objects The propagation time between measurement points (i.e., the time from transmitting the signal to receiving the feedback signal). It is the speed of light.

[0083] When generating ultrasonic point clouds from ultrasonic data, the echo characteristics of ultrasound are used to compensate for the insufficient resolution of millimeter-wave radar, especially in the anchor bolt edge region. Specifically, this is achieved by analyzing the phase of the ultrasonic echo, according to the formula:

[0084] Calculate the detailed features of the anchor bolt boundary; where, The phase difference between the echo signal and the transmitted signal of the ultrasonic radar represents the phase change of the ultrasonic wave as it travels to and from the surface of the target anchor bolt. Target distance, representing the one-way distance from the target anchor to the ultrasonic radar probe. This refers to the wavelength of an ultrasonic wave.

[0085] For binocular vision data of anchor bolts, images of the same scene are acquired from different perspectives using two (or more) cameras. Parallax is used to calculate the 3D spatial information of the anchor bolts within the scene, thereby achieving depth perception of the environment and 3D reconstruction of the anchor bolts. Specifically, feature points such as corners and edges are extracted from the camera images. Feature descriptors (such as SIFT and ORB) are used to match corresponding points across views, and then stereo matching methods (triangulation) are combined to calculate the image depth of the target anchor bolt. Specifically, according to the formula:

[0086] Determine the image depth of the target anchor bolt; where, For depth value, The focal length of a visible light camera. The binocular baseline is the straight-line distance between the optical centers of the two cameras in a binocular vision system. Parallax is the offset of the pixel position of an object in two images taken by two cameras due to their different positions.

[0087] In this embodiment, geometric features such as the thread edge and fracture morphology of the anchor bolt are extracted using multiple methods, and depth calculation is performed on the matching feature points using formulas. The texture details (such as rust spots) of the binocular visual point cloud are combined with the sub-millimeter geometric precision of the ultrasonic point cloud to achieve multi-dimensional characterization of anchor bolt defects.

[0088] Then, a visual-ultrasonic coordinate transformation relationship (error less than 0.1mm) is established through a calibration plate to align the ultrasonic point cloud with the binocular visual point cloud. The visual point cloud provides surface texture semantics (such as "rust area"), while the ultrasonic point cloud supplements internal defects (such as edge deformation caused by anchoring agent debonding). This achieves complementary fusion of the ultrasonic point cloud and the binocular visual point cloud, which is convenient for identifying surface cracks (visual) and crack propagation depth (ultrasonic) simultaneously through the fused point cloud in the detection of fatigue cracks in anchor necks.

[0089] Step S103: Surface reconstruction is performed based on the optimized 3D point cloud data of the anchor bolts, and texture mapping is performed on the continuous 3D anchor bolt model generated by the surface reconstruction to obtain an enhanced 3D anchor bolt model.

[0090] The initial 3D point cloud data of the generated anchor bolts may contain noise, redundant points, or missing regions. This application further optimizes the initial 3D point cloud data to effectively ensure the accuracy and reliability of the 3D modeling. On one hand, a spatiotemporal attention mechanism is used to dynamically adjust weights in the time and space dimensions to remove redundant and noise points from the initial 3D point cloud data. According to the formula:

[0091] Point cloud denoising is performed on the preliminary 3D point cloud data; where, This represents the spatiotemporal fused point cloud data after removing redundant and noise points. These are point cloud data in the time dimension and point cloud data in the spatial dimension of the preliminary three-dimensional point cloud data, respectively. Temporal and spatial attention weights are used for point cloud denoising of preliminary 3D point cloud data.

[0092] In coal mine roadways, anchor bolts are often obscured by gravel and coal dust, with obstruction areas reaching 50% to 90%, resulting in severe deficiencies in the initial 3D point cloud data. In this embodiment, a mobile binocular vision system is used to collect multi-view data of the anchor bolts, performing a coverage scan of the obscured areas to compensate for the data loss from traditional single-view scanning (60% point cloud coverage), effectively improving the point cloud coverage of the anchor bolts (after multi-view fusion, the point cloud coverage can reach over 95%). For the small portion of missing point cloud areas due to obstruction, this embodiment performs point cloud denoising on the initial 3D point cloud data to obtain spatiotemporally fused point cloud data after removing redundant and noise points. Subsequently, a point cloud interpolation algorithm based on non-local means was used to repair the missing point cloud regions caused by occlusion in the initial 3D point cloud data, i.e., to repair the spatiotemporally fused point cloud data. The missing point cloud regions due to occlusion. Specifically, according to the formula:

[0093] Occlusion area repair is performed on the preliminary 3D point cloud data; where, The target interpolation point is the three-dimensional coordinate point within the area to be repaired (such as the theoretical location of the missing thread). For similar regions, spatiotemporal fusion point cloud (i.e., spatiotemporal fusion point cloud data) ) in and Three-dimensional points that are similar in the neighborhood (such as points on the same threaded ring or points on the smooth surface of a rod). Points Point The contribution coefficient reflects the point Neighborhood and points The degree of geometric similarity of the neighborhood For point-based Neighborhood and points Weights of neighborhood similarity For the neighborhood of the point cloud.

[0094] By leveraging the similarity and geometric characteristics of point clouds to interpolate lost points, intelligent repair is performed on a small number of point cloud missing areas caused by occlusion, improving the point cloud integrity of anchor bolts (the point cloud integrity after repair of occluded areas is improved by 30%~40%) and reducing the reconstruction error of anchor bolt edges (the reconstruction error is reduced from 5 mm to 1 mm).

[0095] After denoising and repairing occluded areas in the initial 3D point cloud data, the physical constraints of the anchor bolts are used to optimize the initial 3D point cloud data, generating optimized 3D point cloud data for the anchor bolts. Specifically, based on the geometry and material properties of the anchor bolts, the optimization is performed according to the formula:

[0096] Global optimization is performed on the initial 3D point cloud data; where, For the initial 3D point cloud of the anchor bolt, To generate an optimized 3D point cloud, These are the weighting coefficients. This is a regularization term used to constrain the shape of the anchor bolt. This represents the loss between the predicted and actual positions of the anchor bolts under physical constraints.

[0097] In a specific example, an adversarial training framework based on the physical constraints of anchor bolts (the geometry and material properties of the anchor bolts) is used to construct a generative adversarial network model to generate a fake distribution of anchor bolts. Specifically, the geometry and material properties of the anchor bolts (such as metal reflectivity and structural stability) are incorporated into the model training phase to guide the model in optimizing the reasonableness of the prediction results during training.

[0098] In the generative adversarial network model, the geometry and spatial distribution of the anchor bolts are used as constraints to define the loss function:

[0099] In the formula, To generate the anchor positions predicted by the adversarial network model, This represents the actual position of the anchor bolt.

[0100] In generative adversarial network models, noise is... Input generator ( ), generating fake anchor bolt distribution data Through the discriminator ( Determine the authenticity of the anchor bolt prediction results, that is, determine whether the input data is real anchor bolt data. Or is the anchor bolt distribution data fabricated? Specifically, according to the formula:

[0101] Determine the authenticity of the anchor bolt prediction results; where, The output of the discriminator, ， Characterizing anchor bolt data The probability of actual anchor bolt data. The mathematical expectation is used to measure the overall accuracy of the judgment. The goal is to optimize the anchor bolts using a discriminator.

[0102] In this embodiment, after obtaining the optimized 3D point cloud data of the anchor bolt, the Marching Cubes algorithm is used according to the formula:

[0103] Surface reconstruction is performed on optimized 3D point cloud data to generate a continuous 3D anchor bolt model; where, For the set of vertices of the generated 3D anchor model; To optimize any point in 3D point cloud data; It is an implicit surface function.

[0104] Then, based on the continuous 3D anchor model generated by surface reconstruction, the surface texture features obtained from binocular vision data in multimodal data are mapped onto the surface of the 3D anchor model to enhance its visualization effect, resulting in an enhanced 3D anchor model. In a specific application scenario, firstly, binocular vision texture features are extracted from the anchor through stereo matching and feature enhancement. Specifically, depth information (disparity map) of the anchor surface is obtained through binocular vision calculation, and the 2D image pixels are converted into 3D spatial points by combining camera parameters. ,in The texture color value at that point is used; key points (such as edges and corners) in the texture are extracted using algorithms such as SIFT and SURF, or semantic texture features (such as details like cracks and rust) are extracted using CNN networks to improve the recognizability of the texture.

[0105] Next, the surface of the 3D anchor model is parameterized. This involves UV mapping, where the surface vertices of the 3D anchor model are mapped to a 2D UV coordinate system (unfolded into a plane), with each vertex corresponding to a... Coordinates are used to form a texture coordinate mapping relationship. A unique coordinate is assigned to each vertex of the 3D anchor model. Coordinates are used to associate the vertices of the 3D anchor model with texture coordinates, ensuring the continuity of texture mapping. Here, since the anchor is slender, cylindrical projection or piecewise parameterization can be used to process the anchor to avoid texture distortion in curved parts (such as unfolding it into a rectangular UV map along the axis of the rod).

[0106] Next, through world coordinate transformation and projection matching, the texture coordinates are spatially aligned with the 3D anchor model. Specifically, based on world coordinate transformation, the texture points (containing 3D coordinates and color values) extracted from the binocular vision are transformed from the camera coordinate system to the world coordinate system of the 3D anchor model, ensuring consistency with the model's vertex positions. Using the camera projection matrix, the vertices of the 3D anchor model are projected onto the binocular image plane to find the corresponding pixel positions and obtain the texture values ​​(color, detail) at those positions. Projection matching can be achieved through nearest neighbor mapping, bilinear interpolation, and perspective-corrected interpolation.

[0107] In another specific application scenario, the visualization of a 3D anchor model is enhanced through operations such as texture blending and seam treatment, lighting and shadow model overlay, detail enhancement, and noise reduction. Specifically, multi-view texture blending processes overlapping areas of textures from different perspectives, and adjusting the UV unwrapping cut positions reduces texture splicing marks, achieving texture blending and seam treatment for the 3D anchor model. When overlaying lighting and shadow models on the 3D anchor model, generating normal vectors for the 3D anchor model surface and combining them with texture colors to calculate lighting effects (such as the Phong lighting model) enhances the three-dimensionality of the texture; adding ambient light and reflection maps (such as skyboxes) simulates lighting reflections in a real scene, improving the model's realism.

[0108] When enhancing details and processing noise in anchor bolt models, if the resolution of the binocular image is insufficient, the texture clarity can be improved by using super-resolution algorithms (such as SRCNN), or the texture sampling accuracy can be adjusted by combining point cloud density to achieve high-resolution texture mapping. At the same time, median filtering or bilateral filtering can be applied to noise points in the texture (such as abnormal colors caused by binocular matching errors) to remove interference while preserving details.

[0109] Step S104: Perform deformation detection on the enhanced three-dimensional anchor bolt model to determine the fracture area of ​​the anchor bolt.

[0110] Based on the enhanced 3D anchor bolt model obtained from the 3D reconstruction of the anchor bolt, the dynamic behavior of the anchor bolt is further perceived and analyzed through deformation detection and hazard assessment. Specifically, when detecting deformation of the anchor bolt using the enhanced 3D anchor bolt model, a standard model of the anchor bolt in its original state is constructed, and the enhanced 3D anchor bolt model is compared with the standard anchor bolt model to determine the geometric deviation between them. Furthermore, the deformation range of the anchor bolt under the tunnel collapse scenario is determined by using the geometric deviation between the enhanced 3D anchor bolt model and the standard anchor bolt model. It generates a saliency map of anchor bolt deformation in a roadway collapse scenario, and captures geometric anomalies (such as bending, collapse or abrupt changes) on the anchor bolt surface that are precursors to anchor bolt fracture by the saliency of the deformation area.

[0111] Wherein, according to the formula:

[0112] Calculate the deformation amplitude at each mesh point between the enhanced 3D anchor model and the standard anchor model. In the formula, To enhance the mesh point coordinates of the 3D anchor bolt model, These are the coordinates of the corresponding grid points in the standard anchor bolt model.

[0113] By enhancing the geometric deviation between the 3D anchor bolt model and the standard anchor bolt model, the deformation range of the anchor bolt under the scenario of tunnel collapse was determined. This method generates a saliency map of anchor bolt deformation in a roadway collapse scenario. By capturing the saliency of the deformation region, it identifies geometric anomalies (such as bending, collapse, or abrupt changes) on the anchor bolt surface that are precursors to anchor bolt fracture, thereby enhancing the mesh point deformation amplitude between the 3D anchor bolt model and the standard anchor bolt model. The larger the value, the higher the significance of the corresponding region.

[0114] Then, regarding the deformation amplitude Normalization is performed according to the formula:

[0115] Generate a significant deformation diagram of the anchor bolt; where, Indicates position Significance of deformation at the location.

[0116] Simultaneously, based on the infrared thermal image of the anchor bolts generated by the infrared thermal imager in the tunnel collapse scenario, the temperature distribution matrix of the anchor bolts is extracted, and the local gradient of the temperature distribution of the anchor bolts in the tunnel collapse scenario is calculated based on the temperature distribution matrix to generate a temperature saliency map of the anchor bolts in the tunnel collapse scenario. The calculation follows the formula:

[0117] Identify the temperature anomaly regions corresponding to areas with large temperature gradients in the infrared thermal image; where, In infrared thermal image Temperature gradient at that location; In infrared thermal image The temperature abruptly changed at the location. Next, the temperature anomaly region was normalized, i.e., the temperature abruptly changed at the temperature abruptly changed. Normalization is performed, specifically according to the formula:

[0118] The temperature significance diagram of the generated anchor bolts; where, Indicates position The significance of temperature at that location; Location in infrared thermal image The temperature at which the mutation occurs.

[0119] Then, the deformation significance map and temperature significance map of the anchor bolts in the tunnel collapse scenario are weighted and fused to generate a comprehensive significance map of the anchor bolts in the tunnel collapse scenario. Specifically, according to the formula:

[0120] Position Significance of anchor bolt deformation Significance with temperature Perform weighted fusion to obtain the position. Comprehensive significance of anchor bolts In the formula, For position The significance weight of the deformation of the anchor bolt. For position The significance weight of temperature at the anchor bolt.

[0121] Finally, the comprehensive saliency map of the anchor bolts in the roadway collapse scenario is mapped onto the surface mesh points of the enhanced 3D anchor bolt model. Through the deformation degree analysis of the surface mesh points of the enhanced 3D anchor bolt model, the saliency marking areas exceeding the preset saliency threshold are identified as the fracture areas of the anchor bolts.

[0122] When assessing the potential hazards of anchor bolts in collapse scenarios using an enhanced 3D anchor bolt model, the stress distribution of the anchor bolts under roadway collapse scenarios is determined using the finite element method based on the enhanced 3D anchor bolt model. The mechanical stability of the anchor bolts under roadway collapse scenarios is evaluated based on the material characteristics of the anchor bolts, and possible fracture points are predicted. Simultaneously, by analyzing the load-stress relationship of the anchor bolts under roadway collapse scenarios, potential failure areas of the anchor bolts are predicted. Areas where the anchor bolt stress is less than 50% of the material yield strength are identified as risk-free areas; areas where the anchor bolt stress is between 50% and 80% of the material yield strength are identified as medium-risk areas; and areas where the anchor bolt stress is greater than 80% of the material yield strength are identified as high-risk areas. Furthermore, by combining the spatial distribution and morphology of the anchor bolts, a full-process coverage from dynamic perception to hazard prediction is achieved, helping to assess the stability of roadway rescue channels, providing a scientific basis for roadway rescue, and adapting to the real-time monitoring and rescue needs in coal mine roadways.

[0123] This application addresses the challenge of accurately locating the shape and spatial position of buried anchor bolts in tunnel collapse scenarios. It utilizes multimodal sensors to generate 3D point cloud data of the anchor bolts, effectively overcoming the limitations of single sensors in complex collapse environments. Furthermore, through a spatiotemporal attention mechanism, it dynamically captures the salient regions of the buried anchor bolts in time and space, removing noise points from the 3D point cloud data and effectively separating environmental interference such as gravel and soil, thus enhancing the geometric feature representation of the anchor bolts. Simultaneously, the anchor bolt's geometry and material properties are incorporated as physical constraints into the 3D point cloud optimization process, significantly improving the point cloud quality and ensuring its integrity and accuracy. This enables stable performance in unstructured environments (high noise, low visibility, dusty environments), exhibiting strong noise resistance and stable extraction of the anchor bolt's spatial pose features. Based on the 3D reconstructed point cloud model, the stress distribution and stability of the anchor bolts are evaluated, achieving full-process coverage from dynamic perception to hazard prediction, providing a scientific basis for tunnel rescue.

[0124] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0125] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for three-dimensional positioning of anchor bolts in a roadway collapse scenario based on a dynamic multi-scale void attention network, characterized in that, include: Based on a dynamic perception feature extraction mechanism, dynamic multi-scale dilated convolution operation is performed on the multimodal data of buried anchor bolts in a roadway collapse scenario. The dilation rate of the dilated convolution kernel is dynamically adjusted according to the saliency distribution of the input feature map. Multimodal features of anchor bolts in a roadway collapse scenario are extracted and fused to generate fused features of anchor bolts in a roadway collapse scenario. The multimodal data includes: ultrasonic data, millimeter-wave radar data, and binocular vision data. By using a spatiotemporal attention mechanism to significantly enhance the fusion features, preliminary three-dimensional point cloud data of anchor bolts in a roadway collapse scenario are obtained. Physical constraints are constructed based on the geometry and material properties of the anchor bolts, and the preliminary three-dimensional point cloud data is optimized to obtain optimized three-dimensional point cloud data. Surface reconstruction is performed based on optimized 3D point cloud data of anchor bolts, and texture mapping is performed on the continuous 3D anchor bolt model generated by surface reconstruction to obtain an enhanced 3D anchor bolt model. Deformation detection was performed on the enhanced 3D anchor bolt model to determine the fracture area of ​​the anchor bolt; Specifically, based on the geometric deviation between the enhanced 3D anchor bolt model and the standard anchor bolt model, the deformation amplitude of the anchor bolt in the roadway collapse scenario is determined to generate a deformation saliency map of the anchor bolt in the roadway collapse scenario. Based on the temperature distribution matrix extracted from the infrared thermogram of the anchor bolt in the roadway collapse scenario, the local gradient of the temperature distribution of the anchor bolt in the roadway collapse scenario is calculated to generate a temperature saliency map of the anchor bolt in the roadway collapse scenario. The deformation significance map and temperature significance map of the anchor bolt in the roadway collapse scenario are weighted and fused to generate a comprehensive significance map of the anchor bolt in the roadway collapse scenario. The comprehensive saliency map of the anchor bolts in the roadway collapse scenario is mapped onto the surface mesh points of the enhanced 3D anchor bolt model, and the saliency marker areas exceeding the preset saliency threshold are identified as the fracture areas of the anchor bolts.

2. The method according to claim 1, characterized in that, Based on a dynamic perception-based feature extraction mechanism, features are extracted from multimodal data of anchor bolts in a roadway collapse scenario. Then, based on a spatiotemporal attention mechanism, the extracted multimodal features are fused to generate fused features of the anchor bolts in the roadway collapse scenario, including: Multimodal features are sequentially aligned with time series and dynamically fused to obtain the fused features of anchor bolts in the roadway collapse scenario; Specifically, based on the saliency distribution of the input feature map, the porosity of the dilated convolution kernel is dynamically adjusted to obtain the dynamic porosity of the corresponding dilated convolution kernel; and based on the dynamic porosity of the dilated convolution kernel and the saliency value of the input feature map, the multimodal features of the anchor bolt in the roadway collapse scenario are determined.

3. The method according to claim 2, characterized in that, The ultrasonic data were analyzed by multi-scale weighted window and short-time Fourier transform respectively to extract the multi-scale energy of the anchor bolt in the roadway collapse scenario. The extracted multi-scale energy was then fused to obtain the first edge feature of the anchor bolt in the roadway collapse scenario. Doppler features of anchor bolts in a roadway collapse scenario are extracted from millimeter-wave radar data, and enhanced Doppler features are obtained by dynamic multi-scale dilated convolution. Image inpainting is performed on multi-view binocular vision data based on nonlocal mean filtering to extract the second edge features and surface texture features of anchor bolts in the roadway collapse scene.

4. The method according to claim 3, characterized in that, According to the formula: Determine anchor bolt sampling points in the scenario of tunnel collapse. Use the first Weighted energy eigenvalues ​​obtained by weighting windows at each scale In the formula, For the first The window length of a scale-weighted window. For the first The scale-weighted window at the th ... Position weight, This is the position index within the scale-weighted window. ; For the ultrasonic echo signal at the 1st The amplitude of each sampling point; For sampling points The weighted energy feature values ​​obtained from weighted windows at various scales are fused to obtain the anchor bolt sampling points in the roadway collapse scenario. eigenvectors at location Among them, the feature vector The first boundary features of the anchor bolt are characterized in the scenario of tunnel collapse; The phase information of the ultrasonic echo signal is extracted by short-time Fourier transform, and the phase transition point is detected to obtain the second boundary features of the anchor bolt in the roadway collapse scenario. The first and second boundary features of the anchor rod in the roadway collapse scenario are fused to obtain the first edge feature of the anchor rod in the roadway collapse scenario.

5. The method according to claim 2, characterized in that, According to the formula: Determine the first Dynamic void ratio of each voided convolution kernel In the formula, Input feature map Middle position The significance value at the location, The maximum and minimum dilatation rates during dynamic multi-scale dilated convolution are respectively determined. According to the formula: Determine the location of anchor bolts in a tunnel collapse scenario From the first Local feature values ​​extracted by dilated convolution kernels In the formula, For the first The global porosity of a dilated convolution kernel. , For the first The weights of each dilated convolution kernel, Input feature map At dynamic convolution sampling points Eigenvalues ​​at; This is a two-dimensional coordinate index for the dilated convolution kernel, representing the position of the weights within the kernel. Input feature map The two-dimensional spatial coordinates represent the center position of the current convolution operation. ， The receptive field radius of the dilated convolution kernel; It is a positive integer; Anchor bolt location in a tunnel collapse scenario From the first Local feature values ​​extracted by dilated convolution kernels Perform aggregation operation to obtain the first... Each dilated convolution kernel convolves the input feature map. Global modal features corresponding to convolution operations .

6. The method according to claim 2, characterized in that, According to the formula: The obtained global modal features are dynamically fused to obtain the fused features of anchor bolts in the roadway collapse scenario. ; In the formula, For the first Each dilated convolution kernel convolves the input feature map. Global modal features corresponding to the convolution operation; Global modal features Dynamic weights, The number of dilated convolution kernel types. All are positive integers.

7. The method according to claim 1, characterized in that, A saliency map of anchor bolts is generated by extracting features from multimodal data of anchor bolts in a roadway collapse scenario; the saliency map represents the positional saliency of anchor bolts in a roadway collapse scenario. Based on the saliency map of the anchor bolts, the fusion features are analyzed. Significance enhancement was performed to obtain the enhanced features of the anchor bolts in the scenario of tunnel collapse. ; Based on enhanced features Determine the three-dimensional spatial coordinates of the anchor bolt in a tunnel collapse scenario. To generate preliminary 3D point cloud data of anchor bolts in a roadway collapse scenario.

8. The method according to claim 1, characterized in that, Based on the geometry and material properties of the anchor bolt, the preliminary 3D point cloud data is globally optimized to obtain optimized 3D point cloud data. The Marching Cubes algorithm is then used to reconstruct the surface of the optimized 3D point cloud data to generate a continuous 3D anchor bolt model. The surface texture features obtained from binocular vision data in multimodal data are mapped onto the 3D anchor model to obtain the enhanced 3D anchor model.

9. The method according to claim 1, characterized in that, Also includes: Based on the enhanced three-dimensional anchor bolt model, the stress distribution of the anchor bolt under the roadway collapse scenario is determined by the finite element method, and the mechanical stability of the anchor bolt under the roadway collapse scenario is evaluated according to the material characteristics of the anchor bolt. By analyzing the load and stress relationship of anchor bolts in a roadway collapse scenario, the potential failure area of ​​anchor bolts in such a scenario is predicted.

Images