Roof body sensing method of roadway repair robot based on time information fusion

Abstract

The invention discloses a roadway repair robot roof body sensing method based on time information fusion, and relates to the technical field of roadway repair robots. A multi-sensor system is used for collecting roadway environment images and depth information, and a time sequence data sequence containing roof fall characteristics is constructed; dividing a training set and a test set after preprocessing such as synchronous calibration and de-noising; constructing a time information fusion model containing a residual space-time feature extraction module, a proxy attention mechanism feature focusing module and a hyper-parameter optimization module, performing optimization through the hyper-parameter optimization module, and then extracting and focusing roof fall key features; and through model training and testing, roof fall type key feature information and robot three-dimensional accurate positioning information are output. The method can accurately extract the roof caving features in the underground severe environment, gives consideration to the processing efficiency and the anti-interference capability, improves the sensing and positioning precision of the robot, meets the requirements of complex roadway repair operation, and is remarkable in practicability and popularization value.

Description

Technical Field

[0001] This invention relates to the field of tunnel repair robot technology, specifically to a method for roof embodied perception of tunnel repair robots based on time information fusion. Background Technology

[0002] Underground coal mine roadways are the only core passageways for underground coal mining, personnel passage, and equipment transportation. Their structural stability is a prerequisite for underground operational safety and production efficiency. Roadways are prone to roof falls due to high ground stress, strong mining disturbances, and complex geological environments. These falls not only cause roof collapses and rockfalls, but also directly lead to serious safety accidents such as personnel casualties and equipment damage. Manual repair of roadways with roof falls requires working deep into the high-risk area after the collapse, facing the safety hazards of secondary roof collapses and falling rocks, resulting in extremely low efficiency. To avoid the safety risks of manual repair and improve repair efficiency, roadway roof fall repair robots are gradually becoming core equipment to replace dangerous manual operations. However, the current application of roadway roof fall repair robots still faces key technological bottlenecks:

[0003] (1) Harsh environmental interference makes core feature extraction difficult: When the roof collapse repair robot repairs deformed roof collapses in the narrow mining roadway in deep underground mines, it is necessary to accurately identify the types of roof collapses such as roof cracks, roof bending, roof collapse, and roof settlement. Due to the harsh environment of the narrow mining roadway in deep underground mines, there are multiple interference problems such as dust, water mist and light brightness in the image data collected by various sensors. Dust will absorb and diffuse light through Mie scattering, causing the image to be fogged and blurred, which makes it impossible for the existing feature extraction model to extract key features such as rock mass cracks and anchor bolt outlines; water mist will form a water film or produce light refraction on the surface of the sensor lens, causing local image occlusion and feature shape distortion, which makes it difficult for the existing feature extraction model to accurately judge detailed features such as the fit between the tray and the surrounding rock; due to the limited illumination range, the lighting has an uneven phenomenon of strong light reflection and dark areas coexisting, and the image grayscale gradient is reduced, which makes it impossible for the existing feature extraction model to clearly distinguish features such as roof bending arc and settlement displacement.

[0004] (2) Inaccurate environmental perception of robots: In complex roof collapse environments, it is difficult to accurately perceive the boundaries of the collapsed area and the location of the stable support structure. At the same time, it lacks reliable autonomous positioning capabilities and cannot determine its precise position in the roadway. Summary of the Invention

[0005] The purpose of this invention is to propose a roof embodied perception method for tunnel repair robots based on time information fusion, so as to solve the problems existing in the prior art.

[0006] The technical solution adopted in this invention is: a roof embodied perception method for a tunnel repair robot based on time information fusion, comprising the following steps:

[0007] Step S1: Use a multi-sensor system to collect images and depth information of the tunnel environment and construct a time-series data sequence containing roof fall characteristics; the multi-sensor system includes a high-resolution explosion-proof camera, a lidar and a displacement sensor. The high-resolution explosion-proof camera is used to collect image data of the roof rock layer and the support structure. The lidar is used to collect three-dimensional contour data of the roof rock layer and the support structure. The displacement sensor is used to collect displacement data of the roof rock layer and the support structure.

[0008] Step S2: Preprocess the collected time-series data sequence, including synchronization calibration, noise reduction and data scale normalization, and then divide it into 80% training set and 20% test set;

[0009] Step S3: Construct a time information fusion model, which includes a residual spatiotemporal feature extraction module, a proxy attention mechanism feature focusing module, and a hyperparameter optimization module. First, the hyperparameter optimization module uses the barrel theory optimization algorithm to adaptively optimize the hyperparameters of the network model. Then, the residual spatiotemporal feature extraction module extracts features, the proxy attention mechanism feature focusing module focuses on key features, and finally outputs key feature information of the roof fall type and robot 3D precise positioning information.

[0010] Step S4: Use the local features and long-term dependent features related to the top of the fall in the training set as input to the network model and the key features of the top of the fall as output to train the model;

[0011] Step S5: Input the test set data into the trained time information fusion model. Through the autonomous learning process of the time information fusion model, accurately extract the feature information of the roof fall type from the roof fall image data of the mining roadway, so as to realize the roadway repair robot's accurate identification of roof fall type and accurate navigation requirements for repair operation.

[0012] As a further improvement of the present invention, in step S1, the characteristics of the roof fall include the width and orientation of the cracks in the roof rock layer, the amount of roof delamination, the degree of bending and fracture of the anchor bolts, and the fit between the support plate and the surrounding rock.

[0013] As a further improvement of the present invention, in step S3, the residual spatiotemporal feature extraction module uses the temporal convolutional network (TCN) as its core technology support, extracts its dilated causal convolution and residual block core components, and integrates them with wide kernel convolutional layers, pooling layers, flattening layers and connection layers in a customized manner to construct a spatiotemporal feature mining architecture adapted to the complex downhole environment. The specific working method is as follows:

[0014] (1) Wide kernel convolutional layers perform preliminary feature capture on the original roof fall images acquired by high-resolution explosion-proof cameras;

[0015] (2) The residual block receives the feature map output by the wide kernel convolutional layer and achieves detailed feature preservation through the cross-layer information transmission structure;

[0016] (3) The dilated causal convolutional layer performs temporal correlation mining on the feature map output by the residual block;

[0017] (4) The pooling layer, flattening layer and fully connected layer successively reduce the dimension, flatten and integrate the features output by the dilated causal convolutional layer. The final output includes the anchor bolt outline, pallet fit, local details of small cracks, curvature of the top plate, global outline of the large-scale collapse area, and the time-series correlation of the dynamic change law of features, providing high-quality data support for the subsequent proxy attention mechanism feature focusing module.

[0018] As a further improvement of the present invention, the Temporal Convolutional Network (TCN) achieves accurate capture of the spatiotemporal features of the collapse peak through adaptive feature extraction. Its convolution calculation satisfies the causal dilated convolution form and includes residual connections. The specific adaptive calculation process is as follows:

[0019] (1) Use a one-dimensional sequence input x∈Rn and a filter:

[0020] ;

[0021] (2) Apply the following dilation convolution operation to element s:

[0022] ;

[0023] in, It is the dilated convolution operator. It is the expansion factor. It is the size of the filter;

[0024] (3) The input sequence is:

[0025] ;

[0026] (4) The output sequence after three layers of one-dimensional dilated causal convolution with a kernel size of 3 is:

[0027] ;

[0028] The dilation coefficient d∈N* in convolution calculation.

[0029] (5) Experience the wild The value is related to the size of the convolution kernel, the number of layers in the convolution calculation, and the dilation coefficient. Its calculation formula is:

[0030] ;

[0031] in, The size of the convolution kernel. The number of convolutional layers in the network. As the base of the expansion coefficient, set ;

[0032] (6) In a temporal convolutional network, let the one-dimensional input sequence be... The convolution kernel is The result of the dilated causal full convolution at position s in sequence is:

[0033] ;

[0034] in, The first in the previous layer element.

[0035] As a further improvement of the present invention, in step S3, the working method of the proxy attention mechanism feature focusing module is as follows:

[0036] (1) Introducing a proxy token matrix: The number of new proxy token matrices is much smaller than that of query tokens, reducing computational redundancy and focusing on core feature regions;

[0037] (2) Proxy token aggregation: Proxy tokens are used as query and key-value pairs for attention calculation, which aggregates the value information related to the key features of the global top and reduces interference.

[0038] (3) Proxy token broadcasting: The proxy token interacts with the query token as a key, broadcasting the aggregated global information back to each query token to complete feature details, correct distorted forms, and strengthen the consistency of key features across the entire domain;

[0039] (4) Output updated query token: Obtain the query token containing global information updates, complete key feature focusing and redundancy interference suppression, and accurately extract the feature information of the top collapse type.

[0040] As a further improvement of the present invention, the working method of the hyperparameter optimization module is as follows:

[0041] (1) Initialize the temporal information fusion model: Set the kernel size of the model's wide kernel convolution and dilated causal convolution;

[0042] (2) Set the range of weight allocation parameters, learning rate, regularization coefficient, number of iterations and batch size; set the population size, convergence factor, update probability boundary and maximum number of iterations for the barrel theory optimization algorithm;

[0043] (3) Calculate the fitness value of all individuals in the population. The position of each individual in the population corresponds one-to-one with the hyperparameters of the model. Sort them according to their fitness. Select RMSE and MAE as the fitness functions of the barrel theory optimization algorithm.

[0044] (4) Determine the update probability of each dimension for each individual in the population: individuals with lower fitness correspond to the weakest link and are assigned a higher update probability. The update of hyperparameter dimensions is completed by combining harmonic perturbation and the hyperparameters are pulled back to the effective range through the boundary control mechanism. Individuals with higher fitness have a lower update frequency in order to retain good information and maintain population stability.

[0045] (5) Determine whether the barrel theory optimization algorithm has finished iterating. When the maximum number of iterations is reached, the algorithm iteration is completed. At this time, the position of the population individual with the best fitness is assigned to the time information fusion model. If the barrel theory optimization algorithm iteration has not finished, return to the previous step.

[0046] As a further improvement of the present invention, in step (3), the calculation methods for RMSE and MAE are as follows:

[0047] ;

[0048] ;

[0049] in, Let RUL be the error between the actual RUL and the predicted RUL at time t. Duration.

[0050] As a further improvement of the present invention, in step (4), the update probability is calculated as follows:

[0051] ;

[0052] in, and and represent the lower and upper bounds of the dynamic update probability range, respectively. This range controls the frequency with which the solution variables are adjusted during the search process; in this formula, Indicates the first The normalized fitness of an individual, with a value in the range [0,1], is used to measure the individual's superiority or inferiority relative to other individuals in the current population; the normalization formula is as follows:

[0053] ;

[0054] in Indicates the first The initial fitness value of each individual, and These represent the minimum and maximum fitness values ​​in the current population, respectively.

[0055] Compared with the prior art, the present invention has the following technical advantages:

[0056] 1. Addressing the complex environments of underground mining, including low illumination, high dust levels, and water mist interference, this invention's residual spatiotemporal feature extraction module utilizes a Temporal Convolutional Network (TCN) as its core support. It initially captures global features through wide-kernel convolutional layers, preserves detailed features through cross-layer transfer of residual blocks, and mines temporal correlations using dilated causal convolutional layers, achieving accurate capture of spatiotemporal coupling features in roof collapse scenarios. Even under conditions of image fogging, local occlusion, and reduced grayscale gradients, it can still efficiently extract local detailed features such as rock fracture width and orientation, anchor bolt contours, and pallet fit, as well as global contour features such as roof curvature and collapse area boundaries. Simultaneously, it captures dynamic information such as fracture propagation trajectories and subsidence trends through temporal correlation analysis, completely resolving the pain points of existing technologies, such as difficulty in core feature extraction and susceptibility to environmental interference, providing high-quality and highly reliable feature support for subsequent identification and localization.

[0057] 2. To address the issues of high computational complexity and spatiotemporal asynchrony when fusing multi-source data such as laser, vision, and displacement, the proxy attention mechanism's feature focusing module innovatively introduces a proxy token matrix. Through a two-step interaction of aggregation and broadcasting, it achieves key feature enhancement and redundancy interference suppression: On the one hand, the number of proxy tokens is much smaller than that of query tokens, reducing the computational complexity of attention from O(N²) to O(N×M) (where M is the number of proxy tokens), significantly improving the processing efficiency of high-resolution images and multimodal data, and avoiding the impact of excessive computation time on the robot's real-time operation response; on the other hand, through global information aggregation and global broadcasting, it completes the features of small cracks in dark areas, corrects morphological distortion caused by water mist, strengthens the global consistency of key features, and effectively weakens redundant signals caused by dust and uneven lighting, balancing feature representation capability and anti-interference capability. This solves the defects of low efficiency and insufficient linear attention feature representation in traditional attention mechanisms, significantly improving the accuracy of roof fall type recognition and environmental adaptability.

[0058] 3. This invention provides the robot with comprehensive environmental perception by combining the precise key feature information of roof fall type output by the feature extraction and focusing module with the three-dimensional contour and displacement data collected by lidar and displacement sensors. This enables the robot to accurately identify specific types of roof falls such as cracks, bending, collapse, and settlement, and to clearly define the boundaries of the collapse area and the location of stable support structures. Simultaneously, using the extracted spatiotemporal coupling features as constraints, and combining them with a hyperparameter-optimized positioning algorithm, the robot achieves precise three-dimensional positioning within the tunnel, solving the problems of inaccurate environmental perception and low reliability of autonomous positioning in existing technologies. This integrated output of identification and positioning provides core data support for the robot's autonomous decision-making regarding attachments and planning of safe repair paths, effectively avoiding safety risks such as secondary collapses and falling debris faced by manual repair work, and building a solid technical defense for the safety of tunnel repair operations.

[0059] 4. The hyperparameter optimization module employs a "barrel theory" optimization algorithm, globally optimizing key hyperparameters such as the convolution kernel size and dilation coefficient of the residual spatiotemporal feature extraction module, the weight allocation parameters of the surrogate attention mechanism, and the overall learning rate, regularization coefficient, and number of iterations. This algorithm dynamically allocates update probabilities, focusing on optimizing weaker parameters while preserving superior information in advantageous parameters. It boasts fast convergence and strong optimization capabilities, ensuring the full utilization of the technical advantages of both the residual spatiotemporal feature extraction and surrogate attention modules, thus maximizing model performance. Compared to existing systems lacking global optimization, the network model of this invention maintains stable and efficient operational performance in underground scenarios with varying geological conditions and interference intensities. Environmental adaptability and overall operational efficiency are significantly improved, fully meeting the complex and ever-changing needs of underground coal mine roadway repair operations.

[0060] 5. The technical solution of this invention is based on a standardized data acquisition process of a multi-sensor system. The preprocessing steps are simple and efficient, the network model structure is clear and highly portable, and there is no need for large-scale hardware modifications to existing tunnel repair robots. A leapfrog improvement in perception capabilities can be achieved simply through algorithm upgrades. It not only effectively improves the automation level and operational efficiency of tunnel repair, reduces labor costs and safety risks, but also provides a solid technical foundation for the subsequent expansion of tunnel repair robots' autonomous operation and path planning functions. It has broad application prospects in underground engineering scenarios such as underground coal mines and tunnel projects, possessing both significant technical and practical value. Attached Figure Description

[0061] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Figure 1 This is a flowchart of the roof embodied perception method for a tunnel repair robot based on time information fusion, according to the present invention.

[0063] Figure 2 This is a schematic diagram of the time information fusion framework structure of the present invention.

[0064] Figure 3 This is a schematic diagram of the dilated causal convolution structure in the residual spatiotemporal feature extraction module.

[0065] Figure 4 This is a schematic diagram of the proxy attention mechanism structure in the proxy attention feature focusing module. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention are within the scope of protection of the present invention.

[0067] Please refer to Figure 1 The present invention provides a roof embodied perception method for a tunnel repair robot based on time information fusion, comprising the following steps:

[0068] Step S1: Acquire images and depth information of the tunnel environment using a multi-sensor system to construct a time-series data sequence containing roof fall characteristics. The multi-sensor system includes a high-resolution explosion-proof camera, a lidar, and a displacement sensor. The high-resolution explosion-proof camera is used to acquire image data of the roof strata and support structure, while the lidar and displacement sensor are used to acquire the three-dimensional contour and displacement data of the roof. The high-resolution explosion-proof camera is a Hikvision intrinsically safe mining surveillance camera, the lidar is a DJI LiDAR, and the displacement sensor is a Photonics Precision PDH type laser displacement sensor. Specifically, the characteristics of roof collapse include the width and orientation of cracks in the roof rock strata, the amount of roof delamination, the degree of bending and fracture of anchor bolts, and the fit between the support plate and the surrounding rock. Among them, the width and orientation of cracks in the roof rock strata refer to the vertical distance between the two walls of the cracks in the roof rock mass and the horizontal direction of the crack extension. The amount of roof delamination is the relative displacement value between different rock strata in the roof or between the rock strata and the anchor bolt anchoring structure. The degree of bending and fracture of anchor bolts refers to the amplitude of bending deformation and the location and extent of fracture of the anchor bolt after being subjected to force. The fit between the support plate and the surrounding rock refers to the degree of contact between the anchor bolt support plate and the surface of the roof and surrounding rock, which is usually characterized by the size of the gap or the proportion of the contact area. The time-series data sequence refers to the image data features of targets such as cracks in the roof and surrounding rock strata, anchor bolts and support plates, which are continuously collected and extracted by the roadway roof collapse repair robot at preset time intervals or collection trigger conditions in the underground roadway environment and arranged in chronological order.

[0069] Step S2: Preprocess the collected time-series data sequences, including synchronization calibration, noise reduction, and data scaling normalization, and then divide them into 80% training set and 20% test set.

[0070] Step S3: Construct a temporal information fusion model, which includes a residual spatiotemporal feature extraction module, a surrogate attention mechanism feature focusing module, and a hyperparameter optimization module. First, the hyperparameter optimization module optimizes the hyperparameters of the network model. Then, the residual spatiotemporal feature extraction module extracts features, the surrogate attention mechanism feature focusing module focuses on key features, and finally, the key feature information of the roof fall type and the robot's precise 3D positioning information are output.

[0071] Please refer to Figure 2 The raw image data acquired by the high-resolution explosion-proof camera is input into the residual spatiotemporal feature extraction module to adaptively extract local features. These local features undergo weight redistribution in the proxy attention feature focusing module, outputting enhanced key features of the roof collapse. The output features are then matched and calculated with the robot's own global coordinate system to guide the robot to the target location for repair.

[0072] The residual spatiotemporal feature extraction module consists of a wide-kernel convolutional layer, a residual block, a dilated causal convolutional layer, a pooling layer, a flattening layer, and a fully connected layer connected in series. These components work together to extract the basic features of the landslide image and mine its spatiotemporal correlations. The temporal convolutional network (TCN) is a network structure capable of effectively processing time-series data. By using residual layers to enhance and dilated causal convolutions, TCNs have a better ability to adaptively extract features from the original vibration signals. A TCN mainly includes three modules: dilated causal convolution, a residual module, and a 1D fully-convolutional network.

[0073] Specifically, the residual spatiotemporal feature extraction module works as follows:

[0074] Wide-kernel convolutional layers perform preliminary feature capture on the original roof collapse images acquired by high-resolution explosion-proof cameras: Considering the problem of image grayscale gradient reduction caused by uneven lighting brightness, wide-kernel convolutional layers expand the receptive field of convolution to cover a larger area of ​​the image at once, effectively avoiding local feature breaks caused by dark areas, and initially outlining global features such as the curvature of the roof, the distribution of large-scale rock fissures, and the rough location of the anchor bolt outline, providing a global reference for subsequent detailed feature extraction.

[0075] The residual block receives the feature map output by the wide-kernel convolutional layer and achieves detailed feature preservation through a cross-layer information transfer structure: Considering the problems of local image occlusion and feature shape distortion caused by water mist, resulting in blurred tray edges and loss of fitting details, as well as the reduction of feature signal-to-noise ratio caused by dust, the residual block adopts a small convolutional kernel structure, which directly superimposes the shallow features extracted by the wide-kernel convolution into the deep feature output. This not only preserves shallow detailed features such as anchor outlines and tray edges, but also enhances feature recognition through fine calculation of the small convolutional kernel, avoiding the loss of key details caused by deepening the network.

[0076] The dilated causal convolutional layer performs temporal correlation mining on the feature map output by the residual block: Considering the time-varying nature of the landslide image and the problem of single-frame feature blurring caused by dust fogging, the dilated causal convolutional layer sets the convolution kernel size to 3, the dilation coefficient d∈N* and the default value to 2, which expands the receptive field without increasing the amount of computation and mines feature correlations in the temporal dimension across frames.

[0077] Through formula Complete feature calculation.

[0078] in For dilated convolution operators, For filter size, For the next level Each element is used to complete the blurred feature information such as tiny cracks caused by dust occlusion in a single frame image. At the same time, the temporal causality of causal convolution is used to ensure the rationality of feature association and avoid feature misjudgment caused by temporal disorder.

[0079] Pooling layer, flattening layer and fully connected layer sequentially reduce the dimensionality, flatten and integrate the features output by dilated causal convolutional layer. The final output is a fused feature map that includes local details such as anchor bolt outline, pallet fit and micro cracks, global outlines such as top plate curvature and large-scale collapse area, and time-series correlations such as the dynamic change law of features, providing high-quality data support for the subsequent feature focusing module.

[0080] Specifically, the adaptive feature extraction calculation process of the temporal convolutional network is as follows:

[0081] (1) Use a one-dimensional sequence input x∈Rn and a filter:

[0082] ;

[0083] (2) Apply the following dilation convolution operation to element s:

[0084] ;

[0085] in: It is the dilated convolution operator. It is the expansion factor. It is the size of the filter;

[0086] (3) The input sequence is:

[0087] ;

[0088] (4) The output sequence after three layers of one-dimensional dilated causal convolution with a kernel size of 3 is:

[0089] ;

[0090] In convolution calculations, the dilation coefficient d∈N* is typically taken as 2.

[0091] (5) The receptive field v is related to the size of the convolution kernel, the number of layers in the convolution calculation, and the dilation coefficient. Its calculation formula is:

[0092] ;

[0093] in, The size of the convolution kernel; This represents the number of convolutional layers in the network. The base of the expansion coefficient is usually set to... ;

[0094] (6) In a temporal convolutional network, let there be a one-dimensional input sequence. and convolution kernel The result of the dilated causal full convolution at position s in sequence is:

[0095] ;

[0096] in, The first in the previous layer The elements and other parameters have the same meaning as before.

[0097] A residual block structure is used instead of simple connections between layers to improve the model's generalization ability. Its functional expression is:

[0098] .

[0099] Please refer to Figure 3 and Figure 4 The agent attention feature focusing module employs an agent attention mechanism, an innovative attention mechanism that combines the advantages of both Softmax and linear attention mechanisms. Addressing the core issues of blurred key features and redundant interference in roof collapse images caused by dust atomization, water mist obstruction, and uneven lighting in underground roadways, an agent token is introduced to achieve efficient feature focusing and interference suppression, reducing the original computational complexity from O(n log n) to O(n log n). The Softmax attention mechanism is transformed into a more efficient model that can still capture global information. This solves the pain points of high computational complexity and low efficiency when the traditional attention mechanism is used to process high-resolution images of roof falls, while ensuring the ability to model global features and accurately extract key features of roof falls from image data with multiple disturbances.

[0100] This mechanism achieves a two-step interaction through proxy tokens: First, in the proxy aggregation stage, global information is gathered to the proxy token, effectively integrating scattered key feature clues in the landslide image, such as the crack endpoints across image regions and discretely distributed anchor contours, avoiding feature fragmentation caused by local interference; then, in the proxy broadcast stage, the aggregated information is distributed to each query token, completing the propagation and updating of global features, ensuring the consistency of key features across the entire image domain, while weakening redundant signals caused by dust, water mist, and light interference.

[0101] In practical implementation, the proxy token in the proxy attention mechanism first aggregates global information from key-value pairs through the attention mechanism, focusing on capturing spatiotemporally correlated features such as the crack propagation trajectory and the continuous curvature of the roof bending in the roof fall image. Then, this information is broadcast back to the query token, thereby completing the accurate propagation and optimization of global features. The method is as follows:

[0102] (1) Introducing the proxy token matrix: Assume the input features are , , .in, For query token Key token And numerical tokens The number of features corresponds to the number of feature points in the image of the collapsed roof; The feature dimension encompasses parameters such as rock mass fracture width, anchor bolt length, pallet flatness, and roof bending angle. The proxy attention mechanism introduces a novel proxy token matrix. ,in, For proxy tokens The quantity is usually much smaller By reducing the feature dimensions involved in the calculation, computational redundancy caused by the high resolution of the top image is avoided, while focusing on the core feature regions to avoid interference regions occupying too many computational resources.

[0103] (2) Agent Aggregation: The agent token is first used as a query and key-value pair for attention calculation to aggregate global information. By using the agent token as a query, valuable information related to key features in the collapse image is selectively aggregated. Edge grayscale features of rock fractures are extracted from the blurred background of dust and fog, and structural features of anchor bolt contours are screened from the water mist-covered area. This ensures that information is accurately aggregated from all numerical tokens V to the agent token, reducing the mixing of interfering information. The calculation process is as follows:

[0104] ;

[0105] (3) Agent Broadcast: The agent token acts as the key, interacting with the query token to broadcast aggregated global information back to each query token, achieving global enhancement of key features. For minute cracks difficult to identify in dark areas, the aggregated crack feature template is broadcast to complete the feature details of the cracks in the dark area; for tray edge distortion caused by water mist, the local distorted feature shape is corrected by broadcasting the global tray outline features. This process can be represented as:

[0106] ;

[0107] in, This is a query token that has been broadcast with information and contains updates from global information.

[0108] The computational complexity of Softmax attention is... In comparison, agent attention reduces the complexity to , in For proxy tokens The number is usually much smaller than the query token. The increased number of tokens significantly improves the processing efficiency of high-resolution roof fall images, preventing excessive computation time from affecting the real-time operation response of the tunnel repair robot. Simultaneously, through precise aggregation and broadcasting of proxy tokens, this mechanism maintains strong global semantic modeling capabilities, effectively integrating scattered key features such as cross-regional crack networks and continuous roof bending trajectories in the roof fall images. It also significantly suppresses redundant interference caused by dust atomization, water mist obstruction, and uneven lighting, avoiding the shortcomings of linear attention in feature representation. Thus, it combines the advantages of both Softmax attention and linear attention, achieving the core objective of accurately extracting roof fall type feature information from tunnel roof fall image data.

[0109] (4) Output updated query token: Obtain the query token containing global information updates, complete key feature focusing and redundancy interference suppression, and accurately extract the feature information of the top collapse type.

[0110] The hyperparameter optimization module employs the barrel theory optimization algorithm to optimize the hyperparameters of the temporal information fusion model, thereby improving the model's recognition accuracy and generalization ability. These hyperparameters include the kernel size and dilation coefficient of the dilated causal convolution, the weight allocation parameters of the self-attention mechanism, the learning rate, the regularization coefficient, the number of iterations, and the batch size.

[0111] In specific implementation, the barrel theory optimization algorithm is used to optimize the hyperparameters of the time information fusion model empowered by the proxy attention mechanism, specifically as follows:

[0112] (1) Initialize the time information fusion model empowered by the agent attention mechanism, including setting the kernel size of the wide kernel convolution and dilated causal convolution of the initial time information fusion model empowered by the agent attention mechanism.

[0113] (2) Set the range of weight allocation parameters, learning rate, regularization coefficient, number of iterations and batch size for the self-attention mechanism of the time information fusion model empowered by proxy attention mechanism. Also, set the population size, convergence factor, update probability boundary and maximum number of iterations for the barrel theory optimization algorithm.

[0114] (3) Calculate the fitness value of all individuals in the population. The smaller the fitness value, the better the hyperparameter combination of the individual. The position of the individuals in the population corresponds one-to-one with the model hyperparameters. Sort them according to their fitness value. Select RMSE and MAE as the fitness functions of the barrel theory optimization algorithm. RMSE represents the square root of the ratio of the sum of squares of the errors between the predicted value and the actual value to the duration n. MAE represents the average value of the absolute errors between the predicted value and the actual value. The calculation method is as follows.

[0115] ;

[0116] ;

[0117] in, Let RUL be the error between the actual RUL and the predicted RUL at time t. Duration.

[0118] (4) Determine the update probability of each dimension for each individual in the population: Individuals with lower fitness correspond to the weakest link and are assigned a higher update probability. This is combined with harmonic perturbation to update the hyperparameter dimensions, while a boundary control mechanism pulls the hyperparameters back to the effective range. Conversely, individuals with higher fitness have a lower update frequency to preserve valuable information and maintain population stability. As the optimization process progresses, the upper and lower bounds of the probability gradually decrease, and the update frequency decreases accordingly, thus promoting convergence. The update probability is calculated as follows:

[0119] ;

[0120] in, and These represent the lower and upper bounds of the dynamic update probability range, respectively. This range controls the frequency with which the solution variables are adjusted during the search process. In this formula, Indicates the first The normalized fitness of an individual, with a value range of [0,1], is used to measure the degree of superiority or inferiority of that individual relative to other individuals in the current population.

[0121] The normalization formula is as follows:

[0122] ;

[0123] in Indicates the first The initial fitness value of each individual, and These represent the minimum and maximum fitness values ​​in the current population, respectively.

[0124] Subsequently, probability It will scale within a dynamic probability range, which is determined by two thresholds. and Control. These thresholds are based on the current iteration number. Total number of iterations Problem Dimension and scaling parameters Dynamically changing. Specifically, the minimum and maximum update probabilities are defined as follows:

[0125]

[0126] scaling factor The dimension normalization constant was set to 0.3 according to the experiment. The thresholds are set to 100. As the algorithm runs, both thresholds gradually decrease. As the optimization process progresses, the probability upper and lower limits gradually decrease, and the update frequency decreases accordingly, thereby promoting convergence. The converged data represents the optimal hyperparameter combination for the attention-based temporal information fusion model.

[0127] (5) Determine whether the barrel theory optimization algorithm has finished iterating. When the maximum number of iterations is reached, the algorithm iteration is completed. At this time, the position of the population individual with the best fitness is assigned to the time information fusion model empowered by the agent attention mechanism, that is, the best hyperparameters are assigned to the time information fusion model empowered by the agent attention mechanism. If the barrel theory optimization algorithm iteration has not finished, return to the previous step.

[0128] Step S4: Use the local features and long-term dependent features related to the top of the fall in the training set as the model input and the key features of the top of the fall as the output to train the model.

[0129] In practice, the training set and local features and long-term dependent feature data related to the roof fall are used as input to the time information fusion model empowered by the proxy attention mechanism, and the key features of the roof fall are used as output to complete the extraction and enhancement of key image features of the roof fall in the tunnel. The MAE loss function of Adam optimizer and Softmax classifier is selected. When the set number of iterations is reached, the training stops, the key features of the roof fall are output, and the model training is completed.

[0130] Step S5: Input the test set data into the trained agent attention mechanism-enabled time information fusion model. Through the autonomous learning process of the time information fusion model, the feature information of the roof fall type is accurately extracted from the roof fall image data of the mining roadway, thereby realizing the roadway repair robot's accurate identification of the roof fall type and construction operation requirements.

[0131] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes that can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention are all within the protection scope of the claims of the present invention.

Claims

1. A roof embodied perception method for a tunnel repair robot based on time information fusion, characterized in that, Includes the following steps: Step S1: Use a multi-sensor system to collect images and depth information of the tunnel environment and construct a time-series data sequence containing roof fall characteristics; the multi-sensor system includes a high-resolution explosion-proof camera, a lidar and a displacement sensor. The high-resolution explosion-proof camera is used to collect image data of the roof rock layer and the support structure. The lidar is used to collect three-dimensional contour data of the roof rock layer and the support structure. The displacement sensor is used to collect displacement data of the roof rock layer and the support structure. Step S2: Preprocess the collected time-series data sequence, including synchronization calibration, noise reduction and data scale normalization, and then divide it into 80% training set and 20% test set; Step S3: Construct a time information fusion model, which includes a residual spatiotemporal feature extraction module, a proxy attention mechanism feature focusing module, and a hyperparameter optimization module. First, the hyperparameter optimization module uses the barrel theory optimization algorithm to adaptively optimize the hyperparameters of the network model. Then, the residual spatiotemporal feature extraction module extracts features, the proxy attention mechanism feature focusing module focuses on key features, and finally outputs key feature information of the roof fall type and robot 3D precise positioning information. Step S4: Use the local features and long-term dependent features related to the top of the fall in the training set as input to the network model and the key features of the top of the fall as output to train the model; Step S5: Input the test set data into the trained time information fusion model. Through the autonomous learning process of the time information fusion model, accurately extract the feature information of the roof fall type from the roof fall image data of the mining roadway, so as to realize the roadway repair robot's accurate identification of roof fall type and accurate navigation requirements for repair operation.

2. The roof embodied perception method for a tunnel repair robot based on time information fusion according to claim 1, characterized in that, In step S1, the characteristics of the roof fall include the width and orientation of the cracks in the roof strata, the amount of delamination of the roof, the degree of bending and fracture of the anchor bolts, and the fit between the support plate and the surrounding rock.

3. The roof embodied perception method for a tunnel repair robot based on time information fusion according to claim 1, characterized in that, In step S3, the residual spatiotemporal feature extraction module uses the Temporal Convolutional Network (TCN) as its core technology support, extracts its dilated causal convolution and residual block core components, and integrates them with wide-kernel convolutional layers, pooling layers, flattening layers, and connection layers in a customized manner to construct a spatiotemporal feature mining architecture adapted to the complex downhole environment. Its specific working method is as follows: (1) Wide kernel convolutional layers perform preliminary feature capture on the original roof fall images acquired by high-resolution explosion-proof cameras; (2) The residual block receives the feature map output by the wide kernel convolutional layer and achieves detailed feature preservation through the cross-layer information transmission structure; (3) The dilated causal convolutional layer performs temporal correlation mining on the feature map output by the residual block; (4) The pooling layer, flattening layer and fully connected layer successively reduce the dimension, flatten and integrate the features output by the dilated causal convolutional layer. The final output includes the anchor bolt outline, pallet fit, local details of small cracks, curvature of the top plate, global outline of the large-scale collapse area, and the time-series correlation of the dynamic change law of features, providing high-quality data support for the subsequent proxy attention mechanism feature focusing module.

4. The roof embodied perception method for a tunnel repair robot based on time information fusion according to claim 3, characterized in that, The Temporal Convolutional Network (TCN) achieves accurate capture of spatiotemporal features of the collapse peak through adaptive feature extraction. Its convolution calculation satisfies the causal dilated convolution form and includes residual connections. The specific adaptive calculation process is as follows: (1) Use a one-dimensional sequence input x∈Rn and a filter: ； (2) Apply the following dilation convolution operation to element s: ； in, It is the dilated convolution operator. It is the expansion factor. It is the size of the filter; (3) The input sequence is: ； (4) The output sequence after three layers of one-dimensional dilated causal convolution with a kernel size of 3 is: ； The dilation coefficient d∈N* in convolution calculation; (5) Experience the wild The value is related to the size of the convolution kernel, the number of layers in the convolution calculation, and the dilation coefficient. Its calculation formula is: ； in, The size of the convolution kernel. The number of convolutional layers in the network. As the base of the expansion coefficient, set ; (6) In a temporal convolutional network, let the one-dimensional input sequence be... The convolution kernel is The result of the dilated causal full convolution at position s in sequence is: ； in, The first in the previous layer element.

5. The roof embodied perception method for a tunnel repair robot based on time information fusion according to claim 1, characterized in that... In step S3, the working method of the proxy attention mechanism feature focusing module is as follows: (1) Introducing a proxy token matrix: The number of new proxy token matrices is much smaller than that of query tokens, reducing computational redundancy and focusing on core feature regions; (2) Proxy token aggregation: Proxy tokens are used as query and key-value pairs for attention calculation, which aggregates the value information related to the key features of the global top and reduces interference. (3) Proxy token broadcasting: The proxy token interacts with the query token as a key, broadcasting the aggregated global information back to each query token to complete feature details, correct distorted forms, and strengthen the consistency of key features across the entire domain; (4) Output updated query token: Obtain the query token containing global information updates, complete key feature focusing and redundancy interference suppression, and accurately extract the feature information of the top collapse type.

6. The roof embodied perception method for a tunnel repair robot based on time information fusion according to claim 3, characterized in that, The hyperparameter optimization module works as follows: (1) Initialize the temporal information fusion model: Set the kernel size of the model's wide kernel convolution and dilated causal convolution; (2) Set the range of weight allocation parameters, learning rate, regularization coefficient, number of iterations and batch size; set the population size, convergence factor, update probability boundary and maximum number of iterations for the barrel theory optimization algorithm; (3) Calculate the fitness value of all individuals in the population. The position of each individual in the population corresponds one-to-one with the hyperparameters of the model. Sort them according to their fitness. Select RMSE and MAE as the fitness functions of the barrel theory optimization algorithm. (4) Determine the update probability of each dimension for each individual in the population: individuals with lower fitness correspond to the weakest link and are assigned a higher update probability. The update of hyperparameter dimensions is completed by combining harmonic perturbation and the hyperparameters are pulled back to the effective range through the boundary control mechanism. Individuals with higher fitness have a lower update frequency in order to retain good information and maintain population stability. (5) Determine whether the barrel theory optimization algorithm has finished iterating. When the maximum number of iterations is reached, the algorithm iteration is completed. At this time, the position of the population individual with the best fitness is assigned to the time information fusion model. If the barrel theory optimization algorithm iteration has not finished, return to the previous step.

7. The roof embodied perception method for a tunnel repair robot based on time information fusion according to claim 6, characterized in that, In step (3), the calculation methods for RMSE and MAE are as follows: ； ； in, Let RUL be the error between the actual RUL and the predicted RUL at time t. Duration.

8. The roof embodied perception method for a tunnel repair robot based on time information fusion according to claim 7, characterized in that, In step (4), the update probability is calculated as follows: ； in, and and represent the lower and upper bounds of the dynamic update probability range, respectively. This range controls the frequency with which the solution variables are adjusted during the search process; in this formula, Indicates the first The normalized fitness of an individual, with a value in the range [0,1], is used to measure the individual's superiority or inferiority relative to other individuals in the current population; the normalization formula is as follows: ; in, Indicates the first The initial fitness value of each individual, and These represent the minimum and maximum fitness values ​​in the current population, respectively.

Images