Roof repairing robot roof repairing operation feature body perception and evaluation method and system

Abstract

The application discloses a roof fall repair operation feature body perception and evaluation method and system of a roadway repair robot, and comprises the following steps: step 1, collecting and preprocessing multi-modal data; step 2, labeling a roof fall area of the multi-modal data to obtain roof fall data and construct a training set; step 3, constructing a multi-modal feature fusion model; step 4, optimizing hyperparameters of the multi-modal feature fusion model by using a LEE optimization algorithm; step 5, training the multi-modal feature fusion model by using the training set; and step 6, inputting real-time collected multi-modal data preprocessed in step 1 into the trained multi-modal feature fusion model, and predicting and outputting current roof fall data by using the multi-modal feature fusion model. The application realizes alignment and complementary enhancement of features, and achieves accurate extraction of a roof fall area.

Description

Technical Field

[0001] This invention belongs to the field of mine roadway safety monitoring and intelligent repair technology, and relates to a roadway roof fall autonomous perception, severity classification and quantitative assessment system and method based on multimodal data fusion and deep learning model. Background Technology

[0002] Coal is a crucial cornerstone of energy security, and underground coal mine roadways are the "lifeline" of coal mining, their stability directly impacting mine production safety and efficiency. Roof collapse accidents in deep, narrow roadways are a major safety hazard in mining and underground engineering construction. Roof collapses not only seriously threaten the lives of underground workers but also easily cause equipment damage and even trigger secondary disasters. Traditional roof collapse detection relies mainly on manual inspections and experience-based judgment, which suffers from low efficiency, high subjectivity, high risk, and difficulty in quantification. With the advancement of intelligent mine construction, using robots to replace manual labor in dangerous or demanding positions has become a clear development direction. However, current roadway repair robots generally lack autonomous environmental awareness and decision-making capabilities, cannot automatically identify roof collapse risks in deep, narrow roadways, and struggle to assess their severity, still requiring manual pre-survey and remote control. This severely restricts their level of intelligence and operational efficiency. Therefore, using tunnel repair robots to repair deformations such as roof falls, settlements, and collapses in tunnel roofs is a key link in preventing roof fall accidents, ensuring personnel safety, and ensuring the stable operation of the project. It has significant practical value and real-world significance.

[0003] In the process of roof deformation repair in deep, narrow tunnels, it is necessary to conduct real-time assessment and feedback on the repair progress based on key indicators such as roof rock mass cracks, displacement, and the condition of the support structure to ensure operational safety and repair accuracy. To this end, robots need to be equipped with multiple types of sensors for multimodal data acquisition, including: using depth cameras to acquire visible light and infrared images to obtain texture information such as roof surface cracks, water seepage, and spalling; using multi-line lidar to acquire three-dimensional point clouds to characterize the roof geometry and spatial deformation; using acoustic emission sensors to acquire elastic wave signals generated by rock mass fracturing; and using vibration sensors to acquire structural vibration responses to monitor the internal rock mass condition. However, in deep, narrow tunnel operations, dust generated by rock mass fracturing and water mist from dust suppression sprays can severely interfere with data acquisition, a problem particularly prominent in repair operations. Specifically, dust and water mist significantly reduce image quality, leading to decreased image contrast, blurred textures, and affecting the identification of surface features such as cracks. Point cloud data is prone to noise points and data gaps in dusty environments, affecting the integrity and accuracy of geometric morphology. Vibration and acoustic emission signals are interfered with by mechanical noise and dust particle collisions, reducing the signal-to-noise ratio and submerging effective event components. Existing methods mostly rely on single datasets, making it difficult to accurately extract key features during roof collapse repair, thus hindering real-time assessment and feedback of the extent of roadway roof repair operations. Furthermore, since the assessment needs to be completed in real-time on the robot, the model must be embedded in the robot's local edge computing unit. However, edge devices are limited by computing power and storage resources, requiring a streamlined model structure and a small parameter size, necessitating lightweight design to meet millisecond-level local inference requirements. In addition, the complex selection of hyperparameters in the model directly affects recognition and assessment performance, requiring efficient optimization methods (such as the LEE optimization algorithm) for parameter tuning to improve the model's adaptability and stability in complex roadway environments.

[0004] The roof condition of deep, narrow tunnels is complex and variable, influenced by various factors such as geological conditions, support methods, and mining stress. In tunnel roof safety monitoring, data generated by multiple sensors exhibit significant differences in signal morphology, mathematical structure, and noise characteristics. Currently, tunnel repair robots can comprehensively assess roof stability and the risk of roof collapse by collecting multimodal data, including 3D point clouds, visible and infrared images, and acoustic and vibration signals. However, the robot's use of deep learning methods to process this multimodal data for real-time feedback and evaluation still faces the following limitations:

[0005] (1) When the tunnel repair robot operates in narrow tunnels in deep underground mines, it is necessary to collect image data in real time to assess the degree of repair. The image signal collected by the vision sensor is essentially a two-dimensional discrete matrix. Under the interference of dust and water mist, the image quality is severely degraded, large areas of low contrast appear in the matrix, the spatial correlation between pixels is destroyed, and the edge and texture features are significantly weakened.

[0006] (2) When the tunnel repair robot operates in narrow tunnels in deep underground mines, it needs to rely on point cloud data to perceive changes in the roof morphology in real time. The point cloud signal collected by multi-line lidar is a sparse set of points in three-dimensional space, which can provide high-precision geometric information and has a strong ability to characterize morphological features such as roof deformation and collapse volume. However, in a dusty environment, point cloud data is easily interfered with, and its ability to completely characterize morphological features and the accuracy of geometric information will decrease, thus limiting its reliability and real-time performance in repair assessment.

[0007] (3) When tunnel repair robots operate in narrow tunnels in deep underground mines, they need to monitor the internal state of the rock mass in real time using acoustic emission and vibration data. As a time series, this type of signal has typical non-stationary and nonlinear characteristics, which can reflect the rock mass fracture and stress release process and has the potential for disaster early warning. The signal contains transient impact components and steady-state background noise, which are strongly coupled in the time and frequency domains. The noise spectrum of underground mechanical operations is wide and the energy is high, which can easily drown out the effective signal, making it difficult to achieve signal-to-noise separation by directly performing Fourier transform or wavelet analysis. This renders existing feature extraction methods ineffective, thereby affecting the real-time judgment of the repair level and safety assessment. Summary of the Invention

[0008] The technical problem this invention aims to solve is to overcome the shortcomings of existing technologies and provide a method and system for embodied perception and evaluation of roof fall repair operation characteristics of tunnel repair robots. This research addresses the aforementioned characteristics of multi-source signals in the complex environment of deep, narrow tunnels, aiming to construct a neural network model that can effectively integrate the advantages of heterogeneous signals. This model does not simply superimpose multimodal inputs, but rather analyzes the differences in physical properties, mathematical structures, and noise patterns of different signals to achieve feature alignment and complementary enhancement. This results in a unified representation of the tunnel roof geometry, surface texture, and internal activity information, ultimately achieving accurate extraction, multi-dimensional feature analysis, and intelligent classification of the roof fall area.

[0009] Based on the above analysis, this invention proposes a system and method to address the problems of inaccurate autonomous sensing and difficulties in grading and quantifying roof fall detection in roadways. This system can automatically identify roof fall areas and accurately grade and quantify their severity, thus providing a reliable basis for repair decisions. The method proposed in this invention is of great significance to intelligent mining equipment, represented by roadway repair robots, as it helps to achieve early detection and accurate assessment of roof fall hazards, improve the scientific nature and safety of repair operations, reduce human intervention and related risks, optimize repair resource scheduling, and support applications such as intelligent maintenance decision-making and predictive maintenance.

[0010] Prior to this invention, the present invention provides a method for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot, comprising:

[0011] Step 1: Collect multimodal data for roof fall detection. The multimodal data includes image signals, point cloud signals, acoustic emission signals, and vibration signals of the roof fall.

[0012] The multimodal data is denoised, spatiotemporally registered, and aligned to a coordinate system. The image signal is enhanced and distortion corrected. The point cloud signal and the image signal are preprocessed, including spatiotemporal alignment, normalization, and feature-level fusion, to obtain the preprocessed multimodal data.

[0013] Step 2: Label the roof fall region on the preprocessed multimodal data to obtain roof fall data including the three-dimensional coordinate information of the roof fall region mask, the spatial location of the roof fall region mask, and the geometric range of the roof fall region mask;

[0014] A training set is constructed based on the preprocessed multimodal data and the data showing the collapse of the roof.

[0015] Step 3: Construct a multimodal feature fusion model for roof fall detection;

[0016] Step 4: Optimize the hyperparameters of the multimodal feature fusion model using the LEE optimization algorithm;

[0017] Step 5: Train the constructed multimodal feature fusion model using the training set. Use image signals, point cloud signals, acoustic emission signals, and vibration signals as inputs to the multimodal feature fusion model, and use the roof fall data as the output of the multimodal feature fusion model. Construct the mapping relationship between the input and output of the multimodal feature fusion model, and simultaneously optimize the model parameters of the multimodal feature fusion model.

[0018] Step 6: Input the real-time acquired multimodal data preprocessed in Step 1 into the trained multimodal feature fusion model, and use the multimodal feature fusion model to predict and output the current top-fall data.

[0019] Prior to step 3, a multimodal feature fusion model for roof fall detection is constructed based on the UniConv-Net architecture.

[0020] Firstly, the GELU activation function in the UniConv-Net architecture is replaced with the FReLU activation function.

[0021] Prior to step 5, geometric and appearance features are extracted from the fused image signal, point cloud signal, acoustic emission signal, and vibration signal to obtain multi-dimensional features.

[0022] Prior to step 7, based on the multi-dimensional features obtained in step 5 and the current top-fall data output in step 6, multi-dimensional features of the top-fall region mask are extracted. The multi-dimensional features of the top-fall region mask include geometric features, appearance features, and temporal features. The multi-dimensional features of the top-fall region mask are input into the lightweight fusion hierarchical decision model, and the lightweight fusion hierarchical decision model is used to output classification results including mild, moderate, and severe, and corresponding confidence scores.

[0023] Prior to this, a system for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot includes:

[0024] The data acquisition and preprocessing module is used to acquire multimodal data for roof fall detection, including image signals, point cloud signals, acoustic emission signals, and vibration signals of the roof fall; it performs denoising, spatiotemporal registration, and coordinate system unification on the multimodal data, and enhances and corrects distortion of the image signals; it performs preprocessing on the point cloud signals and image signals, including spatiotemporal alignment, normalization, and feature-level fusion, to obtain preprocessed multimodal data.

[0025] The training set construction module is used to annotate the roof fall region in the preprocessed multimodal data to obtain roof fall data including the three-dimensional coordinate information of the roof fall region mask, the spatial location of the roof fall region mask, and the geometric range of the roof fall region mask; and to construct a training set based on the preprocessed multimodal data and roof fall data.

[0026] A multimodal feature fusion model training module is used to construct a multimodal feature fusion model for roof fall detection; the hyperparameters of the multimodal feature fusion model are optimized using the LEE optimization algorithm; the constructed multimodal feature fusion model is trained using a training set, with image signals, point cloud signals, acoustic emission signals, and vibration signals as inputs to the multimodal feature fusion model, and roof fall data as outputs; a mapping relationship between the inputs and outputs of the multimodal feature fusion model is constructed, and the model parameters of the multimodal feature fusion model are optimized simultaneously.

[0027] The prediction output module is used to input the preprocessed real-time acquired multimodal data into the trained multimodal feature fusion model, and use the multimodal feature fusion model to predict and output the current top-fall data.

[0028] Preferably, the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described herein.

[0029] Preferably, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described herein.

[0030] The beneficial effects achieved by this invention are as follows:

[0031] 1. Addressing the highly heterogeneous nature of multi-source signals in downhole environments, this invention innovatively designs a residual-enabled multimodal feature fusion model. This model achieves efficient fusion and enhanced representation of point cloud geometric features and image texture features through lightweight spatial and temporal information extraction. While ensuring the diversity of feature receptive fields, it significantly improves the ability to identify multi-level and multi-scale features in roof degradation morphology, effectively overcoming the limitations and unstable performance of traditional single sensors in low-light and high-noise environments.

[0032] 2. To address the non-stationary and high-noise characteristics of multimodal data, including time-series signals such as acoustics and vibrations, the proposed model employs a simplified gating structure for its time information extraction module, employing a bidirectional improved cyclic unit. This design maintains robust time-series modeling capabilities while reducing parameter complexity, effectively extracting key time-series dependent features of fracture events within the rock mass, enhancing the model's interpretability, and providing a highly discriminative time-series representation for early warning of roof falls.

[0033] 3. Regarding model optimization, the LangEvin equation evolution algorithm is introduced to adaptively tune the hyperparameters of the multimodal feature fusion model. This algorithm combines the guiding nature of stochastic gradient descent with the stochastic perturbation mechanism of the LangEvin equation, dynamically balancing local exploitation and global exploration during training. This significantly improves the convergence speed and recognition accuracy of the multimodal feature fusion model, enabling it to better adapt to the complex, high-dimensional, non-convex optimization environment of underground mining.

[0034] 4. In the stage of assessing the severity of roof fall, this invention leverages the fast convergence speed and strong optimization capability of the LEE optimization algorithm to efficiently optimize the model parameters (such as learning rate and regularization coefficient) of the lightweight fusion hierarchical decision-making model used for hierarchical decision-making. This enables the lightweight fusion hierarchical decision-making model to achieve objective and precise classification of the severity of roof fall in dealing with time-series classification problems, providing an intuitive and reliable quantitative basis for the formulation of robot autonomous repair strategies and resource optimization scheduling. Attached Figure Description

[0035] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 These are flowcharts of some embodiments of this application; Detailed Implementation

[0037] See Figure 1 This application discloses a method and system for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot, including:

[0038] Step 1: Multimodal data synchronous acquisition and preprocessing. Control the tunnel repair robot to move slowly along the tunnel and synchronously acquire the following signals and perform preprocessing: (1) Image signal: Visible light and infrared images are acquired by the depth camera on the robot to capture texture information such as cracks, seepage, and detachment on the roof surface. The image signal is preprocessed with histogram equalization, adaptive denoising and illumination correction to enhance texture features; (2) Point cloud signal: Three-dimensional spatial point cloud is acquired by multi-line lidar to characterize the geometric shape and spatial deformation of the roof. The point cloud signal is preprocessed with outlier removal, downsampling and coordinate system to the local coordinate system of the tunnel; (3) Acoustic emission signal and vibration signal: Acoustic emission signal is obtained by the transient elastic wave generated by rock fracture by the acoustic emission sensor, and vibration signal is obtained by the vibration sensor of the roof structure vibration response. The acoustic emission signal and vibration signal are preprocessed with bandpass filtering, detrending and segmented normalization to highlight the effective event components.

[0039] Step 2: Construction of the roof fall dataset. A roof fall dataset containing different degrees of severity (mild, moderate, severe) is constructed and divided into training, validation, and test sets in a 7:2:1 ratio. Spatiotemporal alignment and feature-level fusion are performed on the point cloud and image signals: The point cloud signals are converted into voxel grids and projected onto the image pixels; local geometric features and corresponding image patch texture features are extracted from each voxel grid to form a multimodal fusion feature map, which serves as input to the subsequent model. Local geometric features include curvature and normal vector changes, while texture features include gray-level co-occurrence matrix and edge density.

[0040] Step 3: Construct a residual-enabled multimodal feature fusion model to achieve efficient fusion and feature extraction of multimodal data.

[0041] 3.1 Lightweight Spatial Information Extraction. Convolutional neural networks extract local features by sliding convolutional kernels across image signals and expand the receptive field through stacked layers. However, when using extremely large convolutional kernels to capture long-range dependencies, the effective receptive field distribution is disrupted. This phenomenon causes the influence distribution of pixels in the network to deviate from the ideal asymptotic Gaussian distribution, making it difficult for the model to focus on key regions and introducing a large amount of redundant computation, thereby reducing the model's efficiency and expressive power.

[0042] UniConv-Net is a novel general-purpose lightweight convolutional neural network architecture. Through the design of receptive field aggregators and layer operators, its core idea is to use appropriate combinations of smaller convolutional kernels to simulate the effect of large kernels. That is, by stacking the corresponding modules, the effective receptive field (EFR) is expanded to a level comparable to that of existing large kernel convolutional neural networks (CNNs), while maintaining its asymptotic Gaussian distribution characteristics.

[0043] Therefore, in order to fully explore the complex spatial patterns contained in multimodal data, this invention designs a novel multimodal feature fusion model based on the UniConv-Net architecture, replacing the GELU activation function in the UniConv-Net architecture with the FReLU activation function. The multimodal feature fusion model aims to efficiently extract and fuse multi-scale contextual information and local detail features in multimodal data, specifically including the following steps: (1) The input feature map of the multimodal data is segmented into multiple heads for parallel and recursive processing: (1), where, It is the input feature map. It is the main source of recursive processing. , , These are the first auxiliary head, the second auxiliary head, and the Nth auxiliary head. It refers to the number of layers in the receptive field aggregator. This is a channel-sharing operation.

[0044] (2) arrive Projection is performed using a 1x1 convolution to enhance feature diversity: (2), where, The first branch feature tensor after projection. The second branch feature tensor after projection. For 1×1 convolution operations, The input feature map for the nth layer, This is another input feature map for the nth layer. It is by Derived feature representation.

[0045] (3) The projected features are fed into the manipulator: , (3), where the operator includes an amplifier and a discriminator. The result of the amplifier's operation. This represents the result of the discriminator's calculations. The amplifier expands the receptive field and amplifies the influence of key pixels through deep convolution with a large kernel and a gating mechanism. It is the kernel size that increases with the number of layers. This represents element-wise multiplication. The chosen activation function is GELU, which enhances the model's feature discrimination ability in low-contrast, blurred images. This is a depthwise convolution operation. The discriminator incorporates local detail information through parallel convolutional kernels of varying sizes. It is a fixed small convolution kernel size. For depthwise convolution operations, for Standard convolution operations of size, where Let be the side length of the convolution kernel, and As the number of network layers increases. (4) The outputs of the amplifier and discriminator are spliced ​​together in the channel dimension to form an output head with a dual-layer discriminative receptive field. : (4), where, For the concatenation operation, the two feature tensors are connected along the channel dimension, and Dis is the result of the discriminator's operation.

[0046] (5) By recursively calling the operator, the receptive field is gradually aggregated and expanded, eventually forming an effective receptive field A2 with a large scale and maintaining an asymptotic Gaussian distribution: (5), (6), (7), where, It is the principal head projection feature tensor of the nth layer, where n is a positive integer. It is the feature tensor of the nth layer auxiliary head projection. It is the Nth layer auxiliary head projection feature tensor. It is the operator at the nth level. It is the fault characteristic of the final output.

[0047] 3.2 Lightweight Time Information Extraction. Due to the limitations of LSTM and GRU in practical engineering applications, their complex gating mechanisms lead to high computational complexity.

[0048] Therefore, in order to efficiently extract the temporal features of multimodal data, the multimodal feature fusion model in this invention adopts a bidirectional improved recurrent unit module as the temporal information extraction module. The core calculation process of this module is as follows: (1) Multimodal data is propagated forward and backward to capture bidirectional temporal dependencies. This step involves calculating the new state of the input gate, which determines the information allowed to pass through the gate. The forward and backward hidden states are fused through a weight matrix to generate the final hidden state at the current time. The update formula of the bidirectional improved recurrent unit module is as follows: (8), of which This represents the weight matrix during forward transmission. This represents the weight matrix in the backhaul. and These are the hidden states for forward and backward propagation, respectively. Let 't' be the fusion hidden state. This is a bias term used to adjust the output.

[0049] (2) The input signal and the hidden state at the previous time step are linearly transformed to generate a weighted sum, which is used for gating calculation: , (9). The net input at time t, Hidden state The corresponding weights Let be the hidden state at time t−1. For input The corresponding weights Let be the input features at time t. This is a bias term.

[0050] (3) The weighted sum is used to generate the input gate state via the Sigmoid activation function, controlling the retention and forgetting of information: ,(10). The activation output at time t, This is the activation function.

[0051] (4) The candidate hidden state is determined by the current input and the historical state after gating, and is used to update the current state: , (11). Let be the candidate hidden state at time t. The hyperbolic tangent activation function is used. For input The weight matrix The hidden state weight matrix, The bias term for the candidate hidden state.

[0052] (5) The final hidden state is obtained by weighted fusion of the candidate state and the historical state: ,(12).

[0053] (6) The output layer performs a linear transformation and activation on the hidden state to generate the predicted output for the current time step. , (13). The output value at time t. The linear output before activation. To output the weight matrix, This is the output bias term.

[0054] (7) The gradients of the output layer, hidden layer, and each parameter are calculated sequentially using the backpropagation algorithm and used for weight updates in the multimodal feature fusion model: (14), For the loss function on the output gradient, The loss function E at time t t Output y of the model t The partial derivatives, The true label value at time t. (15), The linear output before activation. (16) (17) (18), (19) (20), ,(twenty one), ,(twenty two), ,(twenty three), ,(twenty four), (25), (26), ,(27).

[0055] Through the above steps, the feature processing unit module, while maintaining the ability to model time series, achieves efficient feature extraction from multimodal data with fewer parameters and a simpler structure, providing a highly discriminative time series representation for the autonomous identification and hierarchical quantitative assessment of roof fall disasters in deep, narrow tunnels.

[0056] Step 4: Optimize the hyperparameters of the multimodal feature fusion model using the LEE optimization algorithm. To improve the convergence speed and recognition accuracy of the multimodal feature fusion model, the LangEvin Equation Evolution (LEE) algorithm is used to adaptively optimize the model parameters of the multimodal feature fusion model. The LEE optimization algorithm combines the guidance of stochastic gradient descent with the stochastic perturbation mechanism of the LangEvin equation, and is suitable for high-dimensional non-convex optimization problems. Specifically, it includes: (1) Generating an initial population. This step aims to create an initial population that evolves within a specified iteration range. For a size of The LEE optimization algorithm will randomly generate the population. Each individual in the population is denoted as a position. represents the solution to an optimization problem in D-dimensional space. Initial position. Typically, random generation is based on the following principles: ,in and They represent the first question, second, and third questions, respectively. The lower and upper limits of dimensions, Represents a random number within the interval [0,1]. (2) Transformation of Langevin equation into update rule. Transforming Langevin equation into update rule requires three steps, described in detail below: Step 1, mapping of terms in Langevin equation. As mentioned above, Langevin equation describes the motion of a particle under the simultaneous action of three forces: inertia (acceleration), friction, and random fluctuations caused by molecular collisions, as shown in the following equation: Where m is the particle mass. It is the coefficient of friction. It is thermodynamic temperature. Here, x is the Boltzmann constant, x is the particle position, and t is time. This is a Gaussian white noise term.

[0057] This expression clearly shows that the trajectory of a particle is the result of the interaction between particle acceleration, friction, and random driving force. To encapsulate this physical principle into an optimizable search mechanism, the formula needs to be decomposed into its constituent terms and restated in algorithmic terminology as follows: The inertia term corresponds to the particle's acceleration. In the LEE optimization algorithm, inertia controls candidate solutions and the motion response based on historical velocity and current force. It should be noted that since mass does not affect the optimization process, this is assumed here... ,get .in, This represents acceleration. The friction (damping) term describes the frictional force acting on the particle, and its magnitude is proportional to the particle's velocity. In an optimization context, it acts as a damping effect, preventing the candidate solution set from becoming excessively dispersed and balancing its motion. The mathematical expression of this term is as follows: ,in Represents particle velocity. Let be the friction coefficient. The random force term reflects the unpredictable collisions with surrounding particles. In the LEE optimization algorithm, this force introduces randomness into the update process, enabling candidate solutions to escape local minima and explore the search space more effectively. The random force is defined by the following formula: , As a stochastic force, and combining the three contributions mentioned above, the Langevin equation can be simplified using the following equation. This new mathematical formulation will accelerate Damping terms With random disturbances It is directly related to the process of moving candidate solutions in the LEE optimization algorithm.

[0058] Step 2: Define the equivalent form of the algorithm, including associating each element in the equation with the movement of candidate solutions during the optimization process. Specifically, the... candidate solutions In iteration ( The acceleration at time ) is: ,in This indicates the number of iterations. The formula maintains physical similarity by transforming the three forces in the Langevin equation into optimization dynamics. Damping term ( For stable motion, inertial terms preserve momentum, while random forces introduce diversity. These terms collectively balance exploration and development in the algorithm proposed in this invention. In effect, they enable the algorithm to simulate the physical behavior of particles as they traverse the space of possible solutions.

[0059] The stochastic force represents the random perturbation acting on each candidate solution. This is analogous to the random collisions experienced by a particle in a fluid medium. Within the optimization framework, this force is modeled as: ,in and Let these represent the current solution and temperature, respectively. This represents the average of the two random solutions in the current iteration. Scaling factor. Used to control the intensity of disturbances. Meanwhile, the item... This measures the deviation of the current candidate solution from the average position of two random solutions. This design ensures that the random force is not arbitrary noise, but rather adapts to the population state: a larger deviation from the mean generates stronger random perturbations, promoting thorough exploration; a smaller deviation reduces randomness, accelerating convergence. Therefore, the random force effectively maintains population diversity and helps the algorithm escape early local optima. In the formula, The calculation method is as follows: , Let this be the value of the first variable in the it-th iteration. For the value of the second (fixed or non-iterative) variable, the coefficient and The dynamic changes with the number of iterations. Based on the annealing and damping processes in physics, both of which decay exponentially with time, the model is as follows: , ,in Indicates the maximum number of iterations. Let be the initial temperature, and e be the natural constant. This asymptotic decay mechanism is crucial for ensuring the algorithm focuses on exploration in early iterations, exhibiting strong randomness and weak damping, while shifting towards development and convergence in later iterations (with weak randomness and strong damping). After updating the rate, the new position of each solution... : This update directly connects the future position of the solution to its current path. From a physical perspective, this reflects the motion of a particle under the combined influence of velocity and acceleration; in an optimization sense, it means that potential solutions can continuously escape suboptimal regions and approach better regions.

[0060] Step 5: Training of the multimodal feature fusion model. The multimodal feature fusion model is trained end-to-end using the training set, test set, and validation set. The optimizer uses Adam combined with the LEE optimization algorithm for hyperparameter tuning. During the training process, the following multi-dimensional feature vectors are extracted from the fused features: (1) Geometric features: roof volume, average depth, curvature distribution, and normal vector dispersion; (2) Appearance features: crack density, texture complexity, and grayscale distribution entropy; (3) Temporal features: event frequency, energy release rate, and signal attenuation coefficient.

[0061] Step 6: Identification and segmentation of the collapsing region. Input the test set or real-time acquired multimodal data into the trained multimodal feature fusion model, and output the binary mask of the collapsing region and its three-dimensional spatial coordinates. Further refine the segmentation results through post-processing (such as connected component analysis and morphological operations).

[0062] Step 7: Severity Quantitative Assessment. Based on the extracted multi-dimensional features, a lightweight hierarchical decision sub-network is constructed, outputting three-level classification results (mild, moderate, severe) and confidence scores. Final comprehensive score. for: The weight Determined by adjusting the validation set.

[0063] In this embodiment of the application, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods described above.

[0064] In this application embodiment, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described above.

[0065] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0066] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention described herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not invented herein. The specification and embodiments are to be considered exemplary only.

[0067] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. A method for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot, characterized in that, include: Step 1: Collect multimodal data for roof fall detection. The multimodal data includes image signals, point cloud signals, acoustic emission signals, and vibration signals of the roof fall. The multimodal data is denoised, spatiotemporally registered, and aligned to a coordinate system. The image signal is enhanced and distortion corrected. The point cloud signal and the image signal are preprocessed, including spatiotemporal alignment, normalization, and feature-level fusion, to obtain the preprocessed multimodal data. Step 2: Label the roof fall region on the preprocessed multimodal data to obtain roof fall data including the three-dimensional coordinate information of the roof fall region mask, the spatial location of the roof fall region mask, and the geometric range of the roof fall region mask; A training set is constructed based on the preprocessed multimodal data and the data showing the collapse of the roof. Step 3: Construct a multimodal feature fusion model for roof fall detection; Step 4: Optimize the hyperparameters of the multimodal feature fusion model using the LEE optimization algorithm; Step 5: Train the constructed multimodal feature fusion model using the training set. Use image signals, point cloud signals, acoustic emission signals, and vibration signals as inputs to the multimodal feature fusion model, and use the roof fall data as the output of the multimodal feature fusion model. Construct the mapping relationship between the input and output of the multimodal feature fusion model, and simultaneously optimize the model parameters of the multimodal feature fusion model. Step 6: Input the real-time acquired multimodal data preprocessed in Step 1 into the trained multimodal feature fusion model, and use the multimodal feature fusion model to predict and output the current top-fall data.

2. The method for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot according to claim 1, characterized in that, Step 3: Based on the UniConv-Net architecture, construct a multimodal feature fusion model for roof fall detection.

3. The method for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot according to claim 1, characterized in that, Replace the GELU activation function in the UniConv-Net architecture with the FReLU activation function.

4. The method for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot according to claim 1, characterized in that, In step 5, geometric and appearance features are extracted from the fused image signal, point cloud signal, acoustic emission signal, and vibration signal to obtain multi-dimensional features.

5. The method for embodied perception and evaluation of the characteristics of roof collapse repair operations using a tunnel repair robot according to claim 4, characterized in that, Step 7: Based on the multi-dimensional features obtained in Step 5 and the current roof fall data output in Step 6, extract the multi-dimensional features of the roof fall region mask. The multi-dimensional features of the roof fall region mask include geometric features, appearance features, and temporal features. The multi-dimensional features of the top-fall region mask are input into the lightweight fusion hierarchical decision model, and the lightweight fusion hierarchical decision model outputs classification results including mild, moderate and severe, and corresponding confidence scores.

6. A system for embodied perception and evaluation of roof fall repair operation characteristics of a tunnel repair robot, applied to the method for embodied perception and evaluation of roof fall repair operation characteristics of a tunnel repair robot as described in any one of claims 1-5, characterized in that, include: The data acquisition and preprocessing module is used to acquire multimodal data for roof fall detection, including image signals, point cloud signals, acoustic emission signals, and vibration signals of the roof fall; it performs denoising, spatiotemporal registration, and coordinate system unification on the multimodal data, and enhances and corrects distortion of the image signals; it performs preprocessing on the point cloud signals and image signals, including spatiotemporal alignment, normalization, and feature-level fusion, to obtain preprocessed multimodal data. The training set construction module is used to annotate the roof fall region on the preprocessed multimodal data to obtain roof fall data including the three-dimensional coordinate information of the roof fall region mask, the spatial location of the roof fall region mask, and the geometric range of the roof fall region mask; A training set is constructed based on the preprocessed multimodal data and the data showing the collapse of the roof. A multimodal feature fusion model training module is used to construct a multimodal feature fusion model for roof fall detection; the hyperparameters of the multimodal feature fusion model are optimized using the LEE optimization algorithm; the constructed multimodal feature fusion model is trained using a training set, with image signals, point cloud signals, acoustic emission signals, and vibration signals as inputs to the multimodal feature fusion model, and roof fall data as outputs; a mapping relationship between the inputs and outputs of the multimodal feature fusion model is constructed, and the model parameters of the multimodal feature fusion model are optimized simultaneously. The prediction output module is used to input the preprocessed real-time acquired multimodal data into the trained multimodal feature fusion model, and use the multimodal feature fusion model to predict and output the current top-fall data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 6.

Images