Intelligent detection method for risk of coal mine support member based on multi-modal inversion reasoning

Abstract

The invention provides a coal mine support member risk intelligent detection method based on multi-mode inversion reasoning. Comprising the following steps of state data collection and data set construction of a multi-modal supporting component, defect detection and deformation recognition model design of the supporting component, failure path chain construction based on model inversion and risk assessment and early warning output based on failure path chain driving. According to the method, surface defects, three-dimensional deformation and attitude information can be synchronously obtained and fused, the real state of the component can still be stably recognized in severe environments such as low illumination and dust, and the missing detection and misjudgment rate is remarkably reduced; a failure path chain can be generated based on causal inference, the propagation and influence sequence between defects is disclosed, and a direct basis is provided for fault root positioning and preventive maintenance; and in combination with the defect category, the deformation quantity and the link risk aggregation result, a risk level judgment close to an actual operation safety condition is given, and operation and maintenance personnel are guided to formulate targeted intervention measures.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of coal mine support member defect detection, and particularly relates to a coal mine support member risk intelligent detection method based on multi-modal inversion reasoning. BACKGROUND

[0002] Coal mine underground roadway support members are important infrastructure for ensuring the stability of the roadway and the safety of the operating personnel. However, under the long-term effects of surrounding rock pressure, humidity changes, and mining disturbances, the support members are prone to various defects such as cracks, peeling, corrosion, and the like. These defects not only weaken the load-bearing capacity of the members, but also can cause local failure to spread globally, leading to major safety accidents such as roadway collapse.

[0003] To this end, current engineering practices have adopted single-modal image detection, three-dimensional laser or point cloud measurement, and multi-modal fusion-based structure monitoring techniques to achieve coal mine support member defect detection. Specifically:

[0004] 1) Single-modal image detection technology uses an industrial camera to capture member surface images, and performs defect recognition through traditional image processing or convolutional neural networks. This method has good results under conditions of sufficient light and fully exposed targets, but the detection accuracy significantly decreases in low-illumination, dust-shielded, and texture-inconspicuous underground environments, and cannot obtain member overall deformation information.

[0005] 2) Three-dimensional laser or point cloud measurement technology uses laser scanning or structured light measurement to obtain spatial geometric information of the member, and calculates deformation through point cloud registration and difference. This method is independent of member surface texture and has accurate deformation measurement, but lacks the ability to perceive small surface defects, making it difficult to achieve simultaneous recognition of defects and deformation.

[0006] 3) Multi-modal fusion-based structure monitoring technology attempts to combine image and point cloud data to improve detection accuracy and robustness. However, existing multi-modal methods mostly stop at simple data or feature layer fusion, lack feature enhancement and constraint mechanisms for low-illumination and attitude changes in underground environments, and the cross-modal complementary nature has not been fully explored.

[0007] The inventors of the present application have found that, due to insufficient light, high dust concentration, and various component forms in the underground coal mine environment, significant challenges are brought to defect detection and state evaluation, and single sensor or single modal information is difficult to comprehensively depict the surface features and overall deformation state of the component. On the other hand, these methods generally only aim at defect identification at a single moment, and fail to deeply analyze the causal relationship between defects and their spatial transmission path. Since the defects of the supporting component are not isolated, there may be an evolutionary relationship between different defects, for example, the crack development of component 1 may cause the deformation or peeling of subsequent component 2. This transmission and accumulation effect between defects is an important risk factor that traditional detection methods are difficult to quantify. SUMMARY

[0008] In view of the technical problems that the existing coal mine supporting component defect detection technology still has significant deficiencies in multi-modal information enhancement fusion and defect evolution reasoning in the complex underground environment, the present application provides a coal mine supporting component risk intelligent detection method based on multi-modal inversion reasoning, which has multi-modal perception, causal reasoning and risk warning capabilities.

[0009] To solve the above technical problems, the present application adopts the following technical solutions:

[0010] The coal mine supporting component risk intelligent detection method based on multi-modal inversion reasoning comprises the following steps:

[0011] S1, multi-modal supporting component state data acquisition and data set construction: a low-illumination industrial camera, a three-dimensional point cloud acquisition device and an inertial measurement unit are arranged in the roadway where the supporting component is located, and the component surface image, three-dimensional geometric shape and attitude information are synchronously acquired respectively; the defect type, defect position and deformation amount index are obtained through artificial labeling, the multi-modal supporting component state acquisition data and the labeling results are paired, and a multi-modal supporting component state-defect deformation data set is constructed;

[0012] S2, supporting component defect detection and deformation identification model design: a supporting component defect detection and deformation identification model comprising a weak light texture enhancement, three-dimensional deformation coding, pose feature embedding and cross-modal fusion module is constructed, the model takes the multi-modal supporting component state data acquired in step S1 as input, and simultaneously outputs the supporting component defect category, position and deformation amount identification results; the model is trained based on the multi-modal supporting component state-defect deformation data set constructed in step S1, and the performance is optimized through single-modal pre-training and full-model joint training, so as to realize synchronous identification of defects and deformation in complex environments;

[0013] S3, failure path chain construction based on model inversion: model inversion attribution analysis is performed on each defect identified in step S2, key areas or feature components are extracted, counterfactual intervention is performed and intervention benefit score is calculated; the causal relationship is determined by combining the spatial distance of the defect and the benefit score, and the failure path chain set is generated in the order from upstream to downstream;

[0014] S4, risk assessment and early warning output based on failure path chain driving: combining the identification results of the model in step S2 and the failure path chain set generated in step S3, the risk weight of the defect is assigned and the deformation variable is normalized, the single defect risk value and the link aggregation risk value are calculated; compared with the preset risk level threshold, the risk detection level of the component is determined, and the early warning information containing the main risk link is output.

[0015] Further, the step S1 multi-modal support component state data acquisition and data set construction specifically includes the following steps:

[0016] S11, multi-modal sensor arrangement: the operating personnel carry portable low-illumination industrial cameras and portable three-dimensional point cloud acquisition devices into the roadway where the support component is located, and hold the devices to scan the component in all directions, while using a portable inertial measurement unit to obtain the attitude information of the support component;

[0017] S12, multi-modal support component state data acquisition: based on the sensors selected in step S11, the complete image data Sc Ig of the support component surface is obtained by the industrial camera Sg , the corresponding spatial geometric information Sc Cg is obtained by the three-dimensional point cloud acquisition device, and the component attitude data Sc Ig measured by the inertial measurement unit is recorded synchronously, thereby forming the multi-modal support component state data MSc=[Sc Sg ,Sc Cg ];

[0018] S13, defect deformation label calibration: the collected multi-modal support component state data MSc is manually analyzed and labeled, and the labeling information includes defect type Dt, defect location Dl, and deformation variable Di, thereby forming complete defect deformation label data Del=[Dt,Dl,Di];

[0019] S14, multi-modal support component state-defect deformation data set making: taking the multi-modal support component state data MSc as input and the defect deformation label data Del as output, a set of support component state-defect deformation data set is formed, and based on this method, a large amount of data acquisition and labeling is performed, and finally a complete multi-modal support component state-defect deformation data set is obtained.

[0020] Further, the support member defect detection and deformation recognition model designed in the step S2 includes a weak light texture enhancement and defect perception module, a three-dimensional geometric deformation coding module, a pose constraint feature embedding module, and a cross-modal defect detection and deformation recognition module, and is specifically designed as follows:

[0021] S21, weak light texture enhancement and defect perception module design: this module aims to restore the surface texture details of the member and enhance the distinguishability of the defect features, taking the image data Sc Ig collected by the two-dimensional industrial camera as input. First, the basic texture features under different receptive fields are extracted through a multi-scale convolution layer to obtain multi-scale primary texture features F mst ; Then, it enters the adaptive light estimation layer, which uses a MobileNet lightweight convolution subnet to predict the supplementary full-image light distribution, and adaptively adjusts the brightness and contrast in local areas to suppress shadows and restore dark area details, to obtain light-enhanced texture features F let ;

[0022] After that, the light-enhanced texture features F let are input into seven depth-shareable convolution layers in series, and then the residual connection is used to retain details while reducing computational complexity, to generate defect response enhanced features F dre ; Then, through five dilated convolution layers with increasing dilation rates, the receptive field is expanded to capture the cross-regional crack extension and boundary relationship, and the ReLU activation function is used to suppress negative value noise, and finally the weak light texture enhanced features F Llt are output.

[0023] S22, three-dimensional geometric deformation coding module design: this module is used to extract the geometric contour and deformation variable features of the support member from the three-dimensional point cloud, taking the three-dimensional point cloud data Sc Sg collected by the three-dimensional point cloud acquisition device as input. First, the redundant points are removed and the key geometric structure is retained through voxel downsampling and max-pooling layers to obtain sparse geometric skeleton features F sgk ;

[0024] After that, eight Point Transformer layers are input to encode the sparse geometric skeleton features F sgk layer by layer to obtain global deformation coding features F gdc ; On this basis, a deformation variable display calculation layer is introduced, which specifically calculates the displacement vectors of key cross sections and nodes by registering the scanned point cloud with the pose reference frame, and encodes these physical quantities into additional feature channels, and fuses them with the global deformation coding features F gdc through the ReLU activation function to output three-dimensional deformation variable enhanced features F Tde .

[0025] S23, pose constraint feature embedding module design: this module uses the component attitude data Sc collected by the inertial measurement unit Cg to perform consistency constraints to avoid feature deviation caused by changes in acquisition angle and component attitude, and to obtain the component attitude data Sc as input Cg First, the spatial position coding layer is used to embed the attitude and time information into a learnable vector space to obtain the pose coding feature F pef ;

[0026] Then, the five KAN feature mapping layers in series are input, and the learnable univariate B-spline mapping and linear combination are used to generate the attitude-deformation correlation feature F adc ; Then, the two fully connected layers are sequentially mapped to the same channel dimension as the two-dimensional and three-dimensional features, and the SiLU activation function is used to retain the gradient smoothness to obtain the pose constraint embedding feature F Pce ;

[0027] S24, cross-modal defect detection and deformation identification module design: this module is used to fuse the output features F Llt , F Tde and F Pce of the above three modules to realize joint prediction of defects and deformation variables. First, the weak texture enhancement feature F Llt , the three-dimensional deformation variable enhancement feature F Tde and the pose constraint embedding feature F Pce are respectively compressed in spatial dimension by the global average pooling layer to obtain compact global semantic vectors; then, through the cross-modal gated fusion mechanism, the weights are dynamically allocated according to the complementarity and consistency of each modal feature, and the cross-modal fusion feature vector F mcm is output. This feature vector is sequentially passed through two fully connected layers to generate the prediction feature vector F prf , which is then divided into two branches: one branch outputs the defect type recognition result Dt pre through the Softmax activation function, and the other branch outputs the defect position recognition result Dl pre and the deformation variable index recognition result Di pre ;

[0028] In the inference stage, the confidence Cl pre of the overall recognition process is obtained through the defect classification probability, the defect recognition position, and the deformation variable regression uncertainty, and Dt pre , Dl pre , Di pre and Cl pre are output as the final results of defect and deformation identification;

[0029] S25, model training and optimization: based on the multi-modal support component state-defect deformation data set constructed in step S1, the support component defect detection and deformation recognition model is trained. In the training process, single-channel pre-training is first performed to enhance the respective robustness, and then full-model joint training is performed; when the preset maximum iteration number is reached, the training is stopped, and the final trained support component defect detection and deformation recognition model is obtained.

[0030] Further, the step S3 of constructing the failure path chain based on model inversion specifically comprises the following steps:

[0031] S31, defect level model inversion attribution analysis: for each defect of the defect detection result, the support component defect detection and deformation recognition model trained in step S2 is called to perform model inversion attribution analysis. For the i-th defect in the total number of defects N, i∈[1,N], specifically:

[0032] For the two-dimensional image channel, the Grad-CAM++ attribution method is used to obtain the generated attribution heat map corresponding to the i-th defect, and the key pixel region Mia pix with the largest contribution to the i-th defect is determined by the heat value size.

[0033] For the three-dimensional point cloud channel, the point-level attribution is used to obtain the influence weight of each point on the prediction of the i-th defect, and the key point set Mia int with the highest weight is obtained according to the influence weight.

[0034] For the posture feature, the model feature weight is used to obtain the component with the largest contribution to the detection of the i-th defect, and the component is taken as the key posture component Mia pos .

[0035] S32, counterfactual intervention simulation and benefit evaluation: for the key pixel region Mia pix , pixel filling operation is used to repair and replace the pixels in the region to obtain the virtual image sample Vir pix ; similarly, for the key point set Mia int , geometric smoothing operation is used to repair and replace the point cloud in the region to construct a virtual point cloud sample Vir int ; for the key posture component Mia pos , the normalized alignment method is used to replace the key component to construct a virtual posture sample Vir pos .

[0036] At this point, for the i-th defect instance, a complete set of virtual sample inputs Vir i =[Vir pix ,Vir int ,Vir pos ] is formed, and Viri Input the trained support component defect detection and deformation recognition model to obtain the confidence level Cl under the virtual sample. Vir And calculate its confidence level Cl compared with the original sample. pre The change in the amount of intervention yields the intervention benefit score Scr i =[Scr i 1 ,Scr i 2 ,...,Scr i j ,...,Scr i N ]; among them, Scr i j This represents the intervention benefit score for the j-th defect in the virtual sample with the i-th defect;

[0037] Similarly, counterfactual intervention simulations are performed for each defect to obtain a complete intervention benefit table Scr1, Scr2, ..., Scr N ;

[0038] S33. Failure Path Chain Construction: First, based on the spatial location Dl of each defect identified in step S2... pre To obtain the pairwise distances between defects This represents the distance between the i-th defect and the j-th defect;

[0039] Afterwards, if defects Less than the preset distance discrimination threshold Dl thr And the intervention benefit score Scr i j Scr greater than the preset intervention threshold thr If , it means that the i-th defect is the upstream defect of the j-th defect, that is, in the process of defect evolution, the i-th defect is likely to lead to the generation of the j-th defect;

[0040] Based on this principle, the upstream defects of all defects are identified, and the defects are connected sequentially from upstream to downstream to generate a failure path chain. This process is repeated until all defects are assigned, resulting in the failure path chain set SPC of the component.

[0041] Furthermore, step S4, the risk assessment and early warning output based on the failure path chain, specifically includes the following steps:

[0042] S41. Defect Risk Quantification: First, preset the risk weights w1, w2, ..., w7 for each type of defect, and normalize the deformation to Di according to the physical safety limit. norm ∈[0,1];

[0043] Subsequently, based on the defect location identification results Dl pre Differentiate between different defects and calculate the risk quantification value for each defect, specifically R. i =w i Di i , where R i w represents the risk quantification value of the i-th defect. i Di is the risk weight corresponding to the i-th defect. i Let R1 be the normalized value of the deformation corresponding to the i-th defect; then, based on this method, obtain the risk quantification values ​​R1, R2, ..., R for each defect. N N is the total number of defects;

[0044] S42. Link Risk Aggregation: For the set of failed path chains (SPC), calculate the cumulative node risk value from upstream to downstream for each failed path chain, and divide it by the number of defects to obtain the risk score R for each link. chain The sum of the risk scores of all links is used as the comprehensive risk score R of the current component. total Meanwhile, link C with the highest risk score is retained. max The main source of risk for current components;

[0045] S43. Component Comprehensive Risk Assessment and Source Tracing Output: Preset multi-stage risk level thresholds Risk1, Risk2, Risk3, and Risk4, corresponding to component safety, concern, warning, or severe status, respectively. Based on the comprehensive risk score R... total The risk detection level (State) of the component is ultimately determined by comparing it with multi-stage risk level thresholds.

[0046] Subsequently, based on the risk detection level (State) and the link C with the highest risk score, max In addition, the failure path chain set SPC generates a risk tracing report, providing a direct reference for downhole maintenance and intervention decisions.

[0047] Compared with existing technologies, the intelligent risk detection method for coal mine support components based on multimodal inversion reasoning provided by this invention has the following beneficial effects:

[0048] 1. Achieve comprehensive detection of the condition of downhole support components: Simultaneously acquire and integrate surface defects, three-dimensional deformation and attitude information, and can stably identify the true condition of components even in harsh environments such as low light and dust, significantly reducing the rate of missed detection and false judgment.

[0049] 2. Supports traceable analysis of defect evolution relationships: Based on causal inference, it generates failure path chains, reveals the propagation and impact sequence between defects, and provides direct basis for fault root cause location and preventive maintenance.

[0050] 3. Improve the accuracy and practicality of risk assessment: combine defect categories, deformation variables and link risk aggregation results to give risk level judgments close to actual operation safety conditions, guide operation and maintenance personnel to develop targeted intervention measures. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is the overall flowchart of the coal mine supporting member risk intelligent detection method based on multi-modal inversion reasoning provided by the application.

[0052] Figure 2 is the defect detection and deformation recognition model structure diagram of the supporting member provided by the application.

[0053] Figure 3 is the failure path chain construction flowchart based on model inversion provided by the application.

[0054] Figure 4 is the risk assessment and early warning output flowchart based on the failure path chain driving provided by the application. DETAILED DESCRIPTION

[0055] In order to make the technical means, creative features, purposes and effects realized by the application easy to understand, the application will be further described below in combination with specific drawings.

[0056] Please refer to Figure 1 The application provides a coal mine supporting member risk intelligent detection method based on multi-modal inversion reasoning, which comprises the following steps:

[0057] S1, multi-modal supporting member state data acquisition and data set construction: low-illumination industrial cameras, three-dimensional point cloud acquisition devices and inertial measurement units are arranged in the roadway where the supporting member is located, and the surface image, three-dimensional geometric shape and attitude information of the member are synchronously acquired respectively; the defect type, defect position and deformation variable index are obtained through artificial labeling, the multi-modal supporting member state acquisition data and the labeling results are paired, and a multi-modal supporting member state-defect deformation data set is constructed;

[0058] S2, supporting member defect detection and deformation recognition model design: a supporting member defect detection and deformation recognition model containing weak light texture enhancement, three-dimensional deformation coding, pose feature embedding and cross-modal fusion module is constructed, the model takes the multi-modal supporting member state data acquired in step S1 as input, and simultaneously outputs the supporting member defect category, position and deformation variable recognition result; during model training, the multi-modal supporting member state-defect deformation data set constructed in step S1 is used for training, the performance is optimized through single-modal pre-training and full-model joint training, and synchronous recognition of defects and deformation in complex environments is realized;

[0059] S3, failure path chain construction based on model inversion: model inversion attribution analysis is performed on each defect identified in step S2, key areas or feature components are extracted, counterfactual intervention is performed and intervention benefit score is calculated; the causal relationship is determined by combining the spatial distance of the defect and the benefit score, and the failure path chain set is generated in the order from upstream to downstream;

[0060] S4, risk assessment and early warning output based on failure path chain driving: combining the identification results of the model in step S2 and the failure path chain set generated in step S3, the risk weight of the defect is assigned and the deformation variable is normalized, the single defect risk value and the link aggregation risk value are calculated; compared with the preset risk level threshold, the risk detection level of the component is determined, and the early warning information containing the main risk link is output.

[0061] As a specific embodiment, in order to realize the detection of defects and deformation variables of coal mine underground supporting components in low illumination, dust and other complex environments, the present application first designs a multi-modal sensor acquisition method, and constructs a supporting component state-defect deformation data set based on the acquisition results. Accordingly, the step S1 multi-modal supporting component state data acquisition and data set construction specifically includes the following steps:

[0062] S11, multi-modal sensor arrangement: the operator carries a portable low-illumination industrial camera and a portable three-dimensional point cloud acquisition device into the roadway where the supporting component is located, and holds the device to scan the component in all directions, while using a portable inertial measurement unit (IMU) to obtain the attitude information of the supporting component;

[0063] S12, multi-modal supporting component state data acquisition: based on the sensors selected in step S11, the complete image data Sc Ig of the surface of the supporting component is obtained by the industrial camera Sg ; the corresponding spatial geometric information Sc Cg is obtained by the three-dimensional point cloud acquisition device, and the component attitude data Sc Ig (measured by the inertial measurement unit) is recorded synchronously (including pitch angle, yaw angle and roll angle), to constitute the multi-modal supporting component state data MSc=[Sc Sg ,Sc Cg ];

[0064] S13, defect deformation label calibration: the collected multi-modal supporting component state data MSc is manually analyzed and labeled, the labeling information includes defect type Dt, defect location Dl, and deformation variable Di, to constitute complete defect deformation label data Del=[Dt,Dl,Di];

[0065] Specifically, the defect type Dt includes seven types of cracks, peeling, corrosion, rust, deformation, holes, and surface wear; the defect position Dl is a three-dimensional coordinate point, and the coordinate system is the same as the three-dimensional rectangular coordinate system of the collected point cloud, the coordinate origin is located at the center of the bottom of the support member, the X axis is along the length direction of the roadway, the Y axis is along the width direction of the roadway, and the Z axis is vertically upward; the deformation index Di is obtained by registration calculation of the point cloud and the attitude information, including three parameters of deflection, relative displacement and out-of-plane bulging;

[0066] S14, multi-modal support member state-defect deformation dataset making: taking the multi-modal support member state data MSc as input and the defect deformation label data Del as output, a set of support member state-defect deformation dataset is constituted, a large amount of data is collected and labeled based on the method, and finally a complete multi-modal support member state-defect deformation dataset is obtained.

[0067] As a specific embodiment, in order to realize the synchronous identification of the surface defects and the overall deformation of the support member in the complex coal mine environment, the present application designs a support member defect detection and deformation identification model, which includes a weak light texture enhancement and defect perception module, a three-dimensional geometric deformation encoding module, a pose constraint feature embedding module and a cross-modal defect detection and deformation identification module, and the model structure is as shown in Figure 2 The specific design is as follows:

[0068] S21, weak light texture enhancement and defect perception module design: aiming at the problem of image visibility decline caused by low illumination, uneven illumination and dust shielding in the underground, the module aims to restore the surface texture details of the component and enhance the distinguishability of the defect features, and the specific two-dimensional industrial camera collected image data Sc Ig is input, first, the multi-scale convolution layer (3x3, 5x5, 7x7 convolution in parallel) is used to extract the basic texture features under different receptive fields, and the multi-scale primary texture features F mst are obtained; then, the adaptive illumination estimation layer is entered, the MobileNet light weight convolution subnet is used to predict the supplementary full image illumination distribution, and the brightness and contrast are adaptively adjusted in the local area to suppress the shadow and restore the dark area details, and the illumination enhanced texture features F let are obtained; then, the illumination enhanced texture features F let are input into the seven depth shareable convolution layers (including: DepthwiseConvolution extracts the spatial mode of each channel, Pointwise Convolution fuses channel information) in series, and then the residual connection is used to retain the details while reducing the calculation amount, and the defect response enhanced features F dre; Then, the receptive field is expanded by five dilated convolution layers with increasing dilation rates to capture the crack extension and boundary relationship across regions, and the negative noise is suppressed by the ReLU activation function, and finally the weak light texture enhancement feature F is output Llt ;

[0069] S22, three-dimensional geometric deformation encoding module design: this module is used to extract the geometric profile and deformation variable features of the support member from the three-dimensional point cloud, and the three-dimensional point cloud data Sc Sg is input, first through voxel downsampling and max-pooling layer to remove redundant points and retain key geometric structure, get sparse geometric skeleton feature F sgk ;

[0070] Then, in order to model the long-range dependence and spatial correlation between points, eight Point Transformer layers are input to the sparse geometric skeleton feature F sgk Layer-by-layer encoding is performed to obtain the global deformation encoding feature F gdc ; On this basis, a deformation variable display calculation layer is introduced, which specifically calculates the displacement vector (X, Y, Z direction) of the key section and node by registering the scanned point cloud with the attitude reference frame, and encodes these physical quantities into additional feature channels, and fuses with the global deformation encoding feature F gdc After ReLU activation function, the three-dimensional deformation variable enhancement feature F is output Tde ;

[0071] S23, pose constraint feature embedding module design: this module uses the member attitude data Sc Cg for consistency constraint to avoid feature deviation caused by acquisition angle and member attitude change, and uses the member attitude data Sc Cg as input, first through the spatial position coding layer (Sinusoidal Positional Encoding) to embed the attitude and time information into the learnable vector space to obtain the pose encoding feature F pef ;

[0072] Then input five KAN (Kolmogorov-Arnold Network) feature mapping layers in series, use learnable univariate B-spline mapping and linear combination to generate attitude-deformation correlation feature F adc ; Then, two fully connected layers are sequentially mapped to the same channel dimension as two-dimensional and three-dimensional features, and SiLU activation function is used to retain gradient smoothness, and the pose constraint embedding feature F is obtained Pce ;

[0073] S24, cross-modal defect detection and deformation identification module design: this module is used to fuse the output features FLlt , F Tde and F Pce , the joint prediction of defects and deformation is realized. First, the weak light texture enhancement feature F Llt , the three-dimensional deformation enhancement feature F Tde and the pose constraint embedding feature F Pce are compressed in spatial dimension by global average pooling layer respectively, and compact global semantic vectors are obtained; then through the cross-modal gated fusion mechanism, the weights are dynamically allocated according to the complementarity and consistency of each modal feature, and the cross-modal fusion feature vector F mcm is output. The feature vector sequentially passes through two fully connected layers to generate the predicted feature vector F prf , which is then divided into two branches: one branch outputs the defect type recognition result Dt pre through the Softmax activation function, and the other branch outputs the defect position recognition result Dl pre and the deformation variable index recognition result Di pre through the ReLU activation function.

[0074] In the inference stage, the confidence Cl pre of the overall recognition process is obtained through the defect classification probability, the defect recognition position, and the deformation variable regression uncertainty, and Dt pre , Dl pre , Di pre and Cl pre are output as the final results of defect and deformation recognition.

[0075] S25, model training and optimization: based on the multi-modal support component state-defect deformation data set constructed in step S1, the support component defect detection and deformation recognition model is trained. In the training process, single channel pre-training is first performed to enhance the robustness of each, and then full model joint training is performed. When the preset maximum iteration number is reached, the training is stopped, and the final trained support component defect detection and deformation recognition model is obtained.

[0076] As a specific embodiment, the multi-modal defect detection and deformation recognition results Dt pre , Dl pre , Di pre and Cl preAs different defects may be correlated with each other, the present application proposes a failure path chain construction module based on model inversion, which first performs model inversion attribution analysis on each defect (a total of N) detected in step S2, extracts the key position that has the greatest impact on the judgment; then calculates the influence of the position change on the detection results of all defects through counterfactual intervention simulation, forms an intervention benefit table; finally, the causal chain link between defects is inferred by combining the relationship between defect space position and intervention benefit, and a failure path chain set is generated for risk tracing and intervention decision-making. Accordingly Figure 3 As shown in the figure, the step S3 of constructing the failure path chain based on model inversion specifically includes the following steps:

[0077] S31, defect level model inversion attribution analysis: for each defect of the defect detection result, the supporting member defect detection and deformation recognition model trained in step S2 is called to perform model inversion attribution analysis. For the i-th defect in the total N defects, i∈[1,N], specifically:

[0078] For a two-dimensional image channel, the Grad-CAM++ attribution method is used to obtain the generated attribution heat map corresponding to the i-th defect, and the key pixel region Mia pix with the largest contribution to the i-th defect is determined by the heat value size.

[0079] For a three-dimensional point cloud channel, the point-level attribution (Point Attribution) is used to obtain the influence weight of each point on the prediction of the i-th defect, and the key point set Mia int with the highest weight is obtained according to the influence weight.

[0080] For the posture feature, the model feature weight is used to obtain the component with the largest contribution to the detection of the i-th defect, and the component is taken as the key posture component Mia pos .

[0081] S32, counterfactual intervention simulation and benefit evaluation: for the key pixel region Mia pix , pixel filling operation is used to repair and replace the pixels in the region to obtain a virtual image sample Vir pix ; similarly, for the key point set Mia int , geometric smoothing operation is used to repair and replace the point cloud in the region to construct a virtual point cloud sample Vir int ; for the key posture component Mia pos , the normalized alignment method is used to replace the key component to construct a virtual posture sample Vir pos .

[0082] At this point, for the i-th defect instance, a complete set of virtual sample inputs Vir i =[Virpix Vir int Vir pos ], and Vir i Input the trained support component defect detection and deformation recognition model to obtain the confidence level Cl under the virtual sample. Vir (Including the confidence levels of N defects), and calculate its confidence level Cl compared to the original sample. pre The change in (including the confidence levels of N defects) yields the intervention benefit score Scr. i =[Scr i 1 ,Scr i 2 ,...,Scr i j ,...,Scr i N ]; among them, Scr i j This represents the intervention benefit score for the j-th defect in a virtual sample with the i-th defect. This value quantifies the influence of the i-th defect on the identification of the j-th defect. i j The larger the absolute value, the greater the impact of the i-th defect on the detection of the j-th defect;

[0083] Similarly, counterfactual intervention simulations are performed for each defect to obtain a complete intervention benefit table Scr1, Scr2, ..., Scr N ;

[0084] S33. Failure Path Chain Construction: First, based on the spatial location Dl of each defect identified in step S2... pre To obtain the pairwise distances between defects This represents the distance between the i-th defect and the j-th defect;

[0085] Afterwards, if defects Less than the preset distance discrimination threshold Dl thr And the intervention benefit score Scr i j Scr greater than the preset intervention threshold thr If , it means that the i-th defect is the upstream defect of the j-th defect, that is, in the process of defect evolution, the i-th defect is likely to lead to the generation of the j-th defect;

[0086] Based on this principle, the upstream defects of all defects are identified, and the defects are connected sequentially from upstream to downstream to generate a failure path chain. This process is repeated until all defects are assigned, resulting in the failure path chain set SPC of the component.

[0087] As a specific embodiment, after the failure path chain construction in step S3 is completed, the application further designs a risk assessment and early warning output module driven by the failure path chain. The module takes the identification results of step S2 (including the defect category identification result Dt pre , the defect location identification result Dl pre , and the deformation variable index identification result Di pre ) and the failure path chain set SPC generated in step S3 as inputs, quantifies the overall risk level of the component, and generates early warning information and traceability basis. The overall steps are shown in Figure 4 . The risk assessment and early warning output based on the failure path chain in step S4 specifically includes the following steps:

[0088] S41, defect risk quantification: first, preset the risk weight w1, w2, …, w7 corresponding to each defect, and normalize the deformation variable according to the physical safety limit value as Di norm ∈ [0, 1] (including N deformation variables);

[0089] Then, by the defect location identification result Dl pre , different defects are distinguished, and the risk quantification value of each defect is calculated, specifically R i = w i · Di i , where R i represents the risk quantification value of the i-th defect, w i is the risk weight corresponding to the i-th defect, and Di i is the normalized value of the deformation variable corresponding to the i-th defect; then, based on this method, the risk quantification value R1, R2, …, R N of each defect is obtained, and N is the total number of defects;

[0090] S42, link risk aggregation: for the failure path chain set SPC, the cumulative node risk value of each failure path chain from upstream to downstream is calculated, and divided by the number of defects to obtain the risk score R chain of each link, and the sum of the risk scores of all links is taken as the comprehensive risk score R total of the current component; at the same time, the link C max with the largest risk score is retained as the main risk source link of the current component;

[0091] S43, component comprehensive risk determination and traceability output: preset multi-stage risk level thresholds Risk1, Risk2, Risk3, Risk4, respectively corresponding to component safety, attention, warning or serious state, based on the comprehensive risk score R total and the multi-stage risk level threshold, the risk detection level State of the component is finally determined by comparison;

[0092] Then, based on the risk detection level State, the link C with the maximum risk score max And the failure path chain set SPC generates a risk traceability report, providing direct reference for downhole maintenance and intervention decision-making.

[0093] Compared with the prior art, the coal mine support member risk intelligent detection method based on multi-modal inversion reasoning provided by the present application has the following beneficial effects:

[0094] 1. Realize comprehensive detection of downhole support member state: synchronously acquire and fuse surface defects, three-dimensional deformation and attitude information, and still can stably identify the real state of the member in harsh environments such as low illumination and dust, thereby significantly reducing the missed detection and misjudgment rate.

[0095] 2. Support traceable analysis of defect evolution relationship: generate a failure path chain based on causal inference, reveal the propagation and influence order among defects, and provide direct basis for fault root location and preventive maintenance.

[0096] 3. Improve the accuracy and practicality of risk assessment: combine defect categories, deformation amounts and link risk aggregation results to give risk level judgment close to the actual operation safety condition, and guide operation and maintenance personnel to develop targeted intervention measures.

[0097] Finally, it should be explained that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. A coal mine supporting component risk intelligent detection method based on multi-modal inversion reasoning, characterized in that, Comprise the following steps: S1, multi-modal support component state data acquisition and dataset construction: low-illumination industrial camera, three-dimensional point cloud acquisition device and inertial measurement unit are arranged in the roadway where the support component is located, and the component surface image, three-dimensional geometric shape and attitude information are synchronously collected; the defect type, defect position and deformation amount index are obtained by manual labeling, the multi-modal support component state acquisition data and the labeling result are paired, and a multi-modal support component state-defect deformation dataset is constructed; S2, support component defect detection and deformation recognition model design: a support component defect detection and deformation recognition model containing weak light texture enhancement, three-dimensional deformation coding, pose feature embedding and cross-modal fusion module is constructed, the model takes the multi-modal support component state data collected in step S1 as input, and outputs the support component defect category, position and deformation amount recognition result; the model is trained based on the multi-modal support component state-defect deformation dataset constructed in step S1, the performance is optimized through single-modal pre-training and full-model joint training, and the synchronous recognition of defects and deformations in complex environments is realized; S3, failure path chain construction based on model inversion: model inversion and cause analysis are performed on each defect identified in step S2, key areas or feature components are extracted, counterfactual intervention is performed, and intervention benefit scores are calculated; The causal relationship is determined by combining the spatial distance of the defect and the benefit score, and the failure path chain set is generated in the order from upstream to downstream; S4, risk assessment and early warning output based on failure path chain driving: combining the recognition result of the model in step S2 and the failure path chain set generated in step S3, the risk weight of the defect is allocated and the deformation amount is normalized, the single-defect risk value and the link aggregation risk value are calculated; Compared with the preset risk level threshold, the risk detection level of the component is determined, and the early warning information containing the main risk link is output.

2. The coal mine supporting member risk intelligent detection method based on multi-modal inversion reasoning according to claim 1, characterized in that, The step S1 multi-modal support component state data acquisition and dataset construction specifically comprises the following steps: S11, multi-modal sensor arrangement: the operator carries a portable low-illumination industrial camera and a portable three-dimensional point cloud acquisition device into the roadway where the support component is located, holds the device to scan the component in all directions, and uses a portable inertial measurement unit to obtain the attitude information of the support component; S12, multi-modal support component state data acquisition: based on the selected sensors in step S11, complete image data Sc of the surface of the support component is acquired by an industrial camera Ig ; corresponding spatial geometric information Sc is acquired in combination with a three-dimensional point cloud acquisition device Sg , and component attitude data Sc measured by an inertial measurement unit is recorded synchronously Cg , thereby forming multi-modal support component state data MSc = [Sc Ig , Sc Sg , Sc Cg ] S13, defect deformation label calibration: the collected multi-modal support component state data MSc is manually analyzed and labeled, the labeling information includes defect type Dt, defect position Dl, and deformation amount index Di, which constitute complete defect deformation label data Del=[Dt, Dl, Di]; S14, multi-modal support component state-defect deformation dataset making: taking the multi-modal support component state data MSc as input and the defect deformation label data Del as output, a set of support component state-defect deformation dataset is constructed, a large amount of data is collected and labeled based on this method, and finally a complete multi-modal support component state-defect deformation dataset is obtained.

3. The coal mine supporting member risk intelligent detection method based on multi-modal inversion reasoning according to claim 1, characterized in that, The support member defect detection and deformation identification model designed in the step S2 comprises a weak light texture enhancement and defect perception module, a three-dimensional geometric deformation coding module, a pose constraint feature embedding module, and a cross-modal defect detection and deformation identification module, and is specifically designed as follows: S21, weak light texture enhancement and defect perception module design: this module aims to restore the surface texture details of the component and enhance the distinguishability of the defect features, based on the image data Sc collected by the two-dimensional industrial camera Ig As input, first extract the basic texture features under different receptive fields through multi-scale convolutional layers to obtain multi-scale primary texture features F mst ; then enter the adaptive illumination estimation layer, which uses a MobileNet lightweight convolutional subnet to predict the supplementary full-image illumination distribution, and adaptively adjusts the brightness and contrast in local regions to suppress shadows and restore dark area details, to obtain illumination-enhanced texture features F let ; After that, the light enhances the texture feature F let After inputting seven depth-shareable convolution layers in series, the residual connection is used to reserve details and reduce the amount of calculation, and the defect response enhanced feature F dre is generated. Then, five dilated convolution layers with increasing dilation rates are used to expand the receptive field, capture the crack extension and boundary relationship across the region, and suppress negative noise through the ReLU activation function, and finally output the weak light texture enhanced feature F Llt . S22, three-dimensional geometry deformation coding module design: the module is used for extracting the geometric profile and deformation variable characteristics of the support member from the three-dimensional point cloud, and the three-dimensional point cloud data Sc obtained by the three-dimensional point cloud acquisition device Sg For input, first remove redundant points and retain key geometric structures by voxel downsampling and max-pooling layer to obtain sparse geometric skeleton features F sgk ; After that, eight Point Transformer layers are input to the sparse geometry skeleton feature F sgk Layer-by-layer encoding is performed to obtain the global deformation encoding feature F gdc On this basis, a deformation variable display calculation layer is introduced. Specifically, the scanning point cloud is registered with the pose reference frame, the displacement vectors of the key sections and nodes are calculated, and these physical quantities are encoded into additional feature channels, and are fused with the global deformation encoding feature F gdc The three-dimensional deformation variable enhanced feature F Tde is output after ReLU activation function. S23, pose constraint feature embedding module design: this module uses the component attitude data Sc collected by the inertial measurement unit Cg Conduct consistency constraints to avoid feature deviation caused by changes in acquisition angle and component attitude, using component attitude data Sc Cg as input, first through the spatial position coding layer to embed the attitude and time information into the learnable vector space to obtain the pose coding feature F pef ; Then five KAN feature mapping layers in series are input, and pose-deformation associated features F are generated by using learnable univariate B-spline mapping and linear combination adc ; then two fully connected layers are sequentially passed to the same channel dimension as the two-dimensional and three-dimensional features, and the SiLU activation function is used to retain the gradient smoothness, obtaining the pose constraint embedding features F Pce ; S24, cross-modal defect detection and deformation recognition module design: the module is used for fusing the output features F of the above three module channels Llt 、 Tde and F Pce , realizing joint prediction of defects and deformation variables, first compressing the spatial dimensions of weak light texture enhancement features F Llt , three-dimensional deformation variable enhancement features F Tde and pose constraint embedding features F Pce respectively through global average pooling layers to obtain compact global semantic vectors; then through a cross-modal gating fusion mechanism, dynamically allocating weights according to the complementarity and consistency of the modal features, outputting a cross-modal fusion feature vector F mcm , which is sequentially subjected to two fully connected layers to generate a predicted feature vector F prf , which is then divided into two branches: one branch outputs a defect type recognition result Dt pre through a Softmax activation function, and the other branch outputs a defect position recognition result Dl pre and a deformation variable index recognition result Di pre ; In the inference stage, the confidence Cl of the whole recognition process is obtained by defect classification probability, defect recognition position, and deformation amount regression uncertainty pre , and Dt pre , Dl pre , Di pre , and Cl pre are output as the final results of defect and deformation recognition. S25, model training and optimization: based on the multi-modal support member state-defect deformation data set constructed in step S1, the support member defect detection and deformation identification model is trained. In the training process, single-channel pre-training is first performed to enhance the robustness of each channel, and then full-model joint training is performed; When the preset maximum number of iterations is reached, the training is stopped, and the final trained support member defect detection and deformation identification model is obtained.

4. The coal mine supporting member risk intelligent detection method based on multi-modal inversion reasoning according to claim 1, characterized in that, The step S3 based on model inversion failure path chain construction specifically comprises the following steps: S31, defect level model inversion attribution analysis: for each defect of the defect detection result, the support member defect detection and deformation identification model trained in step S2 is called to perform model inversion attribution analysis. For the i th defect in the total number of defects N, i∈[1, N], specifically: For the two-dimensional image channel, the Grad-CAM++ attribution method is used to obtain the generated attribution heat map corresponding to the ith defect, and the key pixel region Mia that contributes most to the ith defect is determined through the heat value size pix ; For the three-dimensional point cloud channel, the point-level attribution is used to obtain the influence weight of each point on predicting the i-th defect, and the key point set Mia with the highest influence weight is obtained according to the influence weight int ; For the posture feature, the component with the largest contribution to the detection of the ith defect is obtained through the model feature weight, and is taken as the key posture component Mia pos ; S32, counterfactual intervention simulation and benefit evaluation: for the key pixel area Mia pix , the virtual image sample Vir after intervention is obtained by using pixel filling operation to repair and replace the pixels in this area pix ; Similarly, for the key point set Mia int , the virtual point cloud sample Vir is constructed by using geometric smoothing operation to repair and replace the point cloud in this area int ; for the key attitude component Mia pos , the virtual attitude sample Vir is constructed by using the normalized alignment method to replace the key component pos ; So far, for the i-th defect instance, a complete set of virtual sample inputs Vir i = [Vir pix , Vir int , Vir pos ] is formed, and Vir i is input into the trained support member defect detection and deformation recognition model to obtain the confidence Cl Vir under the virtual sample, and the change amount of the confidence Cl pre of the original sample is calculated to obtain the intervention benefit score Wherein, Scr i j represents the intervention benefit score of the j-th defect under the virtual sample of the i-th defect. By the same token, a counterfactual intervention simulation is performed for each defect to obtain a complete table of intervention benefits Scr1, Scr2,..., Scr N ; S33, failure path chain construction: first, according to the spatial position Di of each defect in the identification result of step S2 pre , the pairwise distance between defects is obtained represents the distance between the ith defect and the jth defect; Afterwards, if the defect is less than a preset distance discrimination threshold Dl thr , and the intervention benefit score Scr i j is greater than a preset intervention discrimination threshold Scr thr , it indicates that the ith defect is an upstream defect of the jth defect, that is, in the defect evolution process, the ith defect is prone to cause the generation of the jth defect. Based on this principle, all upstream defects of all defects are discriminated, and defect connection is sequentially performed in the direction from upstream to downstream, finally generating a failure path chain. Repeat the process until all defects are assigned, and obtain the failure path chain set SPC of the component.

5. The coal mine supporting member risk intelligent detection method based on multi-modal inversion reasoning according to claim 1, characterized in that, The step S4 based on failure path chain driven risk assessment and early warning output specifically comprises the following steps: S41, defect risk quantification: first, preset the risk weight w1, w2, …, w7 corresponding to each defect, and normalize the deformation variable according to the physical safety limit value as Di norm ∈[0,1] Afterwards, the defect position recognition result Dl pre Different defects are distinguished, and a risk quantization value of each defect is calculated, specifically R i i ·Di i , wherein R i represents the risk quantization value of the i-th defect, w i is the risk weight corresponding to the i-th defect, and Di i is the deformation quantity normalized value corresponding to the i-th defect; then, based on the method, the risk quantization values R1, R2, …, R N of each defect are obtained, and N is the total number of defects.​ S42, link risk aggregation: for the set of failure path chains SPC, the cumulative node risk value of each failure path chain from upstream to downstream is calculated, and the risk score R of each link is obtained by dividing the number of defects chain , and the sum of the risk scores of all links is taken as the comprehensive risk score R of the current component total ; at the same time, the link C with the largest risk score is retained max as the main risk source link of the current component; S43, component comprehensive risk judgment and traceability output: preset multi-stage risk level thresholds Risk1, Risk2, Risk3, Risk4, respectively corresponding to component safety, attention, warning or serious state, based on the comprehensive risk score R total Compare with the multi-stage risk level threshold to determine the risk detection level State of the component. Afterwards, based on the risk detection level State, the link C with the maximum risk score is selected max and the failure path chain set SPC generates a risk trace report, providing direct reference for downhole maintenance and intervention decision-making.

Images