Intelligent detection method and system for tunnel shotcrete based on multimodal information fusion
Through the intelligent detection method of multimodal information fusion, using deep learning framework and sensor data, the problem of subjectivity and inefficiency of traditional spray quality detection methods is solved, and more accurate and efficient spray layer quality evaluation and construction are achieved. Safety guarantee.
Patent Information
- Application Number
- CN202411067910.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-06
AI Technical Summary
Traditional spray quality detection methods rely on manual inspection and limited instrument measurement, and there are problems such as large differences in subjective judgments, inaccurate detection results, and low efficiency.
Using intelligent detection methods based on multimodal information fusion, the video image data, sound wave detection signals and vibration signals of the spray layer are collected in real time through high-definition cameras, sound wave detection devices and vibration sensors, and the feature fusion layers and crosses in the deep learning framework are used. Modal attention mechanism, dynamically adjusts feature weights, outputs spray quality scores, and sends an alarm when the score is below the threshold.
It improves the accuracy and efficiency of the quality evaluation of the spray layer, can promptly discover quality problems, ensure construction safety, reduce the risk of human misjudgment, and enhances the generalization ability and robustness of the system.
Smart Images

Figure CN118585963B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data processing, and in particular to a tunnel shotcrete intelligent detection method and system based on multi-modal information fusion. Background Art
[0002] In the field of tunnel engineering, shotcrete is widely used as an important support method in the initial support stage after tunnel excavation. It can not only quickly seal the surrounding rock and prevent the surrounding rock from weathering and instability, but also play a certain supporting role to ensure construction safety. However, in the actual construction process, the quality of the shotcrete layer directly affects the safety, stability and service life of the entire tunnel. Therefore, it is crucial to ensure the quality of the shotcrete layer. Traditional shotcrete quality detection methods mainly rely on manual inspection and limited instrument measurement. Although these methods are intuitive and easy to operate, they have obvious limitations.
[0003] Manual inspection is usually performed by experienced engineers or technicians who directly observe the appearance of the shotcrete layer on site, such as surface flatness, presence of obvious cracks, etc., supplemented by simple instrument measurements, such as using a hammer to tap and listen to the sound to determine whether there are cavities inside. This method can detect some obvious quality problems to a certain extent, but due to the large differences in human subjective judgment, the inspection results are often not accurate enough and the efficiency is low. Summary of the invention
[0004] The embodiments of the present invention provide a tunnel shotcrete intelligent detection method and system based on multimodal information fusion, which can at least solve the problem that the detection results are often not accurate enough and the efficiency is low due to the large differences in people's subjective judgment.
[0005] In a first aspect, an embodiment of the present invention provides a tunnel shotcrete intelligent detection method based on multimodal information fusion, comprising:
[0006] During the tunnel spraying process, at least three different types of sensors are used to collect video image data, acoustic wave detection signal data, and vibration signal data of the spraying layer in real time;
[0007] Extracting a plurality of data features from the video image data, the sound wave detection signal data, and the vibration signal data, wherein the plurality of data features have different modal types;
[0008] The features of different modalities are mapped into a shared space through the feature fusion layer in the deep learning framework, where the feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features;
[0009] Introducing a cross-modal attention mechanism in the shared space to learn the multiple data features and outputting correlation information between the data features;
[0010] A shotcrete quality score is determined based on the correlation information and weights corresponding to a plurality of data features, so that when the shotcrete quality score is lower than a threshold, an alarm message is sent to a preset device port.
[0011] Furthermore, features of different modalities are mapped into a shared space through a feature fusion layer in a deep learning framework, wherein the feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features, including:
[0012] Encode multiple data features using an encoder network in a deep learning framework, converting them into vector representations in a high-dimensional space;
[0013] Introducing a gating mechanism in the feature fusion layer to dynamically adjust the weights corresponding to multiple data features;
[0014] Use the decoder network in the deep learning framework to decode the fused features and map them back to the original data space to obtain the decoded features.
[0015] Furthermore, a gating mechanism is introduced into the feature fusion layer to dynamically adjust the weights corresponding to the multiple data features, including:
[0016] Initialize a set of gating units for the feature data of each modality, and the gating units are used to control the contribution of different modal features;
[0017] Update the weights of the gating units by calculating the correlation scores between features of different modalities;
[0018] According to the weight of the gating unit, the features of different modes are weighted to strengthen the contribution of important features;
[0019] The weighted features are fused to obtain the weights corresponding to each data feature.
[0020] Furthermore, the step of updating the weight of the gating unit by calculating the correlation scores between the features of different modalities includes:
[0021] Initialize an initial weight for each gating unit;
[0022] Calculating correlation scores between multiple data features in the video image data, the sound wave detection signal data, and the vibration signal data;
[0023] The weights of the gating units are updated based on the calculated correlation scores.
[0024] Further, the step of determining the shotcrete quality score based on the correlation information and weights corresponding to the plurality of data features comprises:
[0025] The relevance information calculated using the cross-modal attention mechanism and the weights dynamically adjusted by the gating mechanism are used as input, where the relevance information represents the relevance matrix between data features, where each element represents the relevance score between the i-th feature and the j-th data feature;
[0026] Calculate the sum of the weighted correlation scores of each feature. That is, for each feature i, multiply the weight of the ith feature by the sum of the correlation scores of the feature and all other features, and then multiply by the quality score of the ith feature. The final shotcrete quality score is solved based on the weighted correlation scores of all data features.
[0027] Further, the step of solving the final shotcrete quality score according to the weighted correlation scores of all features includes:
[0028] Performing a first-layer nonlinear transformation on the correlation scores of all data features to obtain a first-layer nonlinear information, where the first-layer nonlinear information includes basic information between different sensor data;
[0029] Performing a second layer of nonlinear transformation based on the first layer of nonlinear information to obtain a second layer of nonlinear information, wherein the second layer of nonlinear information includes intrinsic correlation information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data;
[0030] Performing a third layer of nonlinear transformation based on the second layer of nonlinear information to obtain a third layer of nonlinear information, wherein the third layer of nonlinear information includes interaction information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data;
[0031] The shotcrete quality score is calculated based on the third layer of nonlinear information.
[0032] Furthermore, it also includes:
[0033] Extracting features of surface defects from video image data, and extracting features related to the internal structure of the shotcrete layer from the acoustic wave detection signal data and the vibration signal data;
[0034] Calculate the correlation scores between features of different modalities;
[0035] Use Swish function or Sinusoidal function to capture the highest level of nonlinear relationships to learn the complex shape, location, size and orientation of surface defects interacting with the responses in the acoustic inspection signal data and vibration signal data;
[0036] The final shotcrete quality score is determined by averaging the sum of the weighted correlation scores after nonlinear processing of the data features.
[0037] In a second aspect, an embodiment of the present invention provides a tunnel shotcrete intelligent detection system based on multimodal information fusion, comprising:
[0038] An acquisition module is used to acquire video image data, acoustic wave detection signal data and vibration signal data of the shotcrete layer in real time through at least three different types of sensors during the tunnel shotcrete process;
[0039] An extraction module, used to extract multiple data features from the video image data, the sound wave detection signal data and the vibration signal data, wherein the multiple data features have different modal types;
[0040] A learning module for mapping features of different modalities into a shared space through a feature fusion layer in a deep learning framework, where the feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features;
[0041] An introduction mechanism module is used to introduce a cross-modal attention mechanism in a shared space to learn the multiple data features and output correlation information between the data features;
[0042] A scoring module is used to determine a shotcrete quality score based on the correlation information and weights corresponding to a plurality of data features, so as to send an alarm message to a preset device port when the shotcrete quality score is lower than a threshold value.
[0043] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, characterized in that a computer program is stored in the memory, and the processor is configured to run the computer program to execute the tunnel shotcrete intelligent detection method based on multimodal information fusion described in the first aspect.
[0044] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, in which a computer program is stored. The computer program includes a program code for controlling a process to execute a process, and the process includes the intelligent detection method for tunnel grouting based on multimodal information fusion described in the first aspect.
[0045] The embodiment of the present invention collects video image data, acoustic wave detection signal data and vibration signal data of the shotcrete layer in real time through at least three different types of sensors during the tunnel shotcrete process; extracts multiple data features from the video image data, acoustic wave detection signal data and vibration signal data, wherein the modal types of the multiple data features are different; maps features of different modalities into a shared space through a feature fusion layer in a deep learning framework, wherein the feature fusion layer dynamically adjusts weights corresponding to multiple data features using a gating mechanism; introduces a cross-modal attention mechanism in the shared space to learn the multiple data features, and outputs correlation information between the data features; determines a shotcrete quality score based on the correlation information and the weights corresponding to the multiple data features, so that when the shotcrete quality score is lower than a threshold, an alarm message is sent to a preset device port, and the accuracy of the shotcrete layer quality assessment is improved through the steps of feature extraction, feature fusion, cross-modal attention mechanism and shotcrete quality scoring, and can timely detect quality problems, send alarm messages, and ensure construction safety.
[0046] The beneficial effects of the embodiments of the present invention can refer to the technical effects corresponding to the technical features in the specific implementation methods, which will not be described in detail here. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 A flow chart of a tunnel shotcrete intelligent detection method based on multimodal information fusion provided by an embodiment of the present invention;
[0048] Figure 2 A schematic diagram of the modules of the intelligent detection system for tunnel shotcrete based on multimodal information fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0050] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.
[0051] For instrument measurement, although more objective data can be obtained through professional testing equipment, these methods can only sample locally and cannot cover the entire shotcrete surface, resulting in insufficient representativeness of the test results. For example, ultrasonic flaw detectors can be used to detect whether there are cavities or cracks inside the shotcrete layer, but this detection method needs to be performed point by point, which is time-consuming and requires professional technicians to operate, increasing the cost of detection. At the same time, these instruments can usually only detect problems within a certain depth below the surface, and it is difficult to detect deep problems.
[0052] In view of the above problems, it is necessary to develop a method that can automatically and efficiently detect the quality of shotcrete. The new detection technology should be able to overcome the shortcomings of traditional methods and achieve a comprehensive, accurate and efficient evaluation of the quality of the shotcrete layer. Such technology can not only improve detection efficiency and reduce labor costs, but also greatly reduce the risk of misjudgment caused by human factors, thereby effectively ensuring the quality and safety of tunnel projects.
[0053] Based on this, the embodiment of the present invention first provides a tunnel shotcrete intelligent detection method based on multimodal information fusion, such as Figure 1 As shown, including:
[0054] Step S1, during the tunnel shotcrete process, real-time video image data, acoustic wave detection signal data, and vibration signal data of the shotcrete layer are collected by at least three different types of sensors;
[0055] The embodiment of the present invention can collect video image data, sound wave detection signal data and vibration signal data of the shotcrete layer in real time, wherein the video image data: uses a high-definition camera to shoot the surface of the tunnel shotcrete layer in real time; the sound wave detection signal data: uses a sound wave transmitter and a receiver to emit sound waves to the shotcrete layer and record the reflected signals; the vibration signal data: uses a vibration sensor to record the vibration of the shotcrete layer during the construction process.
[0056] Specific example: During the tunnel shotcrete construction process, a high-definition camera is used to capture the surface of the tunnel shotcrete layer in real time, while an acoustic wave detection device is used to emit acoustic waves to the shotcrete layer and record the reflected signals. In addition, a vibration sensor is used to record the vibration of the shotcrete layer during the construction process.
[0057] Beneficial effects: By collecting multimodal data in real time, the status of the shotcrete layer can be fully understood, providing rich information support for subsequent analysis.
[0058] Step S2, extracting multiple data features from the video image data, the sound wave detection signal data, and the vibration signal data, wherein the multiple data features have different modal types;
[0059] The embodiment of the present invention extracts multiple data features from video image data, sound wave detection signal data and vibration signal data, wherein, for video image data, computer vision technology is used to extract the position, size, shape and other features of surface defects; for sound wave detection signal data, signal processing technology is used to extract the reflection intensity, delay time and other features; for vibration signal data, spectrum analysis is used to extract the vibration frequency, amplitude and other features.
[0060] Specific examples: Extracting features such as size, shape, and location of surface defects from video image data; extracting features such as reflection intensity and delay time from acoustic wave detection signal data; extracting features such as vibration frequency and amplitude from vibration signal data.
[0061] Beneficial effects: By extracting the features of different modes, the state of the shotcrete layer can be analyzed more comprehensively, providing diversified information for subsequent fusion.
[0062] Step S3, mapping the features of different modalities into a shared space through a feature fusion layer in a deep learning framework, wherein the feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features;
[0063] The embodiments of the present invention can map features of different modalities into a shared space, and use a gating mechanism to dynamically adjust weights corresponding to multiple data features, wherein feature mapping: using a feature fusion layer in a deep learning framework to map features of different modalities into a shared space; gating mechanism: dynamically adjusting weights corresponding to multiple data features through a gating unit to more accurately reflect the importance of different features.
[0064] Specific example: Use the feature fusion layer to map the features in the video image data, sound wave detection signal data, and vibration signal data into a shared space, and use the gating unit to dynamically adjust the weights of these features.
[0065] Beneficial effects: Through feature fusion and gating mechanism, the importance of different features can be reflected more accurately, thereby improving the accuracy of shotcrete layer quality assessment.
[0066] Step S4, introducing a cross-modal attention mechanism in the shared space to learn the multiple data features, and outputting correlation information between the data features;
[0067] An embodiment of the present invention introduces a cross-modal attention mechanism in a shared space to learn the multiple data features and outputs correlation information between the data features, wherein the cross-modal attention mechanism: uses the attention mechanism to learn the correlation between different modal features, thereby capturing the interaction between different modal features; correlation information output: outputs the correlation scores between the data features, which reflect the strength of the interaction between the features.
[0068] Specific example: Use the cross-modal attention mechanism to learn the correlation between features in video image data, sound wave detection signal data, and vibration signal data, and output the correlation score.
[0069] Beneficial effects: Through the cross-modal attention mechanism, the interaction between different modal features can be more accurately captured, improving the accuracy of shotcrete layer quality assessment.
[0070] Step S5, determining a shotcrete quality score based on the correlation information and weights corresponding to the plurality of data features, so that when the shotcrete quality score is lower than a threshold, an alarm message is sent to a preset device port;
[0071] The embodiment of the present invention determines the shotcrete quality score based on the correlation information and the weights corresponding to the multiple data features, and sends an alarm message when the score is lower than a threshold. The shotcrete quality score: determines the shotcrete quality score based on the correlation information and the weights corresponding to the multiple data features; the alarm mechanism: sends an alarm message to a preset device port when the shotcrete quality score is lower than a preset threshold.
[0072] Specific example: Based on the correlation information calculated by the cross-modal attention mechanism and the weights dynamically adjusted by the gating mechanism, the shotcrete quality score is determined. If the score is lower than the preset threshold (for example, 70 points), an alarm message is sent to the preset device port (for example, mobile phone application, computer software, etc.).
[0073] Beneficial effect: By sending alarm information in a timely manner, quality problems of the shotcrete layer can be quickly discovered, measures can be taken to repair them, and potential safety hazards can be avoided.
[0074] The embodiment of the present invention is based on a tunnel shotcrete intelligent detection method based on multimodal information fusion. The method can collect multimodal data of the shotcrete layer in real time. Through the steps of feature extraction, feature fusion, cross-modal attention mechanism and shotcrete quality scoring, the accuracy of the shotcrete layer quality assessment is improved, and quality problems can be discovered in time, and alarm information can be sent to ensure construction safety. This method not only improves the accuracy of the shotcrete layer quality assessment, but also enhances the generalization ability and robustness of the system, providing effective technical support for tunnel shotcrete engineering.
[0075] In another embodiment provided by the present invention, features of different modalities are mapped into a shared space through a feature fusion layer in a deep learning framework, wherein the feature fusion layer uses a gating mechanism to dynamically adjust weights corresponding to multiple data features, including:
[0076] Step 21: Encode multiple data features using an encoder network in a deep learning framework and convert them into vector representations in a high-dimensional space;
[0077] An encoder network is used to encode features of different modalities and convert them into vector representations in a high-dimensional space, wherein the encoder network: uses an encoder network in a deep learning framework to encode features in video image data, sound wave detection signal data, and vibration signal data; vector representation: converts features of different modalities into vector representations in a high-dimensional space.
[0078] Specific examples: Use convolutional neural networks (CNN) to encode features in video image data, use recurrent neural networks (RNN) or long short-term memory networks (LSTM) to encode features in sound wave detection signal data, and use convolutional neural networks (CNN) to encode features in vibration signal data.
[0079] Beneficial effect: Through the encoder network, the features of different modalities can be converted into vector representations in high-dimensional space, which is convenient for subsequent fusion processing.
[0080] Step 22: Introduce a gating mechanism in the feature fusion layer to dynamically adjust the weights corresponding to multiple data features;
[0081] A gating mechanism is introduced in the feature fusion layer to dynamically adjust the weights corresponding to multiple data features. Feature fusion: Use the feature fusion layer to map features of different modalities into a shared space. Gating mechanism: Dynamically adjust the weights corresponding to multiple data features through the gating unit to more accurately reflect the importance of different features.
[0082] Specific example: Use the feature fusion layer to map the encoded features in the video image data, sound wave detection signal data, and vibration signal data into a shared space, and use the gating unit to dynamically adjust the weights of these features.
[0083] Beneficial effects: Through feature fusion and gating mechanism, the importance of different features can be reflected more accurately, thereby improving the accuracy of shotcrete layer quality assessment.
[0084] Step 23: Use the decoder network in the deep learning framework to decode the fused features and map them back to the original data space to obtain the decoded features.
[0085] Use the decoder network to decode the fused features and map them back to the original data space to obtain the decoded features, where the decoder network: use the decoder network in the deep learning framework to decode the fused features and map them back to the original data space: map the fused features back to the original data space.
[0086] Specific example: Use the decoder network to decode the features after feature fusion, map them back to the original data space, and obtain the decoded features.
[0087] Beneficial effect: Through the decoder network, the fused features can be mapped back to the original data space, which is convenient for subsequent analysis and application.
[0088] The feature fusion step in the tunnel shotcrete intelligent detection method based on multimodal information fusion in the embodiment of the present invention can encode the features of different modes through the encoder network, and then dynamically adjust the feature weights using the gating mechanism in the feature fusion layer, and finally map the fused features back to the original data space through the decoder network. This method not only improves the accuracy of the shotcrete layer quality assessment, but also enhances the generalization ability and robustness of the system, providing effective technical support for tunnel shotcrete engineering.
[0089] In practical applications, in a tunnel shotcrete construction project, the quality of the shotcrete layer needs to be monitored in real time. To this end, the embodiment of the present invention uses three different types of sensors to collect data: a high-definition camera captures the surface image of the tunnel shotcrete layer in real time; an acoustic wave transmitter and receiver are used to detect the internal structure of the shotcrete layer; and a vibration sensor records the vibration of the shotcrete layer during the construction process.
[0090] First, a high-definition camera is used to capture the surface of the tunnel shotcrete layer in real time. An acoustic wave transmitter and receiver are used to emit acoustic waves to the shotcrete layer and record the reflected signals. A vibration sensor is used to record the vibration of the shotcrete layer during construction.
[0091] Then, computer vision technology is used to extract the location, size, shape and other features of surface defects, signal processing technology is used to extract reflection intensity, delay time and other features, and spectrum analysis is used to extract vibration frequency, amplitude and other features.
[0092] Secondly, convolutional neural network (CNN) is used to encode the features in video image data, recurrent neural network (RNN) or long short-term memory network (LSTM) is used to encode the features in sound wave detection signal data, and convolutional neural network (CNN) is used to encode the features in vibration signal data.
[0093] Next, a feature fusion layer is used to map the encoded features into a shared space, and a gating unit is used to dynamically adjust the weights corresponding to multiple data features in order to more accurately reflect the importance of different features. The decoder network is used to decode the fused features and map them back to the original data space to obtain the decoded features.
[0094] Finally, the shotcrete quality score is determined based on the decoded features and the weights dynamically adjusted by the gating mechanism. When the shotcrete quality score is lower than the preset threshold, an alarm message is sent to the preset device port.
[0095] Beneficial Effects
[0096] Improved accuracy: The features of different modalities are converted into vector representations in high-dimensional space through the encoder network, and the feature weights are dynamically adjusted through the gating mechanism in the feature fusion layer, which improves the accuracy of shotcrete layer quality assessment.
[0097] Enhanced generalization capability: By using the encoder network, feature fusion layer, and decoder network in the deep learning framework, the generalization capability of the model is improved, enabling the model to better adapt to different shotcrete environments and conditions.
[0098] Enhanced robustness: By using a gating mechanism to dynamically adjust feature weights, the model can better cope with the impact of noise and abnormal data, thereby improving the robustness of the model.
[0099] Real-time monitoring and alarms: By monitoring the quality of the shotcrete layer in real time and sending alarm messages immediately when problems are found, timely measures can be taken to repair them and avoid potential safety hazards.
[0100] Through the above embodiments, the embodiment of the present invention realizes the feature fusion step in the tunnel shotcrete intelligent detection method based on multimodal information fusion, and specifically demonstrates how to use the encoder network, feature fusion layer and decoder network to improve the accuracy of shotcrete layer quality assessment. This method not only improves the accuracy of shotcrete layer quality assessment, but also enhances the generalization ability and robustness of the system, providing effective technical support for tunnel shotcrete engineering.
[0101] In another embodiment provided by the present invention, a tunnel shotcrete intelligent detection method based on multimodal information fusion is characterized in that a gating mechanism is introduced in a feature fusion layer to dynamically adjust the weights corresponding to multiple data features, including:
[0102] Step S31, initializing a set of gating units for the feature data of each modality, the gating units are used to control the contribution of different modal features;
[0103] A set of gating units is initialized for the characteristic data of each mode to control the contribution of different modal features, wherein the gating units are initialized: the gating units are initialized for the video image data, the sound wave detection signal data and the vibration signal data; the function of the gating units: the gating units control the contribution of different modal features.
[0104] Specific example: a group of gating units are initialized for video image data, sound wave detection signal data and vibration signal data respectively, and each gating unit is responsible for controlling the contribution of the corresponding modal feature.
[0105] Beneficial effect: By initializing the gating unit, the contribution of different modal features can be controlled to more accurately reflect the importance of different features.
[0106] Step S32, updating the weight of the gating unit by calculating the correlation scores between the features of different modalities;
[0107] The weight of the gating unit is updated by calculating the correlation score between the features of different modalities, wherein the correlation score calculation: the correlation score between the features of different modalities is calculated using the cross-modal attention mechanism; the weight update: the weight of the gating unit is updated according to the correlation score.
[0108] Specific example: Use the cross-modal attention mechanism to calculate the correlation scores between video image data, sound wave detection signal data, and vibration signal data, and update the weights of the gated units based on these scores.
[0109] Beneficial effects: By calculating the correlation scores between different modal features and updating the weights of the gating units, the interactions between different features can be more accurately reflected, thereby improving the accuracy of shotcrete layer quality assessment.
[0110] Step S33, weighting the features of different modes according to the weights of the gating units to strengthen the contribution of important features;
[0111] According to the weight of the gating unit, the features of different modes are weighted to strengthen the contribution of important features, wherein the weighting operation: weighting the features of different modes according to the weight of the gating unit; strengthening important features: strengthening the contribution of important features with higher weights.
[0112] Specific example: The video image data, sound wave detection signal data, and vibration signal data are weighted according to the updated weights of the gating unit to enhance the contribution of important features.
[0113] Beneficial effects: The accuracy of shotcrete layer quality assessment can be improved by strengthening the contribution of important features through weighting operation.
[0114] Step S34, the weighted features are fused to obtain the weights corresponding to each data feature.
[0115] The weighted features are fused to obtain the weights corresponding to the various data features, wherein feature fusion: the weighted features are fused; weight determination: the weights corresponding to the various data features are obtained.
[0116] Specific example: The weighted video image data, sound wave detection signal data and vibration signal data are fused to obtain the weight corresponding to each data feature.
[0117] Beneficial effect: The weights corresponding to each data feature are obtained by feature fusion, which can more accurately reflect the importance of different features for the quality assessment of the shotcrete layer.
[0118] The embodiment of the present invention dynamically adjusts the feature weights through a gating mechanism, thereby improving the accuracy of the shotcrete layer quality assessment. By using the gating mechanism to dynamically adjust the feature weights, the generalization ability of the model is improved, so that the model can better adapt to different shotcrete environments and conditions. By using the gating mechanism to dynamically adjust the feature weights, the model can better cope with the influence of noise and abnormal data, thereby improving the robustness of the model. Finally, the shotcrete layer quality can be monitored in real time, and an alarm message can be sent immediately when a problem is found, so that measures can be taken in time to repair it and avoid potential safety hazards. Based on this, further in the embodiment of the present invention, the step of updating the weight of the gating unit by calculating the correlation score between different modal features includes:
[0119] Initialize a set of gating units for the feature data of each modality. The gating units are used to control the contribution of the feature of this type, and initialize an initial weight for each gating unit.
[0120] Calculating correlation scores between multiple data features in the video image data, the sound wave detection signal data, and the vibration signal data;
[0121] The weights of the gating units are updated based on the calculated correlation scores.
[0122] set up is the number of modes, is the number of features for each mode, For the The feature set of the modalities, For the The feature correlation scores of the modalities, For the The normalized value of the feature correlation score of the modality, For the The initial weights of the gating units of the modalities, For the The gate control unit of this mode is The weight after iterations, is the average of all correlation scores, is the learning rate, For the Normalized weights of the gating units of the modality.
[0123] ;
[0124] Where:
[0125] ;
[0126] ;
[0127] ;
[0128] ;
[0129] ;
[0130] ;
[0131] ;
[0132] ;
[0133] In this embodiment of the present invention, the initial weights of all control units are Equal, both , through the function Calculate feature relevance scores for each modality ,in Indicates The k-th feature of the modality.
[0134] pass Normalize the feature correlation scores of each modality, where and Respectively represent The mean and standard deviation of the feature correlation scores of the modalities, A minimum value used to prevent division by zero.
[0135] pass Update the weights by Normalize the weights to ensure that the sum of the weights of all gating units is 1.
[0136] By dynamically adjusting the weights of the gating units, the model is able to automatically adjust the weights of different modal features based on the correlation scores between the features, which helps to improve the generalization ability of the model, meaning that the model can better adapt in different tunnel shotcrete environments and reduce dependence on specific environments or conditions.
[0137] The embodiment of the present invention improves the stability of the model and reduces the model performance fluctuation caused by individual feature anomalies by introducing smoothing terms and average feature score calculation; through dynamic weight updating and normalization processing, the model can more accurately identify and evaluate the quality of the shotcrete layer, thereby improving the accuracy of the detection results; the dynamic weight adjustment mechanism enables the model to better adapt to different shotcrete environments and conditions, thereby improving the generalization ability of the model.
[0138] In summary, the embodiments of the present invention can realize a tunnel shotcrete intelligent detection method based on multimodal information fusion that is more stable, accurate and has stronger generalization ability.
[0139] In another embodiment provided by the present invention, the step of determining the shotcrete quality score based on the correlation information and the weights corresponding to the plurality of data features comprises:
[0140] Step S51, using the correlation information calculated by the cross-modal attention mechanism and the weight dynamically adjusted by the gating mechanism as input, wherein the correlation information represents a correlation matrix between data features, wherein each element represents a correlation score between the i-th feature and the j-th data feature;
[0141] In the embodiment of the present invention, the correlation information calculated by the cross-modal attention mechanism and the weight dynamically adjusted by the gating mechanism are used as input. The correlation information represents the correlation matrix between features, where each element represents the correlation score between the i-th feature and the j-th feature. The weight vector of the gating unit, where each element represents the weight of the i-th feature.
[0142] Step S52, calculate the sum of the weighted correlation scores of each feature, that is, for each feature i, multiply the weight of the ith feature by the sum of the correlation scores of the feature and all other features, and then multiply by the quality score of the ith feature, and solve the final shotcrete quality score based on the weighted correlation scores of all data features.
[0143] In an embodiment of the present invention, a comprehensive evaluation function is defined, which calculates the shotcrete quality score through the following steps: for each feature i, the weight of the i-th feature is multiplied by the weighted sum of the correlation scores of the feature with all other features, and then multiplied by the quality score of the i-th feature, and a nonlinear function is used to further process these weighted correlation scores, such as a Sigmoid function or a ReLU function, to capture the nonlinear relationship, and the sum of the weighted correlation scores of all features is averaged to obtain the final shotcrete quality score.
[0144] Furthermore, in order to realize a more accurate, more generalized and more robust intelligent detection method for tunnel shotcrete based on multimodal information fusion, each layer of nonlinear transformation has its specific purpose and function, and works together to improve the overall performance of the model. The embodiment of the present invention elaborates on the relationship interpretation of the video image data, acoustic wave detection signal data and vibration signal data of the shotcrete layer collected in real time by three different types of sensors, wherein the steps of solving the final shotcrete quality score according to the weighted correlation scores of all features include:
[0145] The first layer of nonlinear transformation is performed on the correlation scores of all data features, that is, using a nonlinear function , processing weighted relevance scores , to capture the nonlinear relationship and obtain the first layer of nonlinear information, which includes basic information between different sensor data;
[0146] Performing a second layer of nonlinear transformation based on the first layer of nonlinear information to obtain a second layer of nonlinear information, wherein the second layer of nonlinear information includes intrinsic correlation information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data;
[0147] In the embodiment of the present invention, the design reasons are:
[0148] Capturing basic nonlinear relationships: The quality assessment of the shotcrete layer involves multiple factors, and the relationships between these factors are often nonlinear. Through the first layer of nonlinear transformation, the embodiment of the present invention can capture these basic nonlinear relationships.
[0149] Relationship with sensor data: The information provided by different types of sensors (video image data, acoustic wave detection signal data, and vibration signal data) may have nonlinear relationships. The first layer of nonlinear transformation helps capture these relationships.
[0150] Technical effect:
[0151] Capturing basic properties of the shotcrete layer: The first layer of nonlinear transformation can capture the basic properties of the shotcrete layer, such as the relationship between surface defects captured by video image data and internal voids captured by acoustic detection signal data and vibration signal data.
[0152] Improved accuracy: Through nonlinear transformation, the model can more accurately reflect the quality status of the shotcrete layer.
[0153] Improved generalization capability: Capturing basic nonlinear relationships helps the model better adapt to different shotcrete environments and conditions.
[0154] In practical applications, the Sigmoid function can be used And the ReLU function The raw weighted correlation scores are processed to capture the underlying nonlinear relationships.
[0155] Performing a third layer of nonlinear transformation based on the second layer of nonlinear information to obtain a third layer of nonlinear information, wherein the third layer of nonlinear information includes interaction information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data;
[0156] In the embodiment of the present invention, the design reasons are:
[0157] Capturing deeper nonlinear relationships: Through the second layer of nonlinear transformation, the embodiments of the present invention can further capture deeper nonlinear relationships, which is crucial to improving the expressiveness and accuracy of the model.
[0158] Improve model expressiveness: Deep models usually have stronger expressiveness and can fit more complex function forms. The second layer of nonlinear transformation helps improve the expressiveness of the model.
[0159] Technical effect:
[0160] Capturing complex characteristics of the shotcrete layer: The second layer of nonlinear transformation is able to capture the complex characteristics of the shotcrete layer, for example, the complex relationship between surface defects captured by the video image data and internal voids captured by the acoustic detection signal data and vibration signal data.
[0161] Improved accuracy: Through deeper nonlinear transformations, the model can more accurately reflect the quality status of the shotcrete layer.
[0162] Improved generalization: Capturing deeper nonlinear relationships helps the model better adapt to different shotcrete environments and conditions.
[0163] In practical applications, the Softplus function can be used:
[0164] ;
[0165] And the Leaky ReLU function:
[0166] ; Further process the results of the first layer of nonlinear transformation to capture deeper nonlinear relationships.
[0167] The shotcrete quality score is calculated based on the third layer of nonlinear information.
[0168] In the embodiment of the present invention, the design reasons are:
[0169] Capturing more complex nonlinear relationships: Through the third layer of nonlinear transformation, the embodiments of the present invention can capture more complex nonlinear relationships, which is crucial to improving the expressiveness and accuracy of the model.
[0170] Improved accuracy: Through more complex nonlinear transformations, the model can more accurately reflect the quality status of the shotcrete layer, thereby improving the accuracy of the detection results.
[0171] The effect achieved:
[0172] Capturing advanced properties of the shotcrete layer: The third layer of nonlinear transformations is able to capture advanced properties of the shotcrete layer, such as the very complex interaction between surface defects captured by the video image data and internal voids captured by the acoustic detection signal data and vibration signal data.
[0173] Improved accuracy: By capturing more complex nonlinear relationships, the model is able to more accurately reflect the quality status of the shotcrete layer.
[0174] Improved generalization capability: Capturing more complex nonlinear relationships helps the model better adapt to different shotcrete environments and conditions.
[0175] Enhanced robustness: By using multiple layers of nonlinear transformations to process data, the model can better cope with the impact of noise and abnormal data, thereby improving the robustness of the model.
[0176] In practical applications, the Exponential Linear Unit (ELU) function can be used:
[0177] ;
[0178] GELU function (Gaussian Error Linear Unit):
[0179] ;
[0180] First layer of nonlinear transformation: At this layer, the embodiments of the present invention mainly capture the basic relationship between different sensor data. For example, the basic correlation between surface defects in video image data and internal cavities in acoustic detection signal data and vibration signal data. These relationships are the basis for the quality assessment of the shotcrete layer.
[0181] Second layer of nonlinear transformation: At this layer, the embodiments of the present invention further capture the deeper nonlinear relationships between different sensor data. For example, the complex relationship between surface defects in video image data and internal cavities in acoustic wave detection signal data and vibration signal data. These relationships may involve more complex patterns, such as how the degree of surface defects affects the response of acoustic wave detection signal data and vibration signal data.
[0182] The third layer of nonlinear transformation: At this layer, the embodiments of the present invention capture very complex nonlinear relationships between different sensor data. For example, the high-level interactions between surface defects in video image data and internal voids in acoustic detection signal data and vibration signal data. These relationships may involve higher-level patterns and complex pattern interactions, such as how the shape and position of surface defects affect the response of acoustic detection signal data and vibration signal data.
[0183] Furthermore, a new step, the fourth layer of nonlinear transformation, can be added to the original three layers of nonlinear transformation. This layer will focus on capturing higher-level nonlinear relationships and further improve the expressiveness and accuracy of the model, including:
[0184] Extracting features of surface defects from video image data, and extracting features related to the internal structure of the shotcrete layer from the acoustic wave detection signal data and the vibration signal data;
[0185] Calculate the correlation scores between features of different modalities;
[0186] Use Swish function or Sinusoidal function to capture the highest level of nonlinear relationships to learn the complex shape, location, size and orientation of surface defects interacting with the responses in the acoustic inspection signal data and vibration signal data;
[0187] The final shotcrete quality score is determined by averaging the sum of the weighted correlation scores after nonlinear processing of the data features.
[0188] In the embodiment of the present invention, through the fourth layer of nonlinear transformation, the embodiment of the present invention can capture higher-level nonlinear relationships, which is crucial to improving the expressiveness and accuracy of the model.
[0189] Improved accuracy: Through more complex nonlinear transformations, the model can more accurately reflect the quality status of the shotcrete layer, thereby improving the accuracy of the detection results.
[0190] Technical effect:
[0191] Capturing advanced properties of the shotcrete layer: The fourth layer of nonlinear transformation is able to capture advanced properties of the shotcrete layer, such as the very complex interaction between surface defects captured by the video image data and internal voids captured by the acoustic detection signal data and vibration signal data.
[0192] Improved accuracy: By capturing higher levels of nonlinear relationships, the model is able to more accurately reflect the quality status of the shotcrete layer.
[0193] Improved generalization: Capturing higher-level nonlinear relationships helps the model better adapt to different shotcrete environments and conditions.
[0194] Enhanced robustness: By using multiple layers of nonlinear transformations to process data, the model can better cope with the impact of noise and abnormal data, thereby improving the robustness of the model.
[0195] Incorporating sensor data from the tunnel shotcrete process
[0196] First layer of nonlinear transformation: At this layer, the embodiments of the present invention mainly capture the basic relationship between different sensor data. For example, the basic correlation between surface defects in video image data and internal cavities in acoustic detection signal data and vibration signal data. These relationships are the basis for the quality assessment of the shotcrete layer.
[0197] Second layer of nonlinear transformation: At this layer, the embodiments of the present invention further capture the deeper nonlinear relationships between different sensor data. For example, the complex relationship between surface defects in video image data and internal cavities in acoustic wave detection signal data and vibration signal data. These relationships may involve more complex patterns, such as how the degree of surface defects affects the response of acoustic wave detection signal data and vibration signal data.
[0198] The third layer of nonlinear transformation: At this layer, the embodiments of the present invention capture very complex nonlinear relationships between different sensor data. For example, the high-level interactions between surface defects in video image data and internal voids in acoustic detection signal data and vibration signal data. These relationships may involve higher-level patterns and complex pattern interactions, such as how the shape and position of surface defects affect the response of acoustic detection signal data and vibration signal data.
[0199] Fourth layer of nonlinear transformation: At this layer, the embodiments of the present invention capture the highest level of nonlinear relationships between different sensor data. For example, extremely complex interactions between surface defects in video image data and internal voids in acoustic detection signal data and vibration signal data. These relationships may involve very high-level patterns and interactions of complex patterns, such as how the complex shape, location, size, and orientation of surface defects interact with the responses in acoustic detection signal data and vibration signal data.
[0200] Through the newly added fourth layer of nonlinear transformation, the embodiment of the present invention can realize a more accurate, more generalized and robust intelligent detection method for tunnel shotcrete based on multimodal information fusion. Each layer of nonlinear transformation has its specific purpose and function, which work together to improve the overall performance of the model and can capture nonlinear relationships at different levels, thereby improving the accuracy of shotcrete layer quality assessment. Through these steps, the embodiment of the present invention can not only capture various basic and complex relationships involved in the shotcrete layer quality assessment, but also improve the generalization ability and robustness of the model, so that it performs better in different shotcrete environments.
[0201] In order to specifically illustrate how the complex shape, position, size and direction of surface defects captured by the fourth layer nonlinear transformation interact with the responses in the acoustic wave detection signal data and the vibration signal data, an embodiment of the present invention will be described in detail.
[0202] Example: Interaction of surface defects with acoustic and vibration signals
[0203] Assume that in the process of evaluating the quality of the tunnel shotcrete layer, the embodiment of the present invention collects the video image data, the acoustic wave detection signal data and the vibration signal data of the shotcrete layer in real time through at least three different types of sensors. In this embodiment, the embodiment of the present invention will focus on the interaction between the surface defects and the acoustic wave detection signal data and the vibration signal data.
[0204] Step 1: Feature extraction and correlation score calculation:
[0205] Video image data: Extract the features of surface defects from video image data, including the shape, location, size and direction of the defects.
[0206] Acoustic wave detection signal data: Extract features related to the internal structure of the shotcrete layer from the acoustic wave detection signal data.
[0207] Vibration signal data: Extract features related to the internal structure of the shotcrete layer from the vibration signal data.
[0208] Correlation score calculation: The correlation scores between different modality features are calculated using the previous method.
[0209] Step 2: Nonlinear transformation:
[0210] The first layer of nonlinear transformation: processes the original weighted correlation scores to capture the underlying nonlinear relationships.
[0211] Second layer of nonlinear transformation: further processes the results of the first layer of nonlinear transformation to capture deeper nonlinear relationships.
[0212] The third layer of nonlinear transformation: further deepens the learning ability of the model and captures more complex nonlinear relationships.
[0213] Step 4: The fourth layer of nonlinear transformation:
[0214] Purpose: To further deepen the learning ability of the model and capture the highest level of nonlinear relationships.
[0215] Function used: Swish function or Sinusoidal function.
[0216] Design reason: To capture the highest level of nonlinear relationships and improve the expressiveness and accuracy of the model.
[0217] Effect achieved: By capturing the highest level of nonlinear relationships, the model is able to more accurately reflect the quality status of the shotcrete layer.
[0218] Complex shape of surface defects: Assume that a surface defect in the video image data presents a complex shape, such as an irregular oval or a composite shape composed of multiple small defects. This complex shape may cause specific pattern changes in the acoustic detection signal data and the vibration signal data.
[0219] Location and size: If the surface defect is located in a specific location in the shotcrete layer, or has a specific size, then these characteristics may have a specific impact on the acoustic and vibration signal data. For example, a small defect located at the edge of the shotcrete layer may not cause significant acoustic reflection, but a larger defect will cause significant acoustic reflection and vibration response.
[0220] Direction: The direction of the surface defect may also affect the acoustic and vibration signal data. For example, a defect perpendicular to the surface of the shotcrete layer may produce a different acoustic reflection pattern than a defect horizontal to the surface.
[0221] Fourth layer of nonlinear transformation: In this layer, embodiments of the present invention use Swish functions or Sinusoidal functions to capture these very complex interactions. These functions can help the model learn how the complex shape, location, size and direction of surface defects interact with the responses in the acoustic detection signal data and the vibration signal data.
[0222] The specific steps are as follows:
[0223] Feature extraction: Extract the features of surface defects from the video image data, and extract the features related to the internal structure of the shotcrete layer from the acoustic detection signal data and vibration signal data.
[0224] Correlation score calculation,The correlation scores between different modality features are calculated using the,above steps.
[0225] The first three layers of nonlinear transformation: use the Sigmoid function or the ReLU function to process the weighted correlation score; use the Softplus function or the Leaky ReLU function for further processing; use the ELU function or the GELU function to further deepen the learning ability.
[0226] Fourth-level nonlinear transformation: Use Swish function or Sinusoidal function to capture the highest level of nonlinear relationship.
[0227] Specifically, the model learns how the complex shape, location, size, and orientation of surface defects interact with the responses in the acoustic detection signal data and the vibration signal data, and averages the weighted correlation scores of all features after nonlinear processing to obtain the final shotcrete quality score.
[0228] Technical effect:
[0229] Improved accuracy: By capturing the highest level of nonlinear relationships, the model is able to more accurately reflect the quality state of the shotcrete layer.
[0230] Improved generalization: Capturing the highest level of nonlinear relationships helps the model better adapt to different shotcrete environments and conditions.
[0231] Enhanced robustness: By using multiple layers of nonlinear transformations to process data, the model can better cope with the impact of noise and abnormal data, thereby improving the robustness of the model.
[0232] Through this specific embodiment, the embodiment of the present invention can see how the complex shape, position, size and direction of surface defects are captured through the fourth layer of nonlinear transformation, how they interact with the responses in the acoustic wave detection signal data and the vibration signal data, and learn these complex relationships through the model, thereby improving the accuracy of the shotcrete layer quality assessment.
[0233] In another embodiment provided by the present invention, in order to describe the four nonlinear transformation steps in detail and explain how each layer captures the nonlinear relationships at different levels between different sensor data, a complete embodiment of a practical application is provided.
[0234] The purpose of the first layer of nonlinear transformation is to process the original weighted correlation scores to capture the basic nonlinear relationship.
[0235] Implementation: Use Sigmoid or ReLU functions to process weighted correlation scores, assuming that there is a basic correlation between surface defects in video image data and internal voids in acoustic detection signal data and vibration signal data. The first layer of nonlinear transformation processes these correlation scores through Sigmoid or ReLU functions to capture these basic relationships.
[0236] Beneficial effects: Through nonlinear transformation, the model can more accurately reflect the quality status of the shotcrete layer and capture the basic correlation between different sensor data.
[0237] The purpose of the second layer of nonlinear transformation is to further process the results of the first layer of nonlinear transformation to capture deeper nonlinear relationships.
[0238] Implementation: The result of the first layer of nonlinear transformation is further processed using the Softplus function or the Leaky ReLU function. In the second layer of nonlinear transformation, the embodiment of the present invention further captures the complex relationship between the surface defects in the video image data and the internal voids in the acoustic wave detection signal data and the vibration signal data. For example, how the degree of the surface defects affects the response of the acoustic wave detection signal data and the vibration signal data. These relationships may involve more complex patterns.
[0239] Beneficial effects: Through deeper nonlinear transformation, the model can more accurately reflect the quality status of the shotcrete layer and capture deeper nonlinear relationships between different sensor data.
[0240] The purpose of the third layer of nonlinear transformation is to further deepen the learning ability of the model and capture more complex nonlinear relationships.
[0241] Implementation: Use ELU function or GELU function to further deepen the learning ability. In the third layer of nonlinear transformation, the embodiment of the present invention captures the high-level interaction between surface defects in the video image data and internal voids in the acoustic wave detection signal data and the vibration signal data. For example, how the shape and position of the surface defects affect the response of the acoustic wave detection signal data and the vibration signal data. These relationships may involve higher-level patterns and complex pattern interactions.
[0242] Beneficial effects: By capturing more complex nonlinear relationships, the model can more accurately reflect the quality status of the shotcrete layer and capture very complex nonlinear relationships between different sensor data.
[0243] The purpose of the fourth layer of nonlinear transformation is to further deepen the learning ability of the model and capture the highest level of nonlinear relationships.
[0244] Implementation: Use Swish function or Sinusoidal function to capture the highest level of nonlinear relationships. In the fourth level of nonlinear transformation, the embodiment of the present invention captures the extremely complex interaction between surface defects in the video image data and internal voids in the acoustic wave detection signal data and the vibration signal data. For example, how the complex shape, position, size and direction of the surface defects interact with the responses in the acoustic wave detection signal data and the vibration signal data. These relationships may involve very high-level patterns and complex pattern interactions.
[0245] Beneficial effects: By capturing the highest level of nonlinear relationships, the model can more accurately reflect the quality status of the shotcrete layer and capture the highest level of nonlinear relationships between different sensor data.
[0246] Through the above four nonlinear transformation steps, the embodiment of the present invention can realize a more accurate, more generalized and robust intelligent detection method for tunnel shotcrete based on multimodal information fusion. Each layer of nonlinear transformation has its specific purpose and function, which work together to improve the overall performance of the model and can capture nonlinear relationships at different levels, thereby improving the accuracy of the shotcrete layer quality assessment. Through these steps, the embodiment of the present invention can not only capture various basic and complex relationships involved in the shotcrete layer quality assessment, but also improve the generalization ability and robustness of the model, so that it performs better in different shotcrete environments.
[0247] In another embodiment provided by the present invention, a tunnel shotcrete intelligent detection system based on multimodal information fusion is provided, such as Figure 2 As shown, including:
[0248] An acquisition module is used to acquire video image data, acoustic wave detection signal data and vibration signal data of the shotcrete layer in real time through at least three different types of sensors during the tunnel shotcrete process;
[0249] An extraction module, used to extract multiple data features from the video image data, the sound wave detection signal data and the vibration signal data, wherein the multiple data features have different modal types;
[0250] A learning module for mapping features of different modalities into a shared space through a feature fusion layer in a deep learning framework, where the feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features;
[0251] An introduction mechanism module is used to introduce a cross-modal attention mechanism in a shared space to learn the multiple data features and output correlation information between the data features;
[0252] A scoring module is used to determine a shotcrete quality score based on the correlation information and weights corresponding to a plurality of data features, so as to send an alarm message to a preset device port when the shotcrete quality score is lower than a threshold value.
[0253] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.
[0254] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.
[0255] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A tunnel shotcrete intelligent detection method based on multimodal information fusion, characterized in that: include: During the tunnel spraying process, at least three different types of sensors are used to collect video image data, acoustic wave detection signal data, and vibration signal data of the spraying layer in real time; Extracting a plurality of data features from the video image data, the sound wave detection signal data, and the vibration signal data, wherein the plurality of data features have different modal types; The features of different modalities are mapped into a shared space through the feature fusion layer in the deep learning framework, where the feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features; Introducing a cross-modal attention mechanism in the shared space to learn the multiple data features and outputting correlation information between the data features; Determine a shotcrete quality score based on the correlation information and weights corresponding to the plurality of data features, so that when the shotcrete quality score is lower than a threshold, an alarm message is sent to a preset device port; The step of determining the shotcrete quality score based on the correlation information and the weights corresponding to the multiple data features includes: The relevance information calculated using the cross-modal attention mechanism and the weights dynamically adjusted by the gating mechanism are used as input, where the relevance information represents the relevance matrix between data features, where each element represents the relevance score between the i-th feature and the j-th data feature; Calculate the sum of the weighted relevance scores of each feature, that is, for each feature i, multiply the weight of the i-th feature by the sum of the relevance scores of the feature and all other features, and then multiply by the quality score of the i-th feature. The final shotcrete quality score is solved based on the weighted relevance scores of all data features. The steps of solving the final shotcrete quality score according to the weighted correlation scores of all data features include: Performing a first-layer nonlinear transformation on the correlation scores of all data features to obtain a first-layer nonlinear information, where the first-layer nonlinear information includes basic information between different sensor data; Performing a second layer of nonlinear transformation based on the first layer of nonlinear information to obtain a second layer of nonlinear information, wherein the second layer of nonlinear information includes intrinsic correlation information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data; Performing a third layer of nonlinear transformation based on the second layer of nonlinear information to obtain a third layer of nonlinear information, wherein the third layer of nonlinear information includes interaction information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data; Calculating a shotcrete quality score based on the third layer of nonlinear information; Among them, it also includes: Extracting features of surface defects from video image data, and extracting features related to the internal structure of the shotcrete layer from the acoustic wave detection signal data and the vibration signal data; Calculate the correlation scores between features of different modalities; Use Swish function or Sinusoidal function to capture the highest level of nonlinear relationships to learn the complex shape, location, size and orientation of surface defects interacting with the responses in the acoustic inspection signal data and vibration signal data; The final shotcrete quality score is determined by averaging the sum of the weighted correlation scores after nonlinear processing of the data features.
2. The tunnel shotcrete intelligent detection method based on multimodal information fusion according to claim 1 is characterized in that: The features of different modalities are mapped into a shared space through the feature fusion layer in the deep learning framework. The feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features, including: Encode multiple data features using an encoder network in a deep learning framework, converting them into vector representations in a high-dimensional space; Introducing a gating mechanism in the feature fusion layer to dynamically adjust the weights corresponding to multiple data features; Use the decoder network in the deep learning framework to decode the fused features and map them back to the original data space to obtain the decoded features.
3. The tunnel shotcrete intelligent detection method based on multimodal information fusion according to claim 2 is characterized in that: The steps of introducing a gating mechanism in the feature fusion layer to dynamically adjust the weights corresponding to multiple data features include: Initialize a set of gating units for the feature data of each modality, and the gating units are used to control the contribution of different modal features; Update the weights of the gating units by calculating the correlation scores between features of different modalities; According to the weight of the gating unit, the features of different modes are weighted to strengthen the contribution of important features; The weighted features are fused to obtain the weights corresponding to each data feature.
4. The tunnel shotcrete intelligent detection method based on multimodal information fusion according to claim 3 is characterized in that: The step of updating the weight of the gating unit by calculating the correlation score between the features of different modalities includes: Initialize an initial weight for each gating unit; Calculating correlation scores between multiple data features in the video image data, the sound wave detection signal data, and the vibration signal data; The weights of the gating units are updated based on the calculated correlation scores.
5. An intelligent tunnel shotcrete detection system based on multimodal information fusion, characterized in that: include: An acquisition module is used to acquire video image data, acoustic wave detection signal data and vibration signal data of the shotcrete layer in real time through at least three different types of sensors during the tunnel shotcrete process; An extraction module, used to extract multiple data features from the video image data, the sound wave detection signal data and the vibration signal data, wherein the multiple data features have different modal types; A learning module for mapping features of different modalities into a shared space through a feature fusion layer in a deep learning framework, where the feature fusion layer uses a gating mechanism to dynamically adjust the weights corresponding to multiple data features; An introduction mechanism module is used to introduce a cross-modal attention mechanism in a shared space to learn the multiple data features and output correlation information between the data features; A scoring module, for determining a shotcrete quality score based on the correlation information and weights corresponding to a plurality of data features, so as to send an alarm message to a preset device port when the shotcrete quality score is lower than a threshold value; Wherein, the shotcrete quality score is determined based on the correlation information and weights corresponding to the plurality of data features, including: The relevance information calculated using the cross-modal attention mechanism and the weights dynamically adjusted by the gating mechanism are used as input, where the relevance information represents the relevance matrix between data features, where each element represents the relevance score between the i-th feature and the j-th data feature; Calculate the sum of the weighted relevance scores of each feature, that is, for each feature i, multiply the weight of the i-th feature by the sum of the relevance scores of the feature and all other features, and then multiply by the quality score of the i-th feature. The final shotcrete quality score is solved based on the weighted relevance scores of all data features. The final shotcrete quality score is solved based on the weighted correlation scores of all data features, including: Performing a first-layer nonlinear transformation on the correlation scores of all data features to obtain a first-layer nonlinear information, where the first-layer nonlinear information includes basic information between different sensor data; Performing a second layer of nonlinear transformation based on the first layer of nonlinear information to obtain a second layer of nonlinear information, wherein the second layer of nonlinear information includes intrinsic correlation information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data; Performing a third layer of nonlinear transformation based on the second layer of nonlinear information to obtain a third layer of nonlinear information, wherein the third layer of nonlinear information includes interaction information between surface defects captured by the video image data and internal cavities captured by the acoustic wave detection signal data and the vibration signal data; Calculating a shotcrete quality score based on the third layer of nonlinear information; Among them, it also includes: Extracting features of surface defects from video image data, and extracting features related to the internal structure of the shotcrete layer from the acoustic wave detection signal data and the vibration signal data; Calculate the correlation scores between features of different modalities; Use Swish function or Sinusoidal function to capture the highest level of nonlinear relationships to learn the complex shape, location, size and orientation of surface defects interacting with the responses in the acoustic inspection signal data and vibration signal data; The final shotcrete quality score is determined by averaging the sum of the weighted correlation scores after nonlinear processing of the data features.
6. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the tunnel shotcrete intelligent detection method based on multimodal information fusion as described in any one of claims 1 to 4.
7. A readable storage medium, characterized in that: A computer program is stored in the readable storage medium, and the computer program includes a program code for controlling a process to execute a process, and the process includes the tunnel shotcrete intelligent detection method based on multimodal information fusion according to any one of claims 1 to 4.
