Vision-guided concrete defect disease detection method, system and device

Abstract

The application discloses a kind of concrete defect disease detection method, system and device based on visual guidance, it is related to civil engineering concrete nondestructive testing and artificial intelligence technical field.The method includes the following steps: obtaining the visual image and time domain vibration velocity signal of concrete surface to be measured, wherein the apparent visual disease identification is carried out to visual image using pre-trained deep learning semantic segmentation model;According to the path planning of the measuring point of surface to be measured according to the saliency score, the time domain vibration velocity signal is collected according to the grid point in laser scanning path;Continuous wavelet transform is carried out to time domain vibration velocity signal to obtain two-dimensional time-frequency spectrum;Two-dimensional time-frequency spectrum and local apparent disease image are discriminated using double-flow multimodal fusion neural network.The application solves the problem of low efficiency of traditional full-field uniform scanning, high false alarm rate of single index discrimination, significantly improves the detection efficiency and intelligent level.

Description

Technical Field

[0001] This invention relates to the field of non-destructive testing and artificial intelligence technology for concrete in civil engineering, and particularly to a method, system and device for detecting concrete defects and malfunctions based on vision guidance. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of infrastructure construction, a large number of tunnels, bridges, and high-rise buildings are entering their maintenance period. Concrete structures often suffer from surface defects such as cracks, spalling, and water leakage due to construction quality or environmental erosion, as well as internal defects such as voids, debonding, and lack of density, seriously threatening structural safety. However, existing technologies for detecting defects in concrete structures mainly face the following problems: (1) Low detection efficiency. Current detection methods mainly rely on manual judgment, which is slow and highly subjective. Non-destructive testing technologies (grounding radar, infrared thermal imaging, ultrasound, etc.) face many bottlenecks, such as small range and inability to detect internal defects. Laser Doppler vibration measurement technology is gradually becoming a research hotspot due to its advantages such as long-distance non-destructive testing and the ability to detect internal structural defects. Existing laser vibration measurement methods usually use uniform full-field scanning with fixed intervals. In order to prevent missing small defects, extremely high scanning density must be set, which results in extremely long detection time for large-area structures (such as tunnel linings and high-rise exterior walls). However, if the density is reduced, it is very easy to miss key defects.

[0004] (2) Insufficient detection accuracy. Existing methods rely solely on the single data feature of laser vibration signals for detection, which is easily affected by the surface roughness of concrete, the difference in aggregate reflectivity, and environmental noise, leading to the misjudgment of normal surface noise as internal defects, resulting in misjudgment.

[0005] (3) The test results cannot be quantified. Existing frequency domain analysis methods mostly output qualitative thermal maps or vibration energy maps, which are difficult to directly quantify the specific burial depth, size and type of defects and diseases. They often require secondary interpretation by experts, which cannot meet the automation and digitalization requirements of refined engineering management. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a visually guided method, system and device for detecting concrete defects and diseases, which solves the problems of low efficiency of traditional full-field uniform scanning and high false alarm rate of single index discrimination, and significantly improves detection efficiency and intelligence level.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution: The first aspect of this invention provides a vision-guided method for detecting concrete defects, comprising the following steps: Visual images and temporal vibration velocity signals of the concrete surface to be tested are acquired. Specifically, a pre-trained deep learning semantic segmentation model is used to identify visible defects in the visual images, obtaining the saliency scores of each pixel region of the surface to be tested and local visible defect images. Based on the saliency scores, path planning is performed on the test points of the surface to be tested to obtain a laser scanning path with varying density. Temporal vibration velocity signals are collected based on the grid points in the laser scanning path. A two-dimensional time-frequency spectrum is obtained by performing continuous wavelet transform on the time-domain vibration velocity signal; A two-stream multimodal fusion neural network is used to identify defects in two-dimensional time-frequency spectra and local apparent disease images.

[0008] Furthermore, after acquiring visual images of the concrete surface to be tested, the visual images are standardized.

[0009] Furthermore, the specific steps for identifying apparent visual defects in visual images using a pre-trained deep learning semantic segmentation model are as follows: Pre-trained deep learning semantic segmentation models are used to extract visual features from visual images to obtain local visual lesion images; The surface to be tested is divided into grid cells, and the significance score is calculated for the local apparent disease images based on the weighted sum of the disease confidence.

[0010] Furthermore, the specific steps for path planning of the test points on the surface to be tested based on the saliency score are as follows: Based on the saliency score, a greedy algorithm or simulated annealing algorithm is used to plan the variable density scanning path with the shortest jump distance of the measurement points, and a variable density laser scanning path is generated.

[0011] Furthermore, based on the saliency score, the surface to be tested is divided into a high-interest area, a buffer area, and a background area, and high-density, medium-density, and low-density scanning dots are assigned to them respectively. The surface to be tested is then traversed according to the planned scanning dots.

[0012] Furthermore, during the spatial mapping process based on the grid points in the laser scanning path and the temporal vibration velocity signal, the coordinates of the scanning grid points are converted into the deflection angle control command of the galvanometer through the rigid body transformation matrix of the pre-calibrated camera coordinate system and laser galvanometer coordinate system.

[0013] Furthermore, the specific steps for defect and disease identification using a two-dimensional time-frequency spectrum with a dual-stream multimodal fusion neural network are as follows: Construct a two-stream multimodal fusion neural network; The two-stream multimodal fusion neural network is trained based on a composite loss function; The trained dual-stream multimodal fusion neural network is used to identify defects and faults in the two-dimensional time-frequency spectrum, generating a cloud map of tunnel fault distribution.

[0014] Furthermore, the dual-stream multimodal fusion neural network includes a vibration perception branch, a visual perception branch, and a multimodal attention fusion module. The vibration perception branch and the visual perception branch are used to extract data features and texture features, respectively, and the data features and texture features are fused by the multimodal attention fusion module.

[0015] A second aspect of the present invention provides a vision-guided concrete defect detection system, comprising: The data acquisition module is configured to acquire visual images and temporal vibration velocity signals of the concrete surface to be tested. Specifically, a pre-trained deep learning semantic segmentation model is used to identify visible defects in the visual images, obtaining saliency scores for each pixel region of the surface to be tested and images of local visible defects. Based on the saliency scores, path planning is performed on the test points of the surface to be tested to obtain a variable density laser scanning path, and temporal vibration velocity signals are collected based on the grid points in the laser scanning path. The data processing module is configured to perform continuous wavelet transform on the time-domain vibration velocity signal to obtain a two-dimensional time-frequency spectrum; The defect and lesion identification module is configured to use a two-stream multimodal fusion neural network to identify defects and lesions from two-dimensional time-frequency spectra and local apparent lesion images.

[0016] A third aspect of the present invention provides a detection device for the vision-guided concrete defect detection method described in the first aspect, comprising: A multimodal sensing unit is used for multimodal data acquisition and coordinate calibration. The vibration excitation unit is used to transmit excitation energy of a specific frequency band to the concrete structure under test, thereby exciting the structure to produce weak vibrations. The control and processing unit is connected to the vibration excitation unit and the multimodal sensing unit, respectively, and is used for data calculation and control execution operations.

[0017] The above one or more technical solutions have the following beneficial effects: This invention discloses a vision-guided method, system, and device for detecting concrete defects and blemishes. It acquires high-resolution images and spatial distance information of the surface to be tested, extracts apparent blemish areas using a semantic segmentation model, and calculates saliency scores to generate a variable-density adaptive scanning path. Subsequently, highly saliency areas are scanned more intensively to collect vibration response signals. Next, continuous wavelet transform is used to convert the one-dimensional time-domain vibration signals into a two-dimensional time-frequency spectrum. Finally, a dual-stream deep neural network is constructed to fuse local apparent blemish features with internal vibration time-frequency features, enabling the identification and dimensional quantification of both internal and apparent defects in concrete. This invention solves the problems of low efficiency and high false alarm rate associated with traditional full-field uniform scanning and single-index discrimination, significantly improving detection efficiency and intelligence.

[0018] This invention abandons the blind uniform scanning method and uses computer vision technology to quickly identify areas of apparent defects in concrete. In typical tunnel inspection scenarios, it can reduce a large number of invalid measuring points compared with traditional methods, significantly improving the inspection speed.

[0019] This invention, by fusing data features from both visual and vibrational modes, fully leverages the complementary advantages of different physical mechanisms. Specifically, visual features can identify surface disturbances, while acoustic and vibrational features can reveal internal structural responses; the deep fusion of these two features enables the system to effectively eliminate false internal defect signals caused by surface roughness or environmental noise.

[0020] This invention introduces time-frequency analysis and deep learning regression technology, breaking through the limitations of traditional methods that can only look at "waveforms" or "heat maps". It can directly output the type, depth and size of defects in digital form, providing accurate data support for the repair and reinforcement of engineering structures.

[0021] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the vision-guided concrete defect detection method in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the adaptive scanning path generation guided by visual saliency in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the structure of the dual-stream multimodal fusion neural network in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of the architecture of the vision-guided concrete defect detection device in Embodiment 3 of the present invention. Detailed Implementation

[0024] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0025] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof. The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0026] Example 1: Embodiment 1 of the present invention provides a visually guided method for detecting concrete defects, as shown in the figure, including the following steps: S1: Acquire visual images and time-domain vibration velocity signals of the concrete surface to be tested.

[0027] S1.1: Obtain a visual image of the concrete surface to be tested and perform standardization operations on the visual image.

[0028] S1.1.1: Obtain visual images and spatial distance information of the concrete surface to be tested, and construct a three-dimensional geometric model of the surface to be tested.

[0029] In one specific implementation, this embodiment uses an industrial camera and a laser rangefinder to collect image data and spatial distance information of the concrete surface to be measured (the surface to be measured), and constructs a three-dimensional geometric model of the surface to be measured.

[0030] Specifically, controlling industrial cameras to acquire high-resolution RGB images The original resolution is 5472×3648.

[0031] Using a laser Doppler vibration meter (LDV) at a sampling rate Collect time-domain vibration velocity signals at each measuring point Sampling time That is, each sample contains One sampling point.

[0032] S1.1.2: Preprocess the visual image.

[0033] In one specific implementation, the visual image is standardized. To eliminate environmental noise and unify data distribution, this embodiment performs standardization operations such as resizing and normalization on the visual image. The specific steps are as follows: S1.1.2.1: Size Adjustment. The original image is resized using bilinear interpolation. Downsampling to model input size .

[0034] in, It represents the height of an image or feature map (the number of pixels in the vertical direction). Indicates the width of the image or feature map (the number of pixels in the horizontal direction).

[0035] S1.1.2.2: Normalization. Maps pixel values ​​from [0, 255] to the interval [-1, 1], using the following formula: .

[0036] Where x is the original pixel value. This is the normalized value.

[0037] S1.2: Acquire the time-domain vibration velocity signal.

[0038] S1.2.1: Use a pre-trained deep learning semantic segmentation model to identify visible defects in visual images, and obtain the saliency scores of each pixel region of the surface under test and the local visible defect images.

[0039] In one specific implementation, this embodiment performs semantic analysis on the visual image to extract structural surface defects, and calculates the salience score of each region based on the defect type and confidence level. Specifically, a pre-trained deep learning semantic segmentation model is used to process the surface image to identify visible surface defects such as cracks, seepage, honeycombing, exposed reinforcement, and peeling, and the salience score of each pixel region is calculated based on the defect type and confidence level. The specific steps are as follows: S1.2.1.1: Extract the appearance features of visual images using a pre-trained deep learning semantic segmentation model.

[0040] This embodiment uses the DeepLabV3+ architecture to extract surface features. The backbone network of the deep learning semantic segmentation model uses ResNet-101, with an output stride of 16. The model output is a local surface disease image of H×W×K, where K=4 (background, cracks, seepage, peeling) represents the disease type.

[0041] S1.2.1.2: Divide the surface to be tested into grid cells and calculate the significance score based on the weighted sum of the disease confidence levels.

[0042] Define a region Significance score : .

[0043] in, The confidence probability for each disease type. These are the weighting coefficients.

[0044] Confidence score refers to the predicted probability value of a deep neural network model for the category to which each pixel in an image belongs. Specifically, the output layer of the semantic segmentation network uses the Softmax function to output the probability that each pixel x belongs to the i-th disease category. This probability value ranges from 0 to 1. The closer the value is to 1, the higher the certainty that the model determines that this type of disease exists at that location. Incorporating confidence into the significance score calculation not only captures obvious typical diseases, but also allows the system to allocate appropriate scanning density for secondary confirmation of suspected areas (low-confidence areas) with unclear features and ambiguous model judgments. This maximizes detection efficiency while avoiding missed detections.

[0045] In this embodiment, the area to be tested is divided into M×N grid cells. For the first... Each grid cell has a significance score. Defined as a weighted sum of confidence levels for multiple diseases: .

[0046] in: Indicates the first The set of all pixels within a grid; This represents the predicted probability that pixel p belongs to the k-th type of disease; This is the risk weighting coefficient. Considering that water leakage often indicates penetrating cracks or cavities, followed by fissures, and then spalling, a risk weighting coefficient for water leakage is set. Crack risk weighting coefficient Peeling risk weighting coefficient .

[0047] S1.2.2: Based on the significance score, the path planning of the measurement points on the surface to be measured is performed to obtain the laser scanning path with variable density.

[0048] In one specific implementation, based on the saliency score, a greedy algorithm or simulated annealing algorithm is used to plan the variable-density scanning path with the shortest jump distance between measurement points, generating a variable-density laser scanning path. In this embodiment, the surface to be measured is divided into a high-interest area, a buffer area, and a background area according to the saliency score, and high-density, medium-density, and low-density scanning point arrays are assigned to them respectively, and the surface to be measured is traversed according to the planned scanning point array.

[0049] Specifically, set a high significance threshold. and low threshold ,like Defined as a high-interest area, the scanning interval is set to... Generate a high-density scanning mesh to ensure that no minute defects are missed; if Defined as a buffer, with the scan interval set to... Generate a medium-density scan grid; if Defined as the background area, with the scan interval set to [value]. This generates a low-density scanning grid, used only for detecting deep, large-area voids; thus, while ensuring detection accuracy in key areas, it significantly reduces the total number of scanning points. .

[0050] Then, a variable density path is generated based on the adaptive scanning grid.

[0051] Specifically, based on significance scores Automatically generate non-uniform sampling point matrix: like (Set to 0.7), generate spacing High-density lattice; like Generate spacing Medium-density lattice; like (Set to 0.3), only one measuring point is retained at the center of the grid.

[0052] Finally, a greedy algorithm is used to connect all measurement points, maximizing the total travel distance of the measurement points. Minimize to obtain a variable-density laser scanning path; the process of generating the variable-density path is as follows: Figure 2 As shown. In this embodiment, the measuring points are the grid points to be detected.

[0053] S1.2.3: Collect time-domain vibration velocity signals based on grid points in the laser scanning path.

[0054] In one specific implementation, the coordinates of the generated scanning grid points are converted into deflection angle control commands for the galvanometers by using a pre-calibrated rigid body transformation matrix between the camera coordinate system and the laser galvanometer coordinate system. The deflection angle control commands, based on the angle and time of the rotation response of multiple galvanometers, allow the laser detection points to be moved quickly without moving the entire device to complete the scan.

[0055] In the tunnel scenario, this embodiment uses deflection angle control commands to coordinate multiple mirror rotation angles, rotation times, and other aspects, enabling rapid scanning of the entire tunnel within a small area (the entire transverse section of the tunnel and 5-8m in the longitudinal direction) without moving equipment. Then, the entire equipment is moved to scan the next area, and so on, to complete the detection of the entire tunnel.

[0056] Vibration is generated by applying sound waves or pulses, and the time-domain vibration signal of the concrete surface is collected point by point according to the adaptive scanning grid by using coordinate mapping relationship to control the laser Doppler vibration meter and the laser scanning device.

[0057] S2: Perform continuous wavelet transform on the time-domain vibration velocity signal to obtain a two-dimensional time-frequency spectrum.

[0058] In one specific implementation, the time-domain vibration signal is then subjected to time-frequency transformation, and the acquired one-dimensional time-domain vibration signal is subjected to continuous wavelet transformation. Morlet and Bump wavelets are selected as basis functions to generate a two-dimensional time-frequency spectrum containing time-frequency-energy information.

[0059] Specifically, a time-frequency transformation operation is performed on the time-domain vibration velocity signal. In this embodiment, to convert the one-dimensional vibration signal into two-dimensional features that are easily processed by neural networks, a continuous wavelet transform (CWT) is used. The complex Morlet wavelet (Cmor3-3) is chosen as the basis function because it has good localization properties in both the time and frequency domains. The transformation formula is as follows: .

[0060] in, Here are the wavelet transform coefficients, and t represents time. The original vibration signal is represented by 'a', where 'a' is the scale factor (corresponding to frequency) and 'b' is the translation factor (corresponding to time). It is the complex conjugate of the mother wavelet.

[0061] The generated time-frequency spectrum was adjusted to a 224×224×3 RGB image format and used as input for the vibration branch. This time-frequency spectrum contains time, frequency, and energy information.

[0062] The specific steps are as follows: In this embodiment, the coordinates of the scanning grid points are converted into deflection angle control commands for the galvanometer by using a pre-calibrated rigid body transformation matrix between the camera coordinate system and the laser galvanometer coordinate system.

[0063] Specifically, firstly, a calibration target is established, and then the industrial camera coordinate system is solved ( , , (to the laser galvanometer coordinate system) , , rigid body transformation matrix (Including rotation matrix R and translation vector t); then, defective pixels are identified using the camera. The spatial distance Z of the point is obtained using a laser rangefinder; then, the spatial point in the camera coordinate system is obtained by back-projection using a pinhole imaging model. Then, using the transformation matrix Calculate the point in the laser coordinate system Finally, Converted to the deflection angle of the scanning device (n is the number of galvanometers), directly controlling the laser to strike the surface of the object corresponding to the pixel.

[0064] It also includes visual cropping operations, which extract local surface image slices from the original high-resolution image based on the coordinates of the measurement points. Specifically, it locates and extracts defects from the image, and then standardizes and formats the extracted defect images to meet the requirements of the input neural network.

[0065] More specifically, the process begins with image acquisition and preprocessing, including distortion correction and illumination equalization. Then, deep learning-based semantic segmentation is used for disease identification and judgment, including sliding window segmentation (slicing a high-resolution image into multiple low-resolution images), input model training (such as DeepLabV3+ or U-Net networks), and full image stitching. The sliding window segmentation involves cutting a high-resolution image into multiple low-resolution images, and the input model is a network such as DeepLabV3+ or U-Net.

[0066] In the vibration sensing branch, visual cropping of the defect image is used to initially determine the location of the apparent defect. High- and low-density scanning grid paths are generated by matching the image pixel coordinates with the spatial coordinates of the laser ranging system, and then the actual equipment inspection is carried out. In the visual sensing branch, the defect image is visually cropped to obtain image slices of local apparent defects, and the slice data is input into the visual sensing branch. Finally, a two-stream neural network is used for fusion analysis and prediction.

[0067] S3: Defect and disease identification is performed on two-dimensional time-frequency spectra and local apparent disease images using a two-stream multimodal fusion neural network.

[0068] In one specific implementation, a two-dimensional time-frequency spectrum and a local apparent defect image are input into a pre-trained two-stream multimodal fusion neural network, which outputs the type probability and size estimate of the defect. Specifically, this embodiment constructs and utilizes a two-stream multimodal fusion neural network to identify defects. First, a convolutional neural network is used to extract data features from the time-frequency spectrum to capture the frequency shift and energy accumulation caused by defects. Then, a convolutional neural network is used to extract texture features from the local apparent defect image to capture surface cracks or roughness information. Finally, a channel attention mechanism is introduced to adaptively weight the feature vectors of the two branches. In particular, when there are no cracks on the surface but abnormal vibration, the network automatically assigns higher weights to the acoustic vibration features, thereby identifying it as a hidden cavity. The network's output layer learns through multiple tasks, simultaneously outputting classification and regression results. The classification result includes the specific type of defect, such as cavity, debonding, non-compact, crack, or intact; the regression result includes the depth value of the defect from the surface and the equivalent size of the defect.

[0069] S3.1: Construct a dual-stream multimodal fusion neural network.

[0070] The dual-stream multimodal fusion neural network includes a vibration sensing branch, a visual sensing branch, and a multimodal attention fusion module. The vibration sensing branch and the visual sensing branch are used to extract data features and texture features, respectively. The vibration feature extraction branch uses a convolutional neural network to process the time-frequency spectrum and extract the time-frequency domain features of the vibration data. The visual feature extraction branch uses a convolutional neural network to process the local visual image and extract apparent disease features. The data features and texture features are then fused through the multimodal attention fusion module. The multimodal attention fusion module introduces a channel attention mechanism or a spatial attention mechanism to weightedly fuse the feature vectors of the two branches. The specific structure diagram is shown below. Figure 3 As shown.

[0071] 1. Vibration sensing branch Input: A time-frequency spectrum of 224×224×3.

[0072] Backbone network: adopts ResNet-18 structure.

[0073] First layer: 7×7 convolution, stride 2, padding 3, followed by BatchNorm and ReLU.

[0074] Residual blocks: Contain 4 stages with channel numbers of 64, 128, 256, and 512 respectively. Each residual block consists of two 3×3 convolutional layers and skip connections.

[0075] Output: After global average pooling, the output is a 1×512 dimensional feature vector. .

[0076] 2. Visual perception branch Input: A 64×64×3 local surface image slice (centered on the laser measurement point).

[0077] Backbone network: The MobileNetV3-Small structure is adopted, and depthwise separable convolutions are used to reduce the number of parameters.

[0078] Output: After GAP processing, the output is a 1×512 dimensional feature vector. .

[0079] 3. Multimodal attention fusion module Introduce a channel attention mechanism (SE-Block) for feature weighting: Feature stitching: Features after stitching The dimension is 1×1024.

[0080] Weighted learning: .

[0081] in, It is a dimension-reduced fully connected layer (dimensionality reduction ratio r=16). It is the ReLU activation function. To upgrade the fully connected layer, This is the Sigmoid function.

[0082] Feature recalibration: Features after calibration That is, the fused features are weighted channel by channel by the learned weights W.

[0083] S3.2: Training the two-stream multimodal fusion neural network based on a composite loss function.

[0084] S3.2.1: Obtain the known dataset and divide the dataset.

[0085] In one specific implementation, a triplet dataset containing "apparent defect image - vibration time-frequency diagram - defect label" is constructed. The data is mainly generated by simulation using COMSOL and ABAQUS software, and field-measured data is used to optimize the model parameters. The dataset is randomly divided into a 70% training set, a 15% validation set, and a 15% test set. The training set is used to optimize model parameters, the validation set is used to select the best model checkpoints, and the test set is used to evaluate the final metrics.

[0086] S3.2.2: Set the composite loss function.

[0087] In one specific implementation, the training objective of the two-stream multimodal fusion neural network is to simultaneously minimize both classification and regression errors. The total loss function... Defined as: .

[0088] (1) Classification loss Used for defect type identification (normal, voids, cracks, loose). Employs the Cross Entropy Loss function. .

[0089] Where n is the batch size and C is the number of categories. For true labels (One-hot encoding) Predict probabilities for the network.

[0090] (2) Deep regression loss Used to predict defect depth. A Smooth L1 loss function is employed to enhance robustness to outliers. .

[0091] Where y is the true depth (obtained through core sampling or CT calibration). For depth prediction, hyperparameters Set it to 1.0.

[0092] (3) Regularization term L2 regularization is applied to the network weights to prevent overfitting. .

[0093] in, Indicates the weight.

[0094] In this embodiment, the weighting coefficient is set as follows: , , .

[0095] S5.2.3: Training based on a composite loss function.

[0096] Hardware environment: A single NVIDIA RTX 3090 GPU (24GB VRAM) is used, the CPU is an Intel i9-10900K, and the deep learning framework is PyTorch 1.10.

[0097] Optimizer settings: Select AdamW optimizer. Initial learning rate set to 1× Momentum parameters , .

[0098] Training strategy: Epochs: The total number of training epochs is set to 200.

[0099] Batch Size: Set to 64.

[0100] Learning rate decay: A cosine annealing strategy is used, resetting the learning rate every 50 epochs, with the minimum learning rate decreasing to 1× .

[0101] Early stopping mechanism: If the validation set loss does not decrease within 20 consecutive epochs, training is terminated early and the current best model is saved.

[0102] S3.3: Use the trained dual-stream multimodal fusion neural network to identify defects and faults in the two-dimensional time-frequency spectrum and generate a cloud map of tunnel fault distribution.

[0103] In one specific implementation, after training, the vibration time-frequency diagram and local apparent defect image of the test point are input into the network. If the output defect probability... If the value is found to be internally damaged, the corresponding predicted depth value is output, and finally a digital cloud map of tunnel damage distribution is generated.

[0104] Example 2: Embodiment 2 of the present invention provides a vision-guided concrete defect detection system, comprising: The data acquisition module is configured to acquire visual images and temporal vibration velocity signals of the concrete surface to be tested. Specifically, a pre-trained deep learning semantic segmentation model is used to identify visible defects in the visual images, obtaining saliency scores for each pixel region of the surface to be tested and images of local visible defects. Based on the saliency scores, path planning is performed on the test points of the surface to be tested to obtain a variable density laser scanning path, and temporal vibration velocity signals are collected based on the grid points in the laser scanning path. The data processing module is configured to perform continuous wavelet transform on the time-domain vibration velocity signal to obtain a two-dimensional time-frequency spectrum; The defect and lesion identification module is configured to use a two-stream multimodal fusion neural network to identify defects and lesions from two-dimensional time-frequency spectra and local apparent lesion images.

[0105] Example 3: Embodiment 3 of the present invention provides a detection device for the vision-guided concrete defect detection method described in Embodiment 1, such as... Figure 4 As shown, it includes: Multimodal sensing unit, used for multimodal data acquisition and coordinate calibration.

[0106] The multimodal sensing unit comprises a visual sensing module, a vibration sensing module, and a coordinate calibration module. The visual sensing module includes an industrial camera and a laser rangefinder, used to acquire high-resolution images and spatial distance information of the concrete structure surface under test. The vibration sensing module includes a laser Doppler vibrometer and a laser scanning device, used for non-contact measurement of vibration signals on the surface under test. The coordinate calibration module is used to establish a spatial coordinate mapping relationship between the visual sensing unit and the vibration sensing unit.

[0107] Specifically, the industrial camera and laser vibrometer are mounted on the same rigid pan-tilt unit; the laser scanning device is either a two-dimensional galvanometer scanning system or a three-dimensional dynamic focusing scanning system, used to control the laser beam to scan the surface to be measured. Considering the relatively flat concrete surface or the large depth of focus of the laser Doppler vibrometer, a two-dimensional galvanometer is preferred to reduce costs. If the surface to be measured has large undulations (such as deep foundation pits or irregular tunnels), the laser scanning device can use a three-dimensional dynamic focusing galvanometer, which, by adjusting the focal length in real time, ensures that the laser focus always falls on the surface to be measured, thereby obtaining the best signal-to-noise ratio.

[0108] The coordinate calibration module is configured to perform the following operations: first, establish a calibration target, and then solve for the industrial camera coordinate system. , , (to the laser galvanometer coordinate system) , , rigid body transformation matrix (Including rotation matrix R and translation vector t); then, defective pixels are identified using the camera. The spatial distance Z of the point is obtained using a laser rangefinder; then, the spatial point in the camera coordinate system is obtained by back-projection using a pinhole imaging model. Then, using the transformation matrix Calculate the point in the laser coordinate system Finally, Converted to the deflection angle of the scanning device (n is the number of galvanometers), directly controlling the laser to strike the surface of the object corresponding to the pixel.

[0109] The vibration excitation unit, including an acoustic wave emitting device or a pulsed laser, is used to emit excitation energy of a specific frequency band to the concrete structure under test, thereby exciting the structure to produce weak vibrations.

[0110] Specifically, the vibration excitation unit can be selected from either a sound wave transmitter or a pulsed laser, depending on the application scenario. When a sound wave transmitter is used, it emits air-coupled sound waves with a specific frequency band, using sound pressure to excite structural resonance; when a pulsed laser is used, it emits high-energy short-pulse lasers, using the thermoelastic effect to excite broadband ultrasonic waves on the concrete surface, achieving all-optical non-contact excitation.

[0111] The control and processing unit is connected to the vibration excitation unit and the multimodal sensing unit, respectively, and is used for data calculation and control execution operations.

[0112] Specifically, the control and processing unit is used to perform adaptive scanning path planning, multimodal data synchronous acquisition, and intelligent diagnosis of defects. First, a deep learning semantic segmentation model is used to identify defect features in the visual image and calculate the region saliency score. Then, based on the saliency score, the area to be tested is divided into a high-interest area, a buffer area, and a background area, and high-density, medium-density, and low-density laser Doppler vibrometer scanning point arrays are assigned to each area. Finally, the laser scanning device is controlled to traverse the surface to be tested according to the planned scanning point array.

[0113] The steps and methods involved in Examples 2 and 3 above correspond to those in Example 1. For specific implementation details, please refer to the relevant description section of Example 1.

[0114] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, an optical medium, or a semiconductor medium, etc.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A vision-guided method for detecting concrete defects, characterized in that, Includes the following steps: Visual images and temporal vibration velocity signals of the concrete surface to be tested are acquired. Specifically, a pre-trained deep learning semantic segmentation model is used to identify visible defects in the visual images, obtaining saliency scores for each pixel region of the surface and images of local defects. Based on the saliency scores, path planning is performed on the test points of the surface to obtain a variable-density laser scanning path. Temporal vibration velocity signals are collected based on the grid points in the laser scanning path. The surface to be tested is divided into a high-interest area, a buffer area, and a background area based on the saliency scores, and high-density, medium-density, and low-density scanning point arrays are assigned to these areas respectively. The surface is then traversed according to the planned scanning point array. A two-dimensional time-frequency spectrum is obtained by performing continuous wavelet transform on the time-domain vibration velocity signal; A two-stream multimodal fusion neural network is used to identify defects in two-dimensional time-frequency spectra and local apparent disease images.

2. The visual-guided concrete defect detection method as described in claim 1, characterized in that, After acquiring the visual image of the concrete surface to be tested, the visual image is standardized.

3. The visual-guided concrete defect detection method as described in claim 1, characterized in that, The specific steps for identifying visual defects in visual images using a pre-trained deep learning semantic segmentation model are as follows: Pre-trained deep learning semantic segmentation models are used to extract visual features from visual images to obtain local visual lesion images; The surface to be tested is divided into grid cells, and the significance score is calculated for the local apparent disease images based on the weighted sum of the disease confidence.

4. The vision-guided concrete defect detection method as described in claim 1, characterized in that, The specific steps for path planning of the test surface and test points based on the significance score are as follows: Based on the saliency score, a greedy algorithm or simulated annealing algorithm is used to plan the variable density scanning path with the shortest jump distance of the measurement points, and a variable density laser scanning path is generated.

5. The vision-guided concrete defect detection method as described in claim 1, characterized in that, During the spatial mapping process based on the grid points in the laser scanning path and the temporal vibration velocity signal, the coordinates of the scanning grid points are converted into the deflection angle control command of the galvanometer through the rigid body transformation matrix of the pre-calibrated camera coordinate system and laser galvanometer coordinate system.

6. The vision-guided concrete defect detection method as described in claim 1, characterized in that, The specific steps for defect identification in two-dimensional time-frequency spectra using a two-stream multimodal fusion neural network are as follows: Construct a two-stream multimodal fusion neural network; The two-stream multimodal fusion neural network is trained based on a composite loss function; The trained dual-stream multimodal fusion neural network is used to identify defects and faults in the two-dimensional time-frequency spectrum, generating a cloud map of tunnel fault distribution.

7. The visual-guided concrete defect detection method as described in claim 1, characterized in that, The dual-stream multimodal fusion neural network includes a vibration sensing branch, a visual sensing branch, and a multimodal attention fusion module. The vibration sensing branch and the visual sensing branch are used to extract data features and texture features, respectively, and the data features and texture features are fused by the multimodal attention fusion module.

8. A vision-guided concrete defect detection system, characterized in that, include: The data acquisition module is configured to acquire visual images and temporal vibration velocity signals of the concrete surface to be tested. Specifically, a pre-trained deep learning semantic segmentation model is used to identify visible defects in the visual images, obtaining saliency scores for each pixel region of the surface to be tested and images of local visible defects. Based on the saliency scores, path planning is performed on the test points of the surface to be tested to obtain a laser scanning path with varying density. Temporal vibration velocity signals are collected based on the grid points in the laser scanning path. The surface to be tested is divided into a high-interest area, a buffer area, and a background area based on the saliency scores, and high-density, medium-density, and low-density scanning point arrays are assigned to them respectively. The surface to be tested is traversed according to the planned scanning point array. The data processing module is configured to perform continuous wavelet transform on the time-domain vibration velocity signal to obtain a two-dimensional time-frequency spectrum; The defect and lesion identification module is configured to use a two-stream multimodal fusion neural network to identify defects and lesions from two-dimensional time-frequency spectra and local apparent lesion images.

9. A detection device for use in the vision-guided concrete defect detection method according to any one of claims 1-7, characterized in that, include: A multimodal sensing unit is used for multimodal data acquisition and coordinate calibration. The vibration excitation unit is used to transmit excitation energy of a specific frequency band to the concrete structure under test, thereby exciting the structure to produce weak vibrations. The control and processing unit is connected to the vibration excitation unit and the multimodal sensing unit, respectively, and is used for data calculation and control execution operations.

Images