A multi-modal detection system for damage in a rock-like material

Abstract

The present application belongs to the technical field of nondestructive testing of engineering materials, and specifically discloses a multi-modal detection system for damage of rock-like materials, wherein an image and ultrasonic detection module is fixed to the side wall of a test cabin in the form of an interleaved array, and microsecond-level time synchronization acquisition is realized through a synchronization module; an image splicing module is used for self-correction of image distortion and cylindrical panoramic splicing; an ultrasonic imaging module is used for reconstruction of a three-dimensional image of internal defects; and a defect identification and segmentation quantification module is used for identification of defects and quantification of internal defects from the three-dimensional image. The present application solves the problems of the prior art, such as asynchronization of multi-source data in time and space, difficulty in registration, low imaging quality of internal defects, and dependence on manual evaluation, and realizes automatic, quantitative and visual detection and evaluation of the whole process of damage of rock-like test pieces from the surface to the interior and from static to dynamic.

Description

Technical Field

[0001] This invention belongs to the field of nondestructive testing technology for engineering materials, and specifically relates to a multimodal detection system for damage to rock-like materials. Background Technology

[0002] Rock-like materials and concrete, widely used in engineering structures, are subjected to multiple environmental-mechanical coupling effects during service, including long-term loads, temperature gradients, humidity changes, and chemical erosion. These factors make them highly susceptible to the initiation of microcracks within the structure, which gradually expand into through-hole or blind-end defects, ultimately leading to degradation of load-bearing capacity or even sudden failure. Accurately and comprehensively acquiring three-dimensional evolution information of defects from their appearance to their internal structure is a crucial prerequisite for assessing material damage status and predicting structural lifespan.

[0003] Currently, the detection of defects in rock-like specimens mainly relies on two technical approaches: Optical vision methods (such as digital image correlation (DIC), high-definition photography, laser scanning, etc.) can capture the geometric shape (length, width, direction) of surface defects with high precision, but they are limited to the outer surface of the specimen and cannot reveal the depth of the defect's extension, branch structure, or hidden defects inside. Ultrasonic nondestructive testing methods (such as pulse echo, transmission method, phased array, etc.) have penetrating power and can detect internal pores, debonding and cracks, but they face serious challenges when applied to curved cylindrical specimens: on the one hand, it is difficult to couple the probe with the curved surface, which easily introduces acoustic energy loss and waveform distortion; on the other hand, traditional ultrasonic imaging has limited resolution, making it difficult to achieve fine characterization of defects, and the reconstruction results lack a spatial coordinate reference shared with optical measurement results.

[0004] However, existing technologies generally employ single-modal detection or loosely coupled data post-processing overlay methods, resulting in inaccurate spatial alignment and temporal synchronization between optical and ultrasonic data. This severely restricts a deeper understanding of the three-dimensional penetration behavior of defects and the internal and external collaborative evolution mechanisms. Furthermore, in mechanical loading tests on high-strength concrete or rock-like specimens, the specimens often experience sudden fractures or fragmentation due to their brittle characteristics, threatening operator safety and leading to the loss of data at critical moments of failure. Summary of the Invention

[0005] The purpose of this invention is to overcome the deficiencies in the existing technology and provide a multimodal detection system for damage to rock-like materials.

[0006] This invention provides a multimodal damage detection system for rock-like materials, comprising: The test chamber has an internal test cavity for fixing rock-like specimens; The image detection module includes multiple miniature industrial cameras, which are fixedly installed in an array around the periphery of the test chamber to acquire images of the side surface of the rock-like specimen. The ultrasonic testing module includes multiple ultrasonic transducers, which are fixedly installed in an array around the periphery of the test chamber and staggered with the miniature industrial camera in the circumferential direction, for acquiring ultrasonic signals that penetrate or are reflected by the rock-like specimen. The loading module includes a loading force transmission block disposed in the upper part of the inner cavity of the test chamber, which is used to apply axial load to the rock-like specimen; The synchronization module is connected to the image detection module, ultrasonic detection module and loading module respectively. Based on a unified hardware clock, it outputs a synchronization trigger signal to the miniature industrial camera, ultrasonic transducer and loading force block to synchronize the acquisition time of image data, ultrasonic data and mechanical data. An image stitching module, connected to the image detection module, is used to process and stitch together images acquired by multiple miniature industrial cameras to generate a panoramic view of the cylindrical surface unfolding of the side surface of a rock-like specimen. An ultrasonic imaging module, connected to the ultrasonic detection module, is used to process ultrasonic signals acquired by multiple ultrasonic transducers to reconstruct a three-dimensional image of internal defects in a rock-like specimen. The defect identification and segmentation quantization module is connected to the image stitching module and the ultrasonic imaging module, respectively, and is used to identify and quantify surface defects from the panoramic image unfolded from the cylindrical surface, and to identify and quantify internal defects from the three-dimensional image.

[0007] A further embodiment is that the test chamber is a double-layered cylindrical open structure, with the inner layer being the test cavity and the outer layer being a cable layer; an annular top cover is provided on the top of the test cavity, the test cavity is filled with an acoustic coupling medium, and a limiting ring is provided in the test cavity to limit the displacement of the loading force transmission block; several mounting holes are provided on the side wall of the test cavity for fixing the miniature industrial camera and the ultrasonic transducer assembly; The cable layer is provided with cables that are electrically connected to the miniature industrial camera and the ultrasonic transducer assembly. An inspection door is provided on the outside of the cable layer.

[0008] A further embodiment is that the ultrasonic transducer assembly includes a metal cavity shell, a spring loading mechanism is provided inside the metal cavity shell, an ultrasonic transducer is provided at one end of the spring loading mechanism away from the metal cavity shell, and a coupling pad is provided at the end of the ultrasonic transducer.

[0009] A further embodiment is that the circumferential mounting angle of the ultrasonic transducer assembly and the intersection angle of the miniature industrial camera are °.

[0010] A further embodiment is that the image stitching module includes a distortion correction unit, a horizontal stitching unit, and an axial fusion unit; The distortion correction unit performs distortion correction on the image data based on the geometric prior formed by the fixed installation position of the miniature industrial camera and the ultrasonic transducer in the test chamber. The horizontal stitching unit uses the contour of the ultrasonic transducer component in the image as a matching feature to stitch together multiple image data of the same layer. The axial fusion unit performs axial fusion of images stitched together from different layers to generate a panoramic view of the cylindrical surface covering the side surface of the rock-like specimen.

[0011] A further embodiment is that the ultrasound imaging module includes a data acquisition and control unit and a three-dimensional reconstruction unit; The data acquisition and control unit is used to control the ultrasound detection module to acquire ultrasound data in full matrix capture mode; The 3D reconstruction unit is configured to perform the following steps: A three-dimensional voxel mesh was defined inside the rock-like specimen. Calculate the acoustic path and flight time from each voxel to the transmitter-receiver pair; All received signals at the corresponding time point are delayed and superimposed, and then assigned to voxel points; After traversing all voxel points, a three-dimensional reflection intensity image reflecting the spatial distribution of internal defects is reconstructed.

[0012] A further embodiment is that the defect identification and segmentation quantization module includes a surface defect processing unit and an internal defect processing unit; The surface defect processing unit performs semantic segmentation on the cylindrical unfolded panoramic image based on a deep learning model to obtain a binary mask of the surface defects and extract the geometric features of the defects. The internal defect processing unit performs semantic segmentation on the three-dimensional reflection intensity image based on a deep learning model to obtain a three-dimensional voxel mask of the internal defect and extract the geometric features of the defect.

[0013] A further solution includes a multimodal spatial registration module, which is communicatively connected to both the image stitching module and the internal ultrasound imaging module. The multimodal spatial registration module is used to establish the pixel coordinate system of the cylindrical unfolded panoramic image and the world coordinate system of the three-dimensional reflection intensity image, and to unify the surface defect coordinates and internal defect coordinates based on the mapping relationship between the pixel coordinate system and the world coordinate system.

[0014] A further embodiment includes a comprehensive evaluation module, which is communicatively connected to the defect identification and segmentation quantification module and the loading module, respectively. The comprehensive evaluation module is used to receive the quantitative characteristics of surface and internal defects, as well as the mechanical data during the loading process, and calculates the multimodal damage index through weighted fusion to evaluate the overall damage state and level of the rock-like specimen.

[0015] A further solution includes a visualization platform, which is communicatively connected to the image stitching module, the ultrasonic imaging module, the defect identification and segmentation quantification module, and the comprehensive evaluation module, respectively. The visualization platform is used to simultaneously render and display panoramic views of the side surface of rock-like specimens, three-dimensional distribution maps of internal defects, mechanical response curves, defect quantification indicators, and comprehensive damage indices, and to export test data.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention features a test chamber designed to accommodate rock-like specimens, preventing fragmentation. A miniature industrial camera and an ultrasonic transducer are fixedly mounted on the side wall of the chamber in an alternating array. A synchronization module based on a unified hardware clock ensures strict synchronization of image, ultrasonic, and mechanical data during acquisition. This allows the apparent deformation, internal acoustic response, and macroscopic mechanical behavior of the rock-like specimen under load to be synchronously recorded and correlated within microseconds, completely solving the problem of time misalignment of multi-source data in traditional methods.

[0017] The miniature industrial camera and ultrasonic transducer of this invention are arranged in a circumferential staggered manner, and the transducer is equipped with a coupling pad with a spring loading mechanism. This ensures that the field of view of each miniature industrial camera includes a clear image of the adjacent ultrasonic transducer, providing a stable visual reference for subsequent image processing. Moreover, the spring mechanism can adaptively maintain stable contact between the ultrasonic transducer and the surface of the curved test piece. This allows for online distortion correction of the camera lens and high-precision image stitching using the image of the ultrasonic transducer. At the same time, it ensures the stability and efficiency of ultrasonic signal coupling on the curved surface, significantly improving the overall detection accuracy and robustness.

[0018] The defect identification and segmentation quantization module of this invention adopts a surface and internal defect processing unit based on a deep learning model. The deep learning algorithm is embedded into the processing flow, enabling it to automatically identify and segment defect areas from complex panoramic images and three-dimensional volume data. Then, it automatically completes the quantification extraction of geometric features such as length, width, volume, and position. This overcomes the subjectivity, inefficiency, and inconsistency of traditional methods that rely on manual interpretation, and improves the accuracy and efficiency of defect analysis. Attached Figure Description

[0019] The following figures are for illustrative purposes only and are not intended to limit the scope of the invention, wherein: Figure 1 : Logic block diagram of the detection system of this invention; Figure 2 Schematic diagram of the test chamber structure; Figure 3 Schematic diagram of the internal structure of the test chamber; Figure 4 Schematic diagram of the ultrasonic transducer assembly; In the diagram: 1. Test chamber; 2. Inspection door; 3. Annular top cover; 4. Limiting ring; 5. Miniature industrial camera; 6. Ultrasonic testing module; 7. Front coupling pad; 8. Ultrasonic transducer; 9. Spring loading mechanism; 10. Metal cavity shell; 11. Cable; 12. Rock-like specimen; 13. Loading force transmission block; 14. Synchronization module; 15. Image detection module; 16. Loading module; 17. Image stitching module; 18. Ultrasonic imaging module; 19. Defect recognition and segmentation quantization module; 20. Multimodal spatial registration module; 21. Comprehensive evaluation module; 22. Visualization platform; 23. Distortion correction unit; 24. Lateral stitching unit; 25. Axial fusion unit; 26. Data acquisition and control unit; 27. 3D reconstruction unit; 28. Surface defect processing unit; 29. ​​Internal defect processing unit; 30. Deep learning model. Detailed Implementation

[0020] To make the objectives, technical solutions, design methods, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0021] like Figure 1-3As shown, this invention provides a multimodal detection system for damage to rock-like materials. The system simultaneously acquires surface images, internal ultrasonic signals, and mechanical response data of a rock-like specimen 12 during loading via a test chamber 1. Multimodal data processing, fusion, and comprehensive evaluation are then performed to achieve visualized monitoring and quantitative assessment of the entire damage process from initiation to failure. The test chamber 1 is a double-layered cylindrical open-top tank structure. Its inner cavity holds a standard cylindrical rock-like specimen 12, while the outer layer is a cable layer 11 for routing the cables of various sensors. The test chamber 1 has an openable annular top cover 3, and multiple mounting holes on its side walls for fixing the miniature industrial cameras 5 and ultrasonic transducer components. A loading force transmission block 13 is positioned above the rock-like specimen 12 to transmit axial loads. Multiple miniature industrial cameras 5 of the image detection module 15 are fixedly mounted in an array on the side wall of the test chamber 1 through the mounting holes, with their lenses pointing towards the side surface of the rock-like specimen 12. Multiple ultrasonic transducer components of the ultrasonic testing module 6 are also fixedly mounted in an array staggered with the miniature industrial camera 5 via mounting holes. The actuator of the loading module 16 is located above the test chamber 01, applying axial pressure to the rock-like specimen 12 via the loading force transmission block 13. Force and displacement sensors are integrated into the loading path. The synchronization module 14 is based on a unified hardware clock, and its output is connected to all miniature industrial cameras 5, ultrasonic transducers, and loading force transmission blocks 13 via hardware cables, ensuring that all three are triggered and begin data acquisition at the same microsecond level. The image stitching module 17 is connected to the image detection module 15 and is used to process and stitch images acquired by multiple miniature industrial cameras 5 to generate a cylindrical panoramic view of the side surface of the rock-like specimen 12. The ultrasonic imaging module 18 is connected to the ultrasonic detection module 6 and is used to process ultrasonic signals acquired by multiple ultrasonic transducers to reconstruct a three-dimensional image of internal defects in the rock-like specimen 12. The defect identification and segmentation quantization module 19 is connected to the image stitching module 17 and the ultrasonic imaging module 18 respectively and is used to identify and quantify surface defects from the cylindrical panoramic view and identify and quantify internal defects from the three-dimensional image.

[0022] like Figure 4 As shown, the ultrasonic transducer assembly includes a metal cavity shell 10. A spring loading mechanism 9 is disposed inside the metal cavity shell 10. An ultrasonic transducer 8 is disposed at one end of the spring loading mechanism 9 away from the metal cavity shell 10. A coupling pad 7 is disposed at the end of the ultrasonic transducer 8. The coupling pad 7 at the front end maintains stable contact with the side surface of the rock-like specimen 12 under the action of the spring loading mechanism 9. In this embodiment, the coupling pad 7 is a replaceable polyurethane or silicone rubber coupling pad, and the stiffness of the spring loading mechanism 9 is... k =3–20 N / mm, stroke 3–15 mm, providing preload Fp=5–40N, ensuring stable contact pressure without damaging the surface of the rock-like specimen 12. Sixteen ultrasonic transducers 8 are arranged orthogonally in a 4-layer × 4-unit configuration, supporting transmission imaging and pulse-echo modes. Sixteen miniature industrial cameras 5 are also arranged orthogonally, supporting hardware synchronous triggering and integrating LED supplementary lighting to adapt to low-light environments. The ultrasonic transducers 8 are arranged in a 4-layer configuration (4 transducers per layer, 4 layers total), with circumferential angles of 45°, 135°, 225°, and 315°, staggered by 45° from the miniature industrial cameras 5. That is, the miniature industrial cameras 5 are arranged in a 4-layer configuration (4 cameras per layer, 4 layers total), with circumferential angles of 0°, 90°, 180°, and 270°. This ensures that the field of view of each miniature industrial camera 5 necessarily includes a portion of the outline of the adjacent ultrasonic transducer 8. These clearly defined, known-size ultrasonic transducers 8 serve as calibration objects in the specimen environment. In subsequent image processing, it is used for distortion self-correction based on geometric priors and image stitching as a strong constraint feature. In this embodiment, the focal length of the miniature industrial camera 5 is f=6–16mm, the pixel size is p=2–5μm, and the resolution is ≥8MP; the hardware trigger synchronization error is <200µs.

[0023] In the above, the image stitching module 17 receives the original image sequence triggered by the synchronization module 14 and acquired by the image detection module 15, and processes multiple local images into a complete panoramic view of the side surface cylinder. The image stitching module 17 specifically includes a distortion correction unit 23, a lateral stitching unit 24, and an axial fusion unit 25. The distortion correction unit 23 is configured to perform the following steps: During system initialization or installation and commissioning, the array parameters of the test chamber 1, the miniature industrial camera 5, and the ultrasonic transducer assembly are used as known geometric priors. These parameters include: the number of layers in the miniature industrial camera 5 L=4, and the installation height of each layer. zl ; Circumferential angle of miniature industrial camera: θc ( l = [0°, 90°, 180°, 270°]; Circumferential angle of ultrasonic transducer: θt ( l = [45°, 135°, 225°, 315°]; Installation radius Rc (Miniature industrial camera 5) Rt (Transducer), usually Rt < Rc Ultrasonic transducer geometric parameters: outer diameter dt ,high ht .

[0024] When the synchronization module sends a trigger signal, the distortion correction unit acquires a single frame image synchronously captured by all the miniature industrial cameras 5. For each image, the distortion correction unit performs the following automated processing to locate the contour of the ultrasonic transducer 8 in the image: Gaussian filtering is applied to the image to suppress noise, and the contrast-limited adaptive histogram equalization (CLAHE) algorithm is used to enhance the overall contrast of the image, making the edges of the metal shell of the ultrasonic transducer 8 clearer. Edge detection is performed using the Canny operator, and all connected components are extracted using a contour finding algorithm based on the known physical dimensions of the ultrasonic transducer 8. dt and ht Based on the approximate field of view of the miniature industrial camera 5, the pixel area that the contour of the ultrasonic transducer 8 should occupy in the image is estimated, thereby selecting contours that meet the criteria. Ellipse fitting is performed on the selected contours to obtain their center coordinates, major and minor axes, and tilt angles. Semantic verification is performed using angular priors. For example, for a camera with an installation angle of 0°, transducer contours with angles of 45° and 315° should theoretically appear in its field of view. This relationship can be used to eliminate falsely detected contours caused by reflections or dirt, ensuring the accuracy of the detection.

[0025] For each detected ultrasonic transducer 8, its 3D model points By projecting the initial intrinsic parameters (K0) and extrinsic parameters (Rj, Tj) of the camera onto the image plane, a theoretically distortion-free point set is obtained. Discretize the fitted ellipse into a point set. Construct a nonlinear optimization problem: ; in, For a distorted projection function, The distortion coefficient to be optimized The radial distortion coefficient of the camera lens. The tangential distortion coefficient of the camera lens. For the ultrasonic transducer, the known 3D CAD model on the first The three-dimensional world coordinates of a feature point To obtain the contour of the ultrasonic transducer from the actually acquired image through image processing (edge ​​detection, ellipse fitting), the first... The observed two-dimensional pixel coordinates of each point; Output the corrected distortion parameters It is used to perform image distortion correction on the original image. This process does not require a calibration plate and achieves self-calibration by utilizing the actual structure in the working state.

[0026] The horizontal stitching unit is used to stitch together multiple corrected images at different circumferential angles on the same height layer into a 360-degree panoramic view of that layer. Specifically, for images on the same layer (e.g., four miniature industrial cameras 5 at 0°, 90°, 180°, and 270°), the horizontal stitching unit performs the following operations: for adjacent cameras on the same layer (e.g., 0° and 90°), extract the fitting center (cA, cB) of the same ultrasonic transducer 8 (e.g., 45° ultrasonic transducer 8) that is visible to both cameras in the two images; use this point pair as a strongly constrained matching point and add it to the visual feature matching result; when estimating the homography matrix (H) using RANSAC, force the reprojection error of this point pair to be ≤1 pixel; stitch the four images sequentially and project them onto the cylindrical unfolded coordinate system to form a 360° panoramic view of that layer.

[0027] Cylindrical expansion formula: ; in For the specimen radius, , The horizontal (circumferential) pixel coordinates of the panoramic image unfolded from the cylindrical surface. For the vertical (axial) pixel coordinates of the panoramic image unfolded from the cylindrical surface, The axial height coordinate of a point on the surface of the specimen in three-dimensional space; The axial fusion unit is responsible for fusing the pre-stitched cylindrical images of different height layers into a continuous panoramic image covering the entire side surface of the specimen. The axial fusion unit uses the stable ultrasonic transducers 8 present in each layer as the axial alignment reference and performs the following operations: Each layer of ultrasonic transducers 8 is located at the center of the image height of that layer. During vertical stitching, the center height (zl) of each layer of ultrasonic transducers 8 is used as the alignment anchor point, and the area of ​​ultrasonic transducer 8 ( zl ± ht / 2) Rigid alignment is employed to prevent texture distortion, and multi-band blending is used in the non-ultrasonic transducer 8 region to achieve a smooth transition; the final output is a high-resolution cylindrical panoramic image. I pano( u , v） It covers the entire 12 side surfaces of the rock-like specimen.

[0028] The ultrasonic imaging module 18 receives the raw ultrasonic signal triggered by the synchronization module 14 and acquired by the ultrasonic detection module 6. This signal is then processed by the data acquisition control unit 26 and the three-dimensional reconstruction unit 27. Specifically, the data acquisition control unit 26 precisely controls the ultrasonic hardware system to operate according to a predetermined advanced mode. Its core task is to perform full matrix acquisition. When the synchronization module 14 sends an acquisition trigger signal, the data acquisition control unit 26 initiates a complete FMC data acquisition cycle, controlling all ultrasonic transducers 8 to perform the following operations sequentially according to a preset timing sequence: The first ultrasonic transducer 8 is selected as the transmitter Tx1, generating an ultrasonic pulse with a center frequency and bandwidth conforming to preset parameters. This pulse is transmitted into the specimen. All ultrasonic transducers 8 are controlled to synchronously act as receivers Rx, starting to record echo signals. Each receiver records a voltage waveform that varies over time, i.e., an A-scan signal. The transmitter role is then sequentially switched to the second ultrasonic transducer 8 Tx2, repeating the above transmission and full reception process until all ultrasonic transducers 8 have completed one transmission in turn. During this process, the data acquisition and control unit 26 ensures that the sampling rate of the ultrasonic transmitter and receiver is not less than 10MHz in each transmit-receive event, and sets a sufficiently long time window to ensure that the maximum possible sound path of the ultrasonic wave propagating within the specimen is covered, for example, the specimen height H plus its half-perimeter πD / 2. After one FMC acquisition cycle is completed, the data acquisition and control unit 26 will obtain a complete 16×16-dimensional ultrasonic signal matrix. This matrix contains 16×15 valid independent propagation paths of A-scan signals, as well as 16 self-excited and self-received signals. The data acquisition and control unit 26 synchronously records the precise three-dimensional spatial coordinates of each transmitting ultrasonic transducer 8 and receiving ultrasonic transducer 8, forming a spatial position matrix, and packages all signals and position information, along with a unified timestamp, and outputs them to the three-dimensional reconstruction unit 27.

[0029] The 3D reconstruction unit 27 is responsible for converting the raw waveform matrix provided by the data acquisition and control unit 26 into an intuitive 3D defect image using an imaging algorithm. After receiving the FMC data packet, the 3D reconstruction unit 27 processes it according to the following procedure: A three-dimensional voxel mesh V={( xi , yj , zk Voxel resolution 2–3 mm, mesh range: x , y ∈[- D / 2, D / 2], z ∈[0, H ].

[0030] Calculate the sound path for each voxel. =( x ,y , z ), calculate its total acoustic path to all Tx-Rx pairs: ; in, Let the coordinates be those of the voxel points. =(x,y,z), For the first The three-dimensional spatial coordinates of the ultrasonic transducer that serves as the transmitter. For the first The three-dimensional spatial coordinates of the ultrasonic transducer that serves as the transmitter; Flight time is: ; in, The initial value is 4000 m / s, which can be corrected later.

[0031] Delayed stacking imaging: ; in, To reconstruct voxel points after full-focusing method The reflection intensity value at that location, Actual measured A-scan signal; As a compensation factor; The final output is three-dimensional volume data. I TFM ( x , y , z） The unit is reflectance intensity (dB). It shows scattering bodies such as internal pores, cracks, and aggregate interfaces.

[0032] In the above description, the defect recognition and segmentation quantization module 19 receives the cylindrical unfolded panoramic image from the image stitching module 17 and the three-dimensional reflection intensity image from the ultrasonic imaging module 18, and outputs the precise geometric features of the surface and internal defects of the specimen. This module includes a surface defect processing unit 28 and an internal defect processing unit 29. The surface defect processing unit 28 performs semantic segmentation on the cylindrical unfolded panoramic image based on a deep learning model 30 to obtain a binary mask of the surface defects and extract the geometric features of the defects. The deep learning model 30 uses the lightweight U-Net to achieve defect recognition. The U-Net network has been widely used in medical image segmentation and industrial defect detection tasks. Its input is the cylindrical panoramic image, and the encoder uses a pre-trained MobileNetV3 or EfficientNet-Lite as the encoder to extract features, balancing accuracy and speed. The decoder uses a standard U-Net decoder to upsample the image, restore spatial resolution, and generate a defect mask. The loss function combines Dice Loss (for handling class imbalance problems) and Boundary Loss (to enhance the segmentation accuracy of defect edges), automatically processes the input image, and outputs a binary mask Msurf(u,v)∈{0,1} for the defect region. M surf Mapping to 3D space: ; A value of 1 indicates a defect area, and a value of 0 indicates a background area.

[0033] The initial binary mask may contain noise, small holes, or breaks due to uneven illumination. Therefore, the surface defect processing unit 28 performs morphological post-processing operations to optimize the segmentation result. Specifically, morphological closing operations are used to fill small gaps in the defects and eliminate noise interference. Elliptical structuring elements (5 pixels on the major axis, 2 pixels on the minor axis) are used to fill breakpoints inside the defects, removing defect discontinuities caused by noise, illumination variations, etc. The centerline of the defect is extracted using the Zhang-Suen skeleton extraction algorithm, and the skeleton points are then used to extract the centerline of the defect. p i Connections are formed. By extracting the skeleton, the geometric features of the defect (such as length and width) can be ensured to be more accurate. The final output is an accurate defect mask and defect skeleton. The surface defect processing unit 28 calculates various geometric features of each individual defect based on the defect mask and defect skeleton, including: Input: For each defect region Ci Based on its corresponding 3D surface coordinate field Psurf( u , v ); Projected length: The projected length of the defect can be calculated by summing the 3D Euclidean distances along the skeleton points. ; in: For the first The three-dimensional projection length of a surface defect. , and It is the first defect on the skeleton and the The 3D coordinates of the points are obtained through a skeleton extraction algorithm.

[0034] Average width: The width of the defect can be calculated by the ratio of the area of ​​the mask region to the length of the skeleton. ; in, For the first The average width of each surface defect For the first The pixel area occupied by a defect in a binary mask, where Apixel is the actual area of ​​each pixel.

[0035] Maximum width: Calculates the maximum span of the defect in the direction perpendicular to the skeleton.

[0036] Orientation angle: Perform principal component analysis (PCA) on the skeleton points to calculate the orientation angle of the defect (the angle between the main direction of the skeleton and the reference direction).

[0037] Spatial location: Calculate the centroid coordinates of the defect , These are the 3D coordinates of each point within the defect area. These points belong to the defect area. A small set, It is the number of points within the defect area, that is, the total number of pixels or voxels contained in the defect mask within the area.

[0038] Height distribution: Calculate the z-coordinate range of the defect [ z min , z max [Evaluate the height distribution of defects.]

[0039] Output: Geometric features of the defect, including projected length, width, orientation angle, spatial location, and height distribution.

[0040] The internal defect processing unit 29 performs semantic segmentation on the three-dimensional reflection intensity image based on the deep learning model 30 to obtain the three-dimensional voxel mask of the internal defects and extract the geometric features of the defects. The deep learning model 30 adopts a 3DU-Net network structure, which is specifically designed for processing three-dimensional volume data and can automatically extract internal defects from ultrasound data. Its input is the original three-dimensional ultrasound imaging data. I TFM (x , y , z） The encoder uses multiple convolutional layers to extract features from volumetric data, progressively reducing resolution to capture greater contextual information. The decoder progressively restores image resolution and accurately locates defect regions. Skip connections preserve low-level details, helping to improve segmentation accuracy for edges and small defects. The loss function uses Dice Loss (a common loss function for measuring segmentation accuracy) to improve the model's accuracy on defect regions. Boundary Loss can also be combined to enhance accurate segmentation of defect edges. The trained model is then used to infer new 3D ultrasound data, outputting a voxel-level defect probability map. The probability map is defined as follows: ∈[0,1], where a value of 1 represents a defect and 0 represents no defect. Binarization is then performed, with a threshold of 0.5. Voxels in the probability map exceeding this threshold are marked as defect regions, and the remaining voxels are marked as background. This process is then applied to the probability map of internal defects. Convert to binary defect mask The value 1 represents a defect, and 0 represents the background. Morphological closing operations are used to fill small holes and fracture areas, employing structuring elements such as cubes or spheres. Finally, the binary mask is divided into several connected regions (i.e., independent defect regions), each representing a defect volume, and each independent defect volume is labeled.

[0041] For each individually marked defect, calculate its three-dimensional geometric features: Volume: The volume of each connected component Calculated by the number of voxels: ; in: It is the first The number of voxels in each defect region. It is the size of the voxel, which is usually determined by the spatial resolution of ultrasound imaging.

[0042] Centroid coordinates: The centroid coordinates of each defect region ( It is a weighted average of the coordinates of all voxels within the region: ; in: It is the first A set of voxels representing a defect region. It is a defective voxel The three-dimensional coordinates It is the number of voxels in the defect region.

[0043] Equivalent sphere radius, through volume Calculate the equivalent sphere radius This radius represents the radius of a sphere with the same volume as the defect: ; in, For the first The equivalent sphere radius of an internal defect; Principal component analysis (PCA) is performed on the voxel coordinates of the defect region to extract the principal direction of the defect.

[0044] Anisotropy index of defects This reflects whether the shape of the defect is biased towards a specific direction: ; in, These are the eigenvalues ​​of the covariance matrix of the defect region, sorted in order of magnitude.

[0045] Calculate the depth, the depth of each defect. It is the minimum distance from the surface to the centroid of the defect: ; in It is the radius of the specimen. These are the coordinates of the center of mass on the plane.

[0046] Height distribution, calculate the range of the z-coordinate of the defect, i.e., from arrive The height range.

[0047] Output the geometric characteristics of the defect, including volume, centroid coordinates, equivalent sphere radius, principal direction and anisotropy index, depth, and height distribution.

[0048] In this embodiment, the defect identification and segmentation quantization module 19 also has the capability to perform spatial correlation analysis and calculate internal defect evaluation indicators based on its output: The spatial correlation analysis process between defects is as follows: For each internal defect body Find the area of ​​influence of its projection on the surface and calculate the centroid of the defect. : Projection onto the surface: The centroid of the internal defect is projected onto the surface to obtain its position on the surface. The formula for calculating the projection is as follows: ; in, It is the radius of the specimen. These are the centroid coordinates of the defect.

[0049] For each projection position In the point cloud of surface defects Search the neighborhood.

[0050] The radius of the neighborhood search is set to 10 mm and the height difference to 5 mm to identify the spatial relationship between surface defects and internal defects.

[0051] If the projection of an internal defect intersects with that of a surface defect (i.e., their spatial locations are close), it is marked as an internal-external related defect pair to indicate that there is a physical connection between the internal defect and the surface defect.

[0052] The calculation process for internal defect evaluation indicators is as follows: Calculate the sound velocity decay rate

[0053] ; in, The speed of sound when there are no defects. The average speed of sound when defects are present.

[0054] Calculate the defect volume growth rate That is, the rate of change of defect volume with load: ; in, For the applied load, Let be the volume of the defect.

[0055] Calculate the critical defect size Defined as when the defect volume exceeds 200 mm 3 At this point, the structure may suffer significant damage or failure, corresponding to a load level known as the critical load. .

[0056] Calculate the internal-external connectivity index It is used to assess defect penetration, and the calculation formula is: ; The total number of internal defects refers to the total number of defects detected within the structure. The number of associated defect pairs refers to the number of defect pairs that are spatially related to surface defects and internal defects.

[0057] Continue to refer to Figure 1 In this embodiment, a multimodal spatial registration module 20 is also included. This module is communicatively connected to the image stitching module 17 and the ultrasound imaging module 18, respectively, and receives the high-resolution cylindrical unfolded panoramic image output from the image stitching module 17 and the three-dimensional reflection intensity image output from the ultrasound imaging module 18. Expand the cylindrical surface to a panoramic image pixel ( u , v） Mapped to world coordinates via cylindrical back projection: ; Constructing the surface coordinate field ( u , v） →( x , y , z） This lays the foundation for understanding the relationship between internal and external defects.

[0058] In this embodiment, a comprehensive evaluation module 21 is also included. This module generates damage status indicators by fusing damage information from different dimensions, enabling dynamic evaluation and early warning of the service safety of the rock-like specimen. This module is communicatively connected to the defect identification and segmentation quantification module 19 and the loading module 16, respectively, and receives three key time-series data streams as input, as shown in the table below:

[0059] All data are aligned with a unified timestamp to form a time series data matrix D(t)=[L,W,V,P,K / K0,η,].

[0060] To eliminate dimensional differences, each indicator is normalized to the [0,1] interval: Apparent damage normalization: ; in , =0.5 width weight, The normalized apparent damage value. This represents the theoretical maximum defect path length. This is the maximum width.

[0061] Internal damage normalization: ; in, This is the normalized internal damage value. This represents the total volume of internal defects. =0.5% × total specimen volume =0.7, =0.3, This is the internal-external connectivity index.

[0062] Mechanical degradation normalization: ; in, This is the normalized mechanical damage value. Given the current instantaneous stiffness, For the initial elastic stiffness, The sound velocity attenuation rate, K fail / K 0 ≈ 0.3 (failure stiffness ratio) λ=0.6.

[0063] Weighting coefficients ( α , βi , λ It can be dynamically adjusted according to the material type or test objective.

[0064] Define the comprehensive damage index: ; in, Let be a multimodal damage index, ranging from [0, 1], with weights satisfying . w 1+ w 2+ w 3=1, Recommended initial configuration: Standard strength rock specimen: w 1 = 0.4, w 2 = 0.3, w 3=0.3; High-strength / ultra-high-performance rock-like materials: w 1 = 0.2, w 2 = 0.5, w 3 = 0.3; Fatigue / Cyclic Loading: w 3 = 0.4–0.5 (emphasizing stiffness degradation).

[0065] MMDI∈[0,1], the larger the value, the more severe the damage.

[0066] Based on the MMDI threshold, the four-level damage state criteria are established as shown in the table below:

[0067] In this embodiment, a visualization platform 22 is also included to synchronously display the apparent defects, internal defects, and mechanical responses of the rock-like specimen 12 during the loading process, thereby achieving visual perception of damage. The visualization platform 22 is connected to the image stitching module 17, the ultrasonic imaging module 18, the defect identification and segmentation quantization module 19, and the comprehensive evaluation module 21, respectively. It receives and synchronously renders the following core data in real time: a panoramic view of the cylindrical surface unfolding of the side surface of the rock-like specimen 12, a three-dimensional spatial distribution rendering model of the internal defects, the load-displacement and MMDI evolution curves during the loading process, and the geometric quantification index and damage level of the defects. It also exports data in the following general formats: defect point cloud (.ply), defect voxel (.vtk), quantification index (.csv), and original synchronous data (.h5).

[0068] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multimodal damage detection system for rock-like materials, characterized in that, include: The test chamber (1) has an internal test cavity for fixing rock-like specimens (12). The image detection module (15) includes multiple miniature industrial cameras (5), which are fixedly installed in an array on the periphery of the test chamber to acquire images of the side surface of the rock-like specimen (12). The ultrasonic testing module (6) includes multiple ultrasonic transducers, which are fixedly installed in an array on the periphery of the test chamber and are staggered with the miniature industrial camera (5) in the circumferential direction to collect ultrasonic signals that penetrate or are reflected by the rock-like specimen (12). The loading module (16) includes a loading force transmission block (13) disposed in the upper part of the inner cavity of the test chamber (1) for applying axial load to the rock-like specimen (12); The synchronization module (14) is connected to the image detection module (15), the ultrasonic detection module (6) and the loading module (16) respectively. Based on a unified hardware clock, it outputs a synchronization trigger signal to the micro industrial camera (5), the ultrasonic transducer and the loading force block (13) to synchronize the acquisition time of image data, ultrasonic data and mechanical data. The image stitching module (17) is connected to the image detection module (15) and is used to process and stitch the images acquired by multiple miniature industrial cameras (5) to generate a panoramic view of the cylindrical surface unfolded on the side surface of the rock-like specimen (12). The ultrasonic imaging module (18) is connected to the ultrasonic detection module (6) and is used to process the ultrasonic signals collected by the multiple ultrasonic transducers to reconstruct a three-dimensional image of the internal defects of the rock-like specimen (12). The defect identification and segmentation quantization module (19) is connected to the image stitching module (17) and the ultrasonic imaging module (18) respectively, and is used to identify and quantify surface defects from the panoramic image unfolded from the cylindrical surface, and to identify and quantify internal defects from the three-dimensional image; The ultrasonic transducer assembly includes a metal cavity shell (10), a spring loading mechanism (9) is provided inside the metal cavity shell (10), an ultrasonic transducer (8) is provided at one end of the spring loading mechanism (9) away from the metal cavity shell (10), and a coupling pad (7) is provided at the end of the ultrasonic transducer (8). The image stitching module (17) includes a distortion correction unit (23), a horizontal stitching unit (24), and an axial fusion unit (25). The distortion correction unit (23) performs distortion correction on the image data based on the geometric prior formed by the fixed installation positions of the micro industrial camera (5) and the ultrasonic transducer in the test chamber (1). The horizontal stitching unit (24) uses the contour of the ultrasonic transducer in the image as a matching feature to stitch together multiple image data of the same layer; The axial fusion unit (25) performs axial fusion of images spliced ​​from different layers to generate a cylindrical panoramic view covering the side surface of the rock-like specimen (12). The ultrasound imaging module (18) includes a data acquisition and control unit (26) and a three-dimensional reconstruction unit (27). The data acquisition control unit (26) is used to control the ultrasound detection module (6) to acquire ultrasound data in full matrix capture mode; The three-dimensional reconstruction unit (27) is configured to perform the following steps: A three-dimensional voxel mesh is defined inside the rock-like specimen (12); Calculate the acoustic path and flight time from each voxel to the transmitter-receiver pair; All received signals at the corresponding time point are delayed and superimposed, and then assigned to voxel points; After traversing all voxel points, a three-dimensional reflection intensity image reflecting the spatial distribution of internal defects is reconstructed. It also includes a multimodal spatial registration module (20), which is communicatively connected to the image stitching module (17) and the internal ultrasound imaging module (18), respectively; The multimodal spatial registration module (20) is used to establish the pixel coordinate system of the cylindrical unfolded panoramic image and the world coordinate system of the three-dimensional reflection intensity image, and to unify the surface defect coordinates and internal defect coordinates based on the mapping relationship between the pixel coordinate system and the world coordinate system.

2. The multimodal detection system for rock-like material damage according to claim 1, characterized in that, The test chamber (1) is a double-layer cylindrical open structure, with the inner layer being the test cavity and the outer layer being the cable layer (11); an annular top cover (3) is provided on the top of the test cavity, the test cavity is filled with an acoustic coupling medium, and a limiting ring (4) is provided in the test cavity to limit the displacement of the loading force transmission block (13); several mounting holes are provided on the side wall of the test cavity for fixing the miniature industrial camera (5) and the ultrasonic transducer assembly; The cable layer (11) is provided with cables, which are electrically connected to the micro industrial camera (5) and the ultrasonic transducer assembly. An inspection door (2) is provided on the outside of the cable layer (11).

3. The multimodal detection system for damage to rock-like materials according to claim 2, characterized in that, The circumferential mounting angle of the ultrasonic transducer assembly and the intersection angle of the miniature industrial camera (5) is 45°.

4. The multimodal damage detection system for rock-like materials according to claim 3, characterized in that, The defect identification and segmentation quantization module (19) includes a surface defect processing unit (28) and an internal defect processing unit (29). The surface defect processing unit (28) performs semantic segmentation on the cylindrical unfolded panoramic image based on the deep learning model (30), obtains the binary mask of the surface defect, and extracts the geometric features of the defect. The internal defect processing unit (29) performs semantic segmentation on the three-dimensional reflection intensity image based on the deep learning model (30), obtains the three-dimensional voxel mask of the internal defect, and extracts the geometric features of the defect.

5. The multimodal damage detection system for rock-like materials according to claim 4, characterized in that, It also includes a comprehensive evaluation module (21), which is connected in communication with the defect identification and segmentation quantification module (19) and the loading module (16), respectively. The comprehensive evaluation module (21) is used to receive the quantitative characteristics of surface and internal defects, as well as the mechanical data during the loading process, and calculate the multimodal damage index by weighted fusion to evaluate the overall damage state and level of the rock-like specimen (12).

6. The multimodal damage detection system for rock-like materials according to claim 5, characterized in that, It also includes a visualization platform (22), which is connected to the image stitching module (17), the ultrasound imaging module (18), the defect identification and segmentation quantification module (19), and the comprehensive evaluation module (21), respectively. The visualization platform (22) is used to synchronously render and display a panoramic view of the side surface of the rock-like specimen (12), a three-dimensional distribution map of internal defects, a mechanical response curve, a defect quantification index and a comprehensive damage index, and to export test data.

Images