Composite damage ultrasonic testing method based on deep learning and augmented reality

Abstract

This invention discloses an ultrasonic inspection method for composite material damage based on deep learning and augmented reality, aiming to solve the problems of limited applicability of traditional flaw detection, insufficient detection accuracy of deep learning, and poor readability of results. The method first constructs an inspection platform including a robotic arm, binocular camera, and AR glasses. The binocular camera replaces the traditional position encoder, and a stereo matching algorithm is used to obtain the ultrasonic probe's position information. This information is then fused with the ultrasonic A-scan signal processed by Kalman filtering to generate an ultrasonic point cloud. An improved YOLOv5 model with an attention mechanism is used to accurately detect the probe. A 1DCNN classifies and colors the defect signals to enhance discriminative power, and an LSC-YOLO network extracts the spatial features of the test piece. Using Apriltag localization and AR glasses, the point cloud is projected onto the test piece to visualize the damage, improving inspection efficiency and accuracy, and making the defect location intuitively presented. This method is suitable for composite material inspection in the aerospace field.

Description

Technical Field

[0001] This invention belongs to the field of material flaw detection technology, and in particular relates to an ultrasonic detection method for composite material damage based on deep learning and augmented reality. Background Technology

[0002] Since the 1970s, the application of composite materials in the aerospace materials field has experienced explosive growth, gradually becoming an indispensable key material in the aerospace industry. Technological advancements and innovations have continuously driven the increasing proportion of composite materials used in aircraft structures. Composite materials are novel materials created by combining two or more materials with different properties through physical or chemical methods. They not only retain certain characteristics of the original materials but also possess new and superior properties. Composite materials have unique advantages such as low density, high specific strength and high specific modulus, strong corrosion resistance, good material designability, and excellent fatigue resistance. In particular, the high specific strength and high specific modulus have enormous potential in aircraft design. During the aircraft structural design process, based on the different forces borne by each component, the main load-bearing components can be strengthened, while non-load-bearing parts can be appropriately weakened, thereby achieving structural optimization. This design method can effectively reduce the weight of structural components and the entire aircraft structure, while improving aircraft performance and efficiency.

[0003] Currently, non-destructive testing (NDT) technologies mainly include magnetic particle testing, penetrant testing, eddy current testing, and ultrasonic testing. Magnetic particle testing can detect surface and near-surface defects in ferromagnetic materials, but it is difficult to determine defect depth and is only applicable to ferromagnetic materials. Penetrant testing is cost-effective and efficient, and can detect defects such as folds, pinholes, cracks, and porosity on the surfaces of metallic and non-metallic materials, but it cannot measure damage depth and is not suitable for porous materials. Eddy current testing is mainly used to detect near-surface or surface defects in conductive materials. It is convenient, cost-effective, and does not require coupling, but it cannot detect deep defects and it is difficult to determine the type of defect. Ultrasonic testing has strong penetrating power, concentrated energy direction, high sensitivity, low testing cost, and is harmless to human health. Ultrasonic waves can penetrate into the interior of the material and are reflected when they encounter defects or the bottom surface of the structure, forming pulse waveforms on the screen. Based on this, the tester can determine the location and size of the defect. This technology can effectively detect surface and internal defects in materials.

[0004] Deep learning technology can effectively extract key features of defects in ultrasonic signals and fit and utilize them efficiently, making it a popular area of ​​development in ultrasonic testing technology. However, the superior performance of deep networks is based on large-scale data. Insufficient data can exacerbate overfitting and reduce detection accuracy. In actual testing, specimens are mostly in a healthy state, with a low probability of defects. Furthermore, the occurrence of defects in specimens is a random phenomenon, with varying defect types, morphological distributions, sizes, orientations, locations, and depths. Therefore, sufficient defect sample data is often lacking. In such environments with scarce samples, deep networks often cannot be trained efficiently and comprehensively, preventing deep learning from realizing its full potential and resulting in significant errors in defect identification accuracy and parameter analysis precision. Summary of the Invention

[0005] The purpose of this invention is to provide a composite material damage ultrasonic detection method based on deep learning and augmented reality, which solves the problems of insufficient application scope and defect judgment of traditional flaw detection in composite material detection, and the problem of accurate identification of existing deep learning-assisted ultrasonic detection, so as to achieve comprehensive, efficient and high-precision detection and readability of surface and internal defects of materials.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is a method for ultrasonic detection of composite material damage based on deep learning and augmented reality, comprising the following steps:

[0007] S1. Set up a testing platform; use the binocular camera in the platform to simultaneously acquire visual images of the ultrasonic probe and the composite material test piece;

[0008] S2. A robotic arm carrying an ultrasonic single-crystal probe scans the composite material plate under test to obtain ultrasonic A-scan signals; the ultrasonic probe target detection algorithm is used in combination with a stereo matching algorithm to obtain the position information of the ultrasonic probe; the ultrasonic A-scan signals are post-processed and the position information is fused with the post-processed ultrasonic A-scan signals to form ultrasonic data with coordinate information, which is presented in the form of point cloud imaging; the ultrasonic probe dataset is subjected to data augmentation and annotation conversion.

[0009] S3. A one-dimensional convolutional neural network is used to classify the defect signals and non-defect signals in the ultrasonic A-scan signals obtained and post-processed in S2. Different RGB color information is assigned to the classified signals to enhance the visual distinction between defective and non-defective areas.

[0010] S4. Extract features of composite material test parts using segmentation model: Based on the visual images of composite material test parts collected by the binocular camera in the detection platform, the composite material test parts themselves are identified by the lightweight edge convolution LSC-YOLO network to obtain their relative position in space.

[0011] S5. Obtain a three-dimensional coordinate system by scanning the Apriltag positioning code through an AR glasses, transmit the ultrasonic point cloud image processed in S3 to a portable computing unit through a network transmission protocol, perform a spatial transformation on the ultrasonic point cloud image using a perspective transformation method, and project the imaging result onto the three-dimensional coordinate area corresponding to the test piece to achieve damage visualization.

[0012] Furthermore, in the step S1, the detection platform includes a robotic arm, an optical platform, a binocular camera, a probe fixture, an ultrasonic single crystal probe, an ultrasonic acquisition card, a composite material plate, a composite material plate fixing fixture, an AR glasses, a supporting portable computing unit Seerpad, an Apriltag positioning code, and a portable processing terminal;

[0013] After the robotic arm is assembled with the probe fixture, it holds the ultrasonic single crystal probe and performs a scanning operation on the composite material test piece along a preset scanning path; the optical platform is used to carry experimental equipment and cooperate with the composite material plate fixing fixture to fix the composite material plate; the binocular camera is used to collect the visual information of the ultrasonic probe and transmit it to the portable processing terminal through a data cable; the ultrasonic signal collected by the ultrasonic single crystal probe is transmitted to the portable processing terminal through the ultrasonic acquisition card; the portable processing terminal is used to run an ultrasonic probe target detection algorithm and a stereo matching algorithm; the portable computing unit supporting the AR glasses is Seerpad, which is used to realize the visualization of the ultrasonic point cloud; the Apriltag positioning code is used to provide a spatial coordinate system.

[0014] Furthermore, in the step S2, the core loss function of the ultrasonic probe target detection algorithm is:

[0015]

[0016] where IoU is the overlap degree, is the center of the prediction box and the center of the true box the distance between them, is a preset distance scale parameter, is a trade-off coefficient, is a measure of the difference in the aspect ratio between the prediction box and the true box.

[0017] Furthermore, in the step S2, the method for determining the disparity of the stereo matching algorithm:

[0018]

[0019] where, is the adjacent disparity aggregation cost corresponding to the disparity candidate value pixel , To achieve the optimal disparity value, To achieve the optimal disparity value, Image pixel coordinates, To optimize the operator, This represents the maximum disparity.

[0020] Furthermore, the post-processing of the ultrasonic signal in step S2 employs a Kalman filtering algorithm, specifically as follows:

[0021]

[0022] in, For Kalman gain, This is the actual measured value. For the first Prior state estimation at time 10:00 For the first Posterior state estimation at time 10:00. For the first The observation matrix at each time point.

[0023] Furthermore, in step S2, the ultrasonic probe target detection algorithm uses an improved YOLOv5 model that incorporates the CBAM attention mechanism. This model replaces the original CSP module in the YOLOv5s network backbone with a CBAM attention module and introduces an NWD loss function to optimize model training; the channel attention of the CBAM attention module... Specifically:

[0024]

[0025] in, For global average pooling, Global max pooling, For channel attention mechanism on feature map The weighted output, It is a multilayer perceptron. This is the Sigmoid activation function.

[0026] Furthermore, the data augmentation process in step S2 includes rotation, scaling, cropping, and color adjustment, expanding the initial dataset size to twice its original size; the annotation conversion involves manually annotating the ultrasound probes in the enhanced images using LabelImg software, and converting all annotation results into YOLO format txt files.

[0027] Furthermore, in step S3, the one-dimensional convolutional neural network includes three feature extraction modules and a classifier structure; the feature extraction module consists of a Conv1D layer, a BN layer, and a Maxpooling layer; the input of the one-dimensional convolutional neural network is the original ultrasound signal with a data dimension of 550×1; the classifier structure sequentially includes a Flatten layer, a Linear layer, a BN layer, and a SoftMax activation function, and the output is a binary classification result, where the classification result for normal signals is 0 and the classification result for defective signals is 1.

[0028] Furthermore, in step S4, the improvements to the LSC-YOLO network include:

[0029] In the backbone, the C2f module is improved by an edge convolutional module (SCM). The SCM uses a SobelConv branch and a CNN dual convolution mode, and the edge feature fusion method is as follows: Where G is the fused edge feature output by the SobelConv branch, G X G represents the horizontal edge gradient feature. Y Vertical edge gradient features;

[0030] In the Neck section, the StarNet architecture is introduced to lightweight the C2f module, resulting in the C2f-star module. The input feature map to the C2f-star module is first processed through a convolution for feature extraction. Then, the Split module divides the feature map into two parts: one part is integrated into a deeper network for further feature extraction, and the other part is fused with the deeper features to form a feature map containing multi-level feature information.

[0031]

[0032] StarBlock is the star operation. Z is a 1×1 convolutional layer, which is the output feature map obtained by splicing and fusing after branching and feature transformation, containing multi-level feature information. X_main is the main branch feature map; X_side is the side branch feature map.

[0033] In the Head section, a lightweight detection head LSCS is used, and the feature fusion method is as follows:

[0034]

[0035] Where S is the output feature map after multi-scale fusion, and Concat is the feature concatenation operation. It is a 3×3 convolutional layer. There are n branch feature maps of different scales, GlobalAvgPool is the global average pooling operation, and F is the base input feature map to be fused.

[0036] Furthermore, in step S5, the network transmission protocol is the HTTP protocol; the image fusion uses a perspective transformation method to spatially transform the ultrasound point cloud image; the ultrasound point cloud image updates the damage visualization result by cyclically reading fixed address files at equal intervals, and repeats the above processing for each frame of the image to obtain a continuous visualization effect.

[0037] Compared with the prior art, the beneficial effects of the present invention include the following:

[0038] This invention innovatively employs a binocular camera combined with a stereo matching algorithm to extract ultrasonic probe position information, replacing the traditional acquisition method that relies on a position encoder. This not only significantly reduces the hardware cost of the testing equipment but also avoids the problems of complex installation and debugging and limited adaptability of position encoders. It achieves flexible and accurate tracking of the ultrasonic probe position, providing a reliable foundation for the efficient generation of ultrasonic point clouds. By aligning the position information acquired through binocular vision with the ultrasonic A-scan signal after Kalman filtering and noise reduction in both time and features, an ultrasonic point cloud with precise coordinate information is successfully generated, effectively solving the pain points of poor readability and ambiguous internal defect location in traditional ultrasonic testing data.

[0039] This invention accurately extracts the spatial location features of composite material test parts using the LSC-YOLO lightweight segmentation model. Combined with Apriltag positioning codes, it achieves a stable establishment of a three-dimensional spatial coordinate system. The ultrasonic point cloud image is projected onto the visualization interface of AR glasses after perspective transformation, enabling real-time fusion of the test results with the actual test part. Inspectors can intuitively observe the specific location and extent of defects on the material surface and internally, without relying on professional experience to interpret complex ultrasonic wave data. This significantly reduces the difficulty of interpreting test results and substantially improves inspection efficiency and accuracy, making it particularly suitable for on-site inspection scenarios of composite materials in the aerospace field.

[0040] This invention addresses the issue of insufficient ultrasound probe positioning accuracy in traditional target detection models by incorporating an improved YOLOv5 model with a CBAM attention mechanism, increasing probe detection accuracy from 0.83 to 0.97 and ensuring the accuracy of location information fusion with ultrasound signals. Simultaneously, a 1DCNN network is used for binary classification of ultrasound A-scan signals, and color grading enhances the visual distinction between defective and non-defective regions, further improving the defect recognition accuracy of ultrasound point clouds and effectively solving the problem of difficulty in identifying minute defects. Furthermore, data augmentation techniques such as rotation and scaling are used to expand the sample size, and the NWD loss function is combined with optimized model training, significantly improving the robustness of the deep learning model in scenarios with scarce defect samples, avoiding the overfitting and accuracy degradation problems caused by insufficient data in traditional deep learning detection methods. Moreover, the detection platform of this invention has a high degree of integration; the robotic arm scanning along a preset path ensures consistent detection; the ultrasound signal processing, model inference, and AR visualization processes are seamlessly integrated; and real-time data transmission via HTTP protocol enables the generation of continuous and stable visualization effects. The overall approach combines the advantages of low cost, high flexibility, high precision and high readability, effectively making up for the limitations of traditional non-destructive testing technology in terms of its applicability and reliance on human experience for defect judgment, as well as the shortcomings of existing deep learning-assisted ultrasonic testing in terms of insufficient visualization and high cost, thus providing an efficient and reliable technical solution for composite material damage detection. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a diagram illustrating the structure of the ultrasonic testing and visualization platform in this embodiment.

[0043] Figure 2 This is a schematic diagram of the working principle of the ultrasonic testing and visualization platform in this embodiment;

[0044] Figure 3 This implementation improves the YOLOv5 network structure;

[0045] Figure 4 This is the data augmentation method used in this implementation;

[0046] Figure 5 These are the original ultrasonic point cloud observation results of this embodiment, where (a) is the original ultrasonic point cloud observation results and (b) is the ultrasonic point cloud observation results after rotation.

[0047] Figure 6 This is a diagram of the 1DCNN network structure in this embodiment;

[0048] Figure 7 This is a diagram of the LSC-YOLO network structure in this implementation method;

[0049] Figure 8 This is a visualization of the algorithm flowchart and its effects in this implementation method;

[0050] Figure 9 The above are the actual detection results of the improved YOLOv5 in this embodiment; (a) is the detection result of the ultrasonic probe of YOLOv5 before the improvement, (b) is the detection result of the ultrasonic probe of YOLOv5 after CBAM fusion, (c) is the detection result of the ultrasonic probe of the improved YOLOv5 at close range, and (d) is the detection result of the ultrasonic probe of the improved YOLOv5 at long range.

[0051] Figure 10 These are the training loss and accuracy curves of 1DCNN; where (a) is the loss curve during training and (b) is the accuracy curve during training.

[0052] Figure 11 It is the confusion matrix of the 1DCNN classification model;

[0053] Figure 12 This is a diagram showing the effect of point cloud coloring observation;

[0054] Figure 13 This is a flowchart of the augmented reality visualization system workflow. Detailed Implementation

[0055] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] This embodiment provides a method for ultrasonic detection of composite material damage based on deep learning and augmented reality; firstly, a system is built as follows: Figure 1 The ultrasonic testing and visualization platform shown; such as Figure 2The platform utilizes a robotic arm to hold an ultrasonic probe and perform reciprocating scans on the test piece. The ultrasonic detection signal is transmitted via an ultrasonic acquisition card to a computer for post-processing, becoming an ultrasonic A-scan signal. A binocular camera is positioned to the left of the ultrasonic probe, and a stereo matching algorithm based on binocular vision is used to determine the probe's position. Aligning the position information with the ultrasonic signal in time yields an ultrasonic point cloud. This point cloud image is then uploaded to a portable computing unit paired with AR glasses via a network transmission protocol. Finally, the point cloud is projected onto the ultrasonic test piece using Apriltag positioning codes for visualization.

[0057] In addition, this implementation optimizes and improves the YOLOv8 model: by introducing an edge convolution module, the edge detection accuracy is improved; the StarNet architecture is embedded to reduce the number of parameters and enhance the real-time inference performance of the model; and a lightweight detection head LSCS is adopted to further reduce the number of model parameters and computational complexity.

[0058] In some specific implementations, the ultrasonic detection method for composite material damage based on deep learning and augmented reality is implemented according to the following steps:

[0059] S1. Testing Platform Setup:

[0060] like Figure 1 The detection platform described in this embodiment includes a robotic arm, an optical platform, a binocular camera, a probe holder, an ultrasonic single-crystal probe, an ultrasonic acquisition card, a composite material plate, a composite material plate fixing fixture, AR glasses, a portable computing unit Seerpad, an Apriltag positioning code, and a portable processing terminal.

[0061] After the robotic arm is equipped with the probe fixture, it holds the ultrasonic single-crystal probe and performs scanning operations on the composite material test piece along a preset scanning path. The optical platform is used to carry various experimental instruments and works in conjunction with the composite material plate fixing fixture to fix the composite material plate. The binocular camera is used to collect visual information from the ultrasonic probe and transmits the collected visual information to a portable processing terminal via a data cable. The ultrasonic single-crystal probe is used to collect ultrasonic signals, which are transmitted to the portable processing terminal via an ultrasonic acquisition card. The portable processing terminal is used to run the probe detection target detection model and a stereo matching algorithm that converts visual information into positional information. The portable computing unit Seerpad, which is paired with AR glasses, is used to visualize the ultrasonic point cloud. The Apriltag positioning code is used to provide a spatial coordinate system.

[0062] S2. Acquiring ultrasound data using visual methods:

[0063] A robotic arm carrying an ultrasonic A-scan probe scans the specimen to obtain ultrasonic A-scan signals. Simultaneously, an ultrasonic probe target detection algorithm runs on a computer to track the probe's real-time position. The core loss function of this ultrasonic probe target detection algorithm is:

[0064]

[0065] Where IoU represents the degree of overlap. Center of the prediction box Center of the real frame The distance between them The preset distance scale parameters, It is a trade-off coefficient. It is a measure of the aspect ratio difference between the predicted bounding box and the ground truth bounding box.

[0066] The core loss function of the ultrasonic probe target detection algorithm described in this embodiment describes the bounding box regression more accurately than traditional IoU and GIoU. After YOLOv5 detects the target ultrasonic probe, it generates a target detection box. The stereo camera uses a stereo matching algorithm to determine the position information of the pixels corresponding to the four corner points of the detection box, thus obtaining the position information of the ultrasonic probe. The core of the stereo matching algorithm is to calculate the disparity, and the formula for calculating the disparity is:

[0067]

[0068] in, yes Corresponding disparity candidate pixel The cost of adjacent disparity aggregation, To achieve the optimal disparity value, To achieve the optimal disparity value, Image pixel coordinates, To optimize the operator, This represents the maximum disparity.

[0069] After obtaining the ultrasonic signal and position information, the position information is fused with the post-processed ultrasonic signal to form ultrasonic data with coordinate information, and the internal condition of the material is displayed intuitively using point cloud imaging.

[0070] Specifically, the post-processing of the ultrasonic signal involves filtering the ultrasonic signal to ensure that the detection results are not interfered with by noise. This invention employs the Kalman filtering algorithm:

[0071]

[0072] in, For Kalman gain, This is the actual measured value. For the first Prior state estimation at time 10:00 For the first Posterior state estimation at time 10:00. For the first The observation matrix at each time point.

[0073] Specifically, the fusion of positional information is achieved by aligning and standardizing the length of ultrasound signals. The specific method is as follows: First, all ultrasound signals are adjusted to be uniformly 550 data points, and the positional deviation between signals is reduced by aligning feature points. Then, the second extreme point of the signal is found. Taking the second extreme point as the starting point, the signal part before the point is truncated, so that the key feature positions of all signals are aligned.

[0074] Given the limited size of the acquired ultrasound probe dataset, and to enhance its robustness by encompassing richer sample features, this embodiment employs data augmentation techniques to expand the dataset, thereby increasing the quantity and diversity of samples. Ultimately, the initial dataset size is expanded to twice its original size. Figure 4 This shows some specific data augmentation methods. Figure 5 (a) and Figure 5 (b) Demonstrates the observation effect after rotation enhancement; the optional data augmentation methods in this embodiment include rotation, scaling, cropping, and color adjustment. During the data augmentation process, the ultrasound probes in the enhanced images are manually labeled using the labeling software LabelImg; to ensure data consistency and compatibility, this embodiment converts all labeling results into YOLO format txt files to adapt to subsequent model training and data processing needs.

[0075] In some specific embodiments, the following are adopted: Figure 3 The improved YOLOv5 model, incorporating the CBAM attention mechanism, is used as a detection model for an ultrasound probe. When directly embedding the CBAM attention module into the backbone of the YOLOv5 network, structural compatibility issues arise between the CBAM attention module and the existing CSP module in the backbone, leading to model training instability and less-than-expected improvement in detection accuracy. This embodiment replaces the existing CSP module in the backbone with the CBAM attention module and introduces the NWD loss function to optimize the model training process, effectively solving the aforementioned structural compatibility problem and significantly improving the model's detection performance. Specifically, in this embodiment, the CBAM attention module is introduced into the backbone of the YOLOv5 network, replacing the existing CSP module. Furthermore, the NWD loss function is introduced into the model to achieve better detection performance. The CBAM attention mechanism is as follows:

[0076]

[0077] in, For global average pooling, Global max pooling, For channel attention mechanism on feature map The weighted output, It is a multilayer perceptron. The sigmoid activation function is used. The YOLOv5 model, which incorporates the CBAM attention mechanism, takes the ultrasound probe image data as input and outputs the ultrasound probe's bounding box position, category, and confidence score within the image. The improved YOLOv5 model achieves significantly enhanced detection performance, such as... Figure 9 (a) ~ Figure 9 (d) In this embodiment, the detection accuracy of the ultrasonic probe is increased from the original 0.83 to 0.97.

[0078] S3. Ultrasonic point cloud enhancement based on one-dimensional convolutional neural network (1DCNN)

[0079] After completing the acquisition and construction of the internal damage point cloud of the composite material plate in the aforementioned steps, the point cloud data constructed in this embodiment uses the numerical gradient of the intensity signal as the color mapping reference. Different gradients of color are applied to the point clouds of defective and non-defective regions to achieve differentiation. However, actual observation revealed that although there is a certain color difference, the distinction between the point cloud at the defect and the point clouds on the upper and lower surfaces is still low: it is difficult to observe defects from the front, and small defects cannot be directly identified. Therefore, this embodiment requires further processing to differentiate the defective point cloud from the point clouds on the upper and lower surfaces. This embodiment introduces a one-dimensional convolutional neural network to classify the defect signal and the non-defect signal; such as... Figure 12 In this implementation, the signals are marked synchronously during the data acquisition stage. During subsequent signal processing, red RGB color information is added to signals that are determined to be defects, and yellow RGB color information is assigned to non-defect signals to enhance the visual distinction between defective and non-defective areas.

[0080] An ultrasound A-scan consists of a one-dimensional array of signal amplitudes, which cannot be learned using common image processing neural networks. This implementation uses a one-dimensional convolutional neural network to perform defect detection on the ultrasound A-scan signal; its network structure diagram is shown below. Figure 6As shown. Changes in ultrasound signals are generally controlled within a single dimension; therefore, this network employs a stepwise extraction approach to process the signal. The network has three feature extraction modules, each consisting of a Conv1D (one-dimensional convolutional layer), a Batch Normalization (BN) layer, and a Maxpooling layer. Following the feature extraction modules is a classifier structure used to discriminate the ultrasound signal. Specifically, the input is the original ultrasound signal, with a data dimension of 550×1, indicating that a set of ultrasound signals acquired is a one-dimensional vector of length 550. The ultrasound signal is then fed into the first feature extraction module, first undergoing feature extraction through a one-dimensional convolutional layer. Conv1D(512, 32) represents a one-dimensional convolutional layer with 512 channels and a kernel size of 32, expressed as:

[0081]

[0082] in, For input, It has 32 convolution kernels. This is the output of a one-dimensional convolutional layer. This is the index of the input channel. Here, is the index of the output channel, and t is the index of the position within the convolution kernel. This refers to the location index of the output feature map. The convolution bias corresponding to the output channel. This is the Sigmoid activation function.

[0083] The signal is then fed into the batch normalization layer. BN(512, 32) maintains the same dimensions as the convolutional layer. This layer is used to speed up training and improve model stability. The batch normalization layer is as follows:

[0084]

[0085] in, For input, For output, is the input to the batch normalization layer, and n is the sample index of the batch processing dimension. For channel The corresponding scaling factor, where l is the spatial dimension index. For channel The corresponding within-batch feature mean, For channel The corresponding offset factor, For channel The corresponding within-batch feature variance, This is the minimum value. Finally, the signal is fed into a max-pooling layer. Maxpooling(256, 32) means halving the spatial dimension to 256 while keeping the number of channels unchanged at 32. The purpose of this layer is to reduce computational complexity and extract the most salient features. The subsequent two feature extraction modules have the same function, but increase the number of channels to extract more advanced features. Specifically:

[0086]

[0087] in, The pooling kernel size, For channel indexing, is the step size, and g is the position index inside the pooling kernel. The signal after feature extraction enters the classifier for classification. Flatten (16384) is a flattening layer used to convert the input multidimensional feature map into a one-dimensional vector with a dimension of 16384, and also serves as a connection between the convolutional network and the fully connected layer. The one-dimensional vector then enters the fully connected layer. Linear (64) reduces the feature dimension to 64 and is used to non-linearly combine high-level features. The following batch normalization layer BN (64) is applied to the output of the fully connected layer. The next Linear (2) outputs two neurons, each corresponding to a category in the binary classification task. BN (2) is applied to the features before the final prediction. The final SoftMax (2) is the softmax activation function, which converts the output into a probability distribution function. The sum of the two output values ​​is 1, which can perform binary classification prediction faster. In this implementation, the ultrasound signal is divided into two categories: one is normal signal, which is represented by the number "0" and the other is defect data, which is represented by the number "1". Figure 10 As shown, this figure contains two sub-figures: Figure 10 (a) shows the loss change curve during the training process (blue is the training loss train_loss, orange is the test loss test_loss). It can be seen that as the number of training iterations increases, the loss decreases rapidly and gradually stabilizes. Figure 10 (b) shows the accuracy change curve during the training process (blue represents training accuracy train_acc, orange represents test accuracy test_acc). It can be seen that the accuracy gradually increases with the number of iterations, and the final test accuracy is close to 0.95, indicating that the model training effect is good. Figure 11As shown, the matrix rows represent the true categories and the columns represent the predicted categories, with color intensity corresponding to the predicted probabilities (the redder the color, the higher the probability). Specifically, the probability of the true category 0 being correctly predicted as 0 is 0.98, and the probability of being incorrectly predicted as 1 is 0.02; the probability of the true category 1 being correctly predicted as 1 is 0.99, and the probability of being incorrectly predicted as 0 is 0.01, indicating that the model has high classification accuracy for both categories.

[0088] S4. Extracting features from composite material test specimens using segmentation models.

[0089] In this embodiment, to enable the ultrasonic point cloud results to be displayed on AR glasses, the relative position of the composite material test piece in space needs to be obtained. Identifying the test piece is a prerequisite for determining its relative position. Since the test piece is typically rectangular with significant corner and edge features, a segmentation model is used to identify it. The visualization algorithm flow and effects of this embodiment are as follows: Figure 8 As shown.

[0090] To address the shortcomings of the YOLOv8 model in terms of detection accuracy, this implementation method further designs, such as... Figure 7 The lightweight edge convolutional network LSC-YOLO (Lightweight Sobel Convolutional Networks-YOLO) structure shown is used to improve detection performance.

[0091] This implementation improves the three main parts of the network model—Backbone, Neck, and Head—using different methods to achieve the desired effect for the scenario. Specifically, it enhances the C2f module in the Backbone by using a Sobel Convolution Module (SCM) to improve edge detection accuracy. The SCM module employs a Sobel Convolution branch and a CNN double convolution pattern to strengthen edge information extraction during feature extraction, while also retaining some spatial information learning capability. The input at this stage is an image, and pixel values ​​are converted through a convolution factor G. X and G Y Edge operations are performed horizontally and vertically separately. The gradient values ​​in both directions are added together to output the final image value. The root of the sum of the squares of the two gradient values ​​is taken as the final superimposed output value. Where G is the fused edge feature output by the SobelConv branch, G X G represents the horizontal edge gradient feature. Y This represents the vertical edge gradient feature.

[0092] Meanwhile, introducing new modules increases the number of model parameters, necessitating lightweighting of the remaining structure of the modules to ensure that the model's real-time detection performance is not significantly affected. Therefore, this invention introduces StarNet from CVPR 2024 into the Neck section to create a lightweight C2f network module, C2f-star. The introduction of C2f-star can improve the learning ability of high-dimensional nonlinear features and enhance the model's expressive power while maintaining the same network width, and can also reduce the number of parameters to improve the model's real-time performance. This implementation introduces star operations to replace Bottleneck, a simplified element-wise multiplication method for nonlinear learning and fusion between different features. The C2f-star module's workflow is as follows: the input feature map first undergoes a convolution for feature extraction, then the Split module divides the feature map into two parts: one part is integrated into a deep network for feature extraction, and the other part is fused with deep features to form a feature map containing multi-level feature information.

[0093]

[0094] Here, StarBlock is the star operation, and X=[X_main,X_side] divides the input feature map into two parts: X_main is the main branch feature map, and X_side is the side branch feature map. Z is a 1×1 convolutional layer, and Z is the output feature map obtained by splicing and fusing after branching and feature transformation, which contains multi-level feature information.

[0095] In the head section, this implementation employs a lightweight detection head, LSCS (Lightweight Shared Convolutional Segmentation), to further reduce the number of parameters and computational cost. This detection head uses a shared convolution method to optimize resource consumption during detection, minimizing accuracy loss while requiring fewer parameters and less computation.

[0096]

[0097] Where S is the output feature map after multi-scale fusion, and Concat is the feature concatenation operation. It is a 3×3 convolutional layer. There are n branch feature maps of different scales, GlobalAvgPool is the global average pooling operation, and F is the base input feature map to be fused.

[0098] Compared to the previous Head part, a shared weight W is added, which enables the model to obtain stronger feature representation capabilities and multi-scale adaptability with almost no increase in computation.

[0099] As shown in Table 1, the ablation experiment results show that the model improvement effect is significant, with mAP50 reaching 0.995, while the processing speed is moderate. Overall, the model improvement has met expectations.

[0100] Table 1 Ablation Experiment Results

[0101]

[0102] S5 and Apriltag code positioning and AR glasses imaging

[0103] like Figure 1 As shown in the image, the lower right corner displays the Apriltag location code. Apriltag is a typical vision-based localization method. The tags are usually composed of black and white squares, and their outer contour typically resembles a quadrilateral. It is widely used in robot navigation, augmented reality, and object recognition. The core principle of this method is to determine the position and orientation of a target object in three-dimensional space by detecting and recognizing specific QR code shape markers.

[0104] When identifying Apriltags, a spatial coordinate system is obtained. By calculating the relationship between the pixel coordinates of the four corner points and the coordinates of the Apriltag origin, the spatial mapping of the four corner points is performed to obtain their order. Regardless of the viewing angle, the filled area maintains accurate orientation, ensuring the correctness of the visualization results. After completing the vectorization extraction of the detection area, the next step is to fill the real-time generated ultrasonic point cloud image. The image reading interval and method adopts an equal-interval cyclic reading of files at fixed addresses to update the damage visualization results. Image fusion uses a perspective transformation method to spatially transform the ultrasonic point cloud image of the composite material and project the imaging results onto the identified filled area. Finally, the same steps are repeated for each extracted image to obtain a continuous visualization effect.

[0105] like Figure 13 As shown, the working process of the augmented reality visualization system disclosed in this embodiment is as follows: First, the composite material test piece is detachably fixed to the experimental platform using a composite material plate fixing fixture. The composite material plate fixing fixture can achieve stable positioning of the composite material test piece, avoiding displacement of the test piece during subsequent scanning and thus affecting the detection accuracy.

[0106] Subsequently, the robotic arm is activated. The end effector of the robotic arm is rigidly connected to the ultrasonic probe through a probe holder. Driven by a preset control program, the robotic arm moves the ultrasonic probe to perform a uniform "S"-shaped scan within the detection area of ​​the composite material test piece. During the scan, the ultrasonic probe continuously acquires ultrasonic detection signals from the composite material test piece. These ultrasonic detection signals are then conditioned (including filtering, amplification, and analog-to-digital conversion) by an ultrasonic acquisition card and transmitted to a computer for signal post-processing. The post-processed ultrasonic A-scan signal is displayed in real time on the computer's display terminal so that the operator can observe the detection signal status in real time.

[0107] To achieve precise positioning of the ultrasound probe, a binocular camera is fixedly installed at a preset position on the left side of the ultrasound probe. The binocular camera establishes a communication connection with the computer and starts synchronously. The computer runs a preset binocular stereo matching algorithm to perform feature extraction, matching, and three-dimensional reconstruction calculation on the binocular images containing the ultrasound probe acquired by the binocular camera. This enables precise detection of the real-time position information of the ultrasound probe during scanning and stores the detected position information in the computer's storage unit in real time.

[0108] After the robotic arm completes the preset "S"-shaped scanning trajectory, the portable processing terminal automatically starts the position information and ultrasonic signal fusion program. The fusion program calls the ultrasonic detection signal and the corresponding ultrasonic probe position information stored in the storage unit, and achieves precise alignment of the ultrasonic signal and position information through the coordinate calibration algorithm, generating an ultrasonic point cloud image containing spatial position information and ultrasonic detection data. The ultrasonic point cloud image can intuitively reflect the defect distribution and position characteristics inside the composite material test piece.

[0109] Finally, the portable processing terminal establishes a communication connection with the Seerpad portable computing unit mounted on the AR glasses via the HTTP protocol, transmitting the generated ultrasonic point cloud image to the Seerpad portable computing unit in the form of data frames for caching. After the operator wears the AR glasses, they control the image acquisition module of the AR glasses to scan the Apriltag positioning codes preset around the experimental platform or composite material test piece. The AR glasses use the built-in visual positioning algorithm to identify and solve the Apriltag positioning codes, obtaining the three-dimensional coordinates in the world coordinate system corresponding to the Apriltag positioning codes. The Seerpad portable computing unit calls the cached ultrasonic point cloud image, combines it with the solved three-dimensional coordinates, and uses the augmented reality rendering algorithm to accurately project the ultrasonic point cloud image to the corresponding three-dimensional coordinate position, realizing the fusion visualization of ultrasonic test data and real scene. At this point, the entire workflow of the augmented reality visualization system is completed.

[0110] This invention utilizes a binocular camera to extract ultrasonic probe position information, replacing the traditional method of acquiring position information using a position encoder. Compared to traditional methods, this significantly reduces the cost and improves the efficiency of ultrasonic testing.

[0111] This invention uses a one-dimensional convolutional neural network to intelligently judge ultrasound A-scan signals and presents them through visual fusion. It innovates the method for generating ultrasound point clouds, improving their readability.

[0112] By projecting the ultrasonic point cloud results onto AR glasses and using augmented reality to fuse the test results with the composite material test piece, the specific location of the damage can be seen intuitively, making it easier for the testers to interpret the results and greatly improving the efficiency of ultrasonic testing.

[0113] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for composite damage ultrasonic detection based on deep learning and augmented reality, characterized in that, It includes the following steps: S1. Build a detection platform; synchronously collect visual images of the ultrasonic probe and the composite material to be tested through the binocular cameras in the platform; S2. Use the robotic arm to carry the ultrasonic single crystal probe to scan the composite material plate to be tested and obtain the ultrasonic A-scan signal; use the ultrasonic probe target detection algorithm, combine with the stereo matching algorithm to obtain the position information of the ultrasonic probe; post-process the ultrasonic A-scan signal, and fuse the position information with the post-processed ultrasonic A-scan signal to form ultrasonic data with coordinate information, which is presented in the form of point cloud imaging; perform data enhancement processing and annotation conversion on the ultrasonic probe data set; S3. Use a one-dimensional convolutional neural network to classify the defect signals and non-defect signals in the ultrasonic A-scan signal obtained and post-processed in S2, and assign different RGB color information to the classified signals to enhance the visual distinguishability between the defect and non-defect regions; S4. Adopt a segmentation model to extract the features of the composite material to be tested: based on the visual image containing the composite material to be tested collected by the binocular cameras in the detection platform, identify the composite material to be tested itself through the lightweight edge convolution LSC-YOLO network to obtain its relative position in space; S5. Scan the Apriltag positioning code through the AR glasses to obtain a three-dimensional coordinate system, transmit the ultrasonic point cloud image processed in S3 to the portable computing unit through the network transmission protocol, use the perspective transformation method to perform spatial transformation on the ultrasonic point cloud image, and project the imaging result to the corresponding three-dimensional coordinate area of the component to be tested to achieve damage visualization.

2. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, In the step S1, the detection platform includes a robotic arm, an optical platform, binocular cameras, a probe fixture, an ultrasonic single crystal probe, an ultrasonic acquisition card, a composite material plate, a composite material plate fixing fixture, an AR glasses, a supporting portable computing unit Seerpad, an Apriltag positioning code, and a portable processing terminal; After the robotic arm is assembled with the probe fixture, it holds the ultrasonic single crystal probe and performs a scanning operation on the composite material to be tested along a preset scanning path; the optical platform is used to carry experimental equipment and cooperate with the composite material plate fixing fixture to fix the composite material plate; the binocular cameras are used to collect the visual information of the ultrasonic probe and transmit it to the portable processing terminal through a data cable; the ultrasonic signal collected by the ultrasonic single crystal probe is transmitted to the portable processing terminal through the ultrasonic acquisition card; the portable processing terminal is used to run the ultrasonic probe target detection algorithm and the stereo matching algorithm; the portable computing unit supporting the AR glasses is Seerpad, which is used to realize the visualization of the ultrasonic point cloud; the Apriltag positioning code is used to provide a spatial coordinate system.

3. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, In the step S2, the core loss function of the ultrasonic probe target detection algorithm is: ； Where IoU represents the degree of overlap. Center of the prediction box Center of the real frame The distance between them The preset distance scale parameters, It is a trade-off coefficient. It is a measure of the aspect ratio difference between the predicted bounding box and the ground truth bounding box.

4. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, In the step S2, the method for determining the disparity of the stereo matching algorithm: ； in, yes Corresponding disparity candidate pixel The cost of adjacent disparity aggregation, To achieve the optimal disparity value, To achieve the optimal disparity value, Image pixel coordinates, To optimize the operator, This represents the maximum disparity.

5. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, In the step S2, the post-processing of the ultrasonic signal adopts the Kalman filtering algorithm, specifically: ； in, For Kalman gain, This is the actual measured value. For the first Prior state estimation at time 10:00 For the first Posterior state estimation at time 10:

00. For the first The observation matrix at each time point.

6. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, The ultrasonic probe target detection algorithm in step S2 uses an improved YOLOv5 model that integrates the CBAM attention mechanism. This model replaces the original CSP module in the YOLOv5s network backbone with a CBAM attention module and introduces an NWD loss function to optimize model training. The channel attention of the CBAM attention module... Specifically: ； in, For global average pooling, Global max pooling, For channel attention mechanism on feature map The weighted output, It is a multilayer perceptron. This is the Sigmoid activation function.

7. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, The data augmentation process in step S2 includes rotation, scaling, cropping, and color adjustment, which expands the initial dataset size to twice its original size. The annotation conversion involves manually annotating the ultrasound probes in the enhanced images using LabelImg software and converting all annotation results into YOLO format txt files.

8. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, In step S3, the one-dimensional convolutional neural network includes three feature extraction modules and a classifier structure; the feature extraction module consists of a Conv1D layer, a BN layer, and a Maxpooling layer; the input of the one-dimensional convolutional neural network is the original ultrasound signal with a data dimension of 550×1; the classifier structure includes a Flatten layer, a Linear layer, a BN layer, and a SoftMax activation function in sequence, and the output is a binary classification result, in which the classification result of normal signal is 0 and the classification result of defect signal is 1.

9. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, In step S4, the improvements to the LSC-YOLO network include: In the backbone, the C2f module is improved by an edge convolutional module (SCM). The SCM uses a SobelConv branch and a CNN dual convolution mode, and the edge feature fusion method is as follows: Where G is the fused edge feature output by the SobelConv branch, G X G represents the horizontal edge gradient feature. Y Vertical edge gradient features; In the Neck section, the StarNet architecture is introduced to lightweight the C2f module, resulting in the C2f-star module. The input feature map to the C2f-star module is first processed through a convolution for feature extraction. Then, the Split module divides the feature map into two parts: one part is integrated into a deeper network for further feature extraction, and the other part is fused with the deeper features to form a feature map containing multi-level feature information. ； StarBlock is the star operation. Z is a 1×1 convolutional layer, which is the output feature map obtained by splicing and fusing after branching and feature transformation, containing multi-level feature information. X_main is the main branch feature map; X_side is the side branch feature map. In the Head section, a lightweight detection head LSCS is used, and the feature fusion method is as follows: ； Where S is the output feature map after multi-scale fusion, and Concat is the feature concatenation operation. It is a 3×3 convolutional layer. There are n branch feature maps of different scales, GlobalAvgPool is the global average pooling operation, and F is the base input feature map to be fused.

10. The ultrasonic detection method for composite material damage based on deep learning and augmented reality according to claim 1, characterized in that, In step S5, the network transmission protocol is the HTTP protocol; the image fusion uses the perspective transformation method to perform spatial transformation on the ultrasound point cloud image; the ultrasound point cloud image updates the damage visualization result by reading fixed address files at equal intervals in a loop, and repeats the above processing for each frame of the image to obtain a continuous visualization effect.

Images