Sound barrier surface defect evaluation method and system and computer

Abstract

The invention relates to the technical field of sound barrier detection, and provides a sound barrier surface defect evaluation method and system and a computer, and the sound barrier surface defect evaluation method comprises the following steps: collecting an initial multi-modal image set; performing image de-noising processing on the initial multi-modal image set to obtain a de-noised multi-modal image set; performing image enhancement processing on the de-noised multi-modal image set to obtain an enhanced multi-modal image set; obtaining a registration multi-modal image set, and cutting the registration multi-modal image set into a cutting multi-modal image set; on the basis of cutting the multi-modal image set, extracting multi-modal features and fusing the multi-modal features to obtain a multi-modal feature image; and predicting the multi-modal feature map to obtain a plurality of defect types, a plurality of defect positions and a plurality of confidence coefficients, and evaluating defect levels. By adopting the method, the defect identification precision is improved, and the defects on the surface of the sound barrier are comprehensively evaluated from multiple dimensions.

Description

Technical Field

[0001] This invention relates to the field of sound barrier detection technology, and in particular to a method, system and computer for evaluating surface defects in sound barriers. Background Technology

[0002] Noise barriers, as important facilities for controlling traffic noise, are widely used in highways, railways, and rail transit. However, long-term exposure to the complex outdoor environment makes them prone to various defects, such as cracks, dents, corrosion, and coating peeling. These defects not only affect the aesthetics of the noise barriers but also reduce their structural strength and noise reduction performance, and may even pose safety hazards. Therefore, timely and accurate detection of surface defects in noise barriers is crucial for ensuring their normal operation and extending their service life.

[0003] Currently, the detection of surface defects in sound barriers mainly relies on manual inspection and image processing technology. Manual inspection involves inspectors visually examining the sound barrier surface or using simple tools. However, manual inspection is inefficient, especially for sound barriers with long installation distances, requiring significant manpower, resources, and time. Furthermore, the accuracy of manual inspection is limited by the inspector's experience, sense of responsibility, and fatigue level, easily leading to missed or false detections. In addition, manual inspection poses safety risks for sound barriers located at high altitudes or in dangerous positions. Therefore, image processing-based methods for sound barrier defect detection are gradually emerging.

[0004] Existing image processing-based methods for detecting sound barrier defects typically use a single type of image for defect identification. However, sound barrier surface defects exhibit diverse morphologies, and under varying lighting and weather conditions, a single modal image often fails to comprehensively and clearly reveal the defect features. For example, in backlit environments, visible light images are prone to casting shadows, obscuring some defects; while at night, the quality of visible light images deteriorates significantly, making defects difficult to identify. Furthermore, most existing image processing-based methods rely solely on traditional feature extraction algorithms, resulting in low accuracy and weak generalization ability for identifying subtle defects against complex backgrounds. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a method, system, and computer for assessing surface defects of sound barriers. This invention combines multimodal images, performing various image processing steps and extracting fused features to achieve a more comprehensive and accurate identification of surface defects in sound barriers. The present invention aims to solve the technical problems of existing technologies that rely on feature extraction from single-modal images of sound barrier surface defects, resulting in insufficient recognition accuracy and poor versatility.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution: A method for assessing surface defects in a sound barrier includes the following steps: Initial visible light images, initial infrared thermal images, and initial ultrasonic images of the sound barrier surface were acquired to form an initial multimodal atlas. The initial multimodal image set is subjected to image denoising processing to obtain a denoised multimodal image set; The denoised multimodal image set is subjected to image enhancement processing to obtain an enhanced multimodal image set; Image registration is performed on the enhanced multimodal atlas to obtain a registered multimodal atlas, and the registered multimodal atlas is then cropped into a cropped multimodal atlas. Based on the cropped multimodal graph set, multimodal features are extracted and fused to obtain a multimodal feature map; The multimodal feature map is predicted to obtain several defect types, several defect locations and several confidence levels. Based on the several defect types, several defect locations and several confidence levels, the defect level is evaluated.

[0007] Furthermore, the steps of acquiring the initial visible light image, initial infrared thermal image, and initial ultrasonic image of the sound barrier surface include: An initial visible light image of the sound barrier surface is acquired vertically using a camera, an initial infrared thermal image of the sound barrier surface is acquired using an infrared thermal imager, and an initial ultrasonic image of the sound barrier surface is acquired based on ultrasonic waves with a frequency of 1MHz to 10MHz.

[0008] Furthermore, the denoised multimodal image set includes denoised visible light images, denoised infrared thermal imaging images, and denoised ultrasound images. The step of performing image denoising processing on the initial multimodal image set to obtain the denoised multimodal image set includes: Remove Gaussian noise from the initial visible light image and smooth the image edges to obtain the denoised visible light image; The pulse noise in the initial infrared thermal imaging image is removed by median filtering to obtain the denoised infrared thermal imaging image; The initial ultrasound image is decomposed by wavelet to remove noise components and then reconstructed by wavelet to obtain the denoised ultrasound image. The denoised visible light image, the denoised infrared thermal imaging image, and the denoised ultrasound image are combined into a denoised multimodal image set.

[0009] Furthermore, the enhanced multimodal image set includes enhanced visible light images, enhanced infrared thermal imaging images, and enhanced ultrasound images. The step of performing image enhancement processing on the denoised multimodal image set to obtain the enhanced multimodal image set includes: The denoised visible light image is converted to YCrCb format to obtain a first transition visible light image. The brightness channel Y of the first transition visible light image is processed by a histogram equalization algorithm to obtain a second transition visible light image. The second transition visible light image is converted to RGB format to obtain a third transition visible light image. The enhanced infrared thermal imaging image is obtained by stretching the grayscale distribution of the denoised infrared thermal imaging image using a histogram equalization algorithm. Based on several initial gray values ​​of the denoised ultrasound image, an initial gray range is obtained, a target gray range is set based on the initial gray range, and several initial gray values ​​are linearly mapped to the target gray range to obtain several target gray values ​​to form a transition ultrasound image. A sharpening filter is constructed based on the Laplacian operator, and the high-frequency components of the third transition visible light image and the transition ultrasound image are enhanced by the sharpening filter to obtain the enhanced visible light image and the enhanced ultrasound image.

[0010] Furthermore, the step of performing image registration on the enhanced multimodal atlas to obtain a registered multimodal atlas includes: Several feature points are extracted from the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image, respectively. A matching algorithm is used to perform feature matching on the several feature points of the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image to obtain several matching point pairs. Based on several matching point pairs, several transformation matrices are calculated, and image registration is performed on the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image based on the several transformation matrices to obtain the visible light image, the infrared thermal imaging image, and the ultrasound image to be evaluated, so as to form a multimodal image set to be evaluated. Set a root mean square error threshold and a peak signal-to-noise ratio (PSNR) threshold. Based on the enhanced multimodal image set and the multimodal image set to be evaluated, calculate several root mean square errors and several PSNRs. Based on several root mean square errors and the root mean square error threshold, determine whether the multimodal image set to be evaluated meets the registration accuracy. Based on several PSNRs and the PSNR threshold, determine whether the multimodal image set to be evaluated meets the image quality. If the multimodal image set to be evaluated meets the registration accuracy and image quality requirements, then the multimodal image set to be evaluated is established as a registered multimodal image set.

[0011] Furthermore, the registration multimodal atlas includes registered visible light images, registered infrared thermal imaging images, and registered ultrasound images, and the step of cropping the registration multimodal atlas into a cropped multimodal atlas includes: The positions of the sound barriers in the registered visible light image, the registered infrared thermal image, and the registered ultrasound image are determined respectively, and the registered visible light image, the registered infrared thermal image, and the registered ultrasound image are cropped according to several sound barrier positions to obtain a cropped multimodal atlas.

[0012] Furthermore, the cropped multimodal image set includes cropping visible light images, cropping infrared thermal imaging images, and cropping ultrasound images. The step of extracting and fusing multimodal features based on the cropped multimodal image set to obtain a multimodal feature map includes: Obtain the first neural network, the second neural network, and the third neural network; Based on the first neural network, feature extraction is performed on the cropped visible light image, and residual connections are made to obtain image semantic features; Based on the second neural network, feature extraction is performed on the cropped infrared thermal imaging image to obtain temperature distribution features and abnormal region features. Based on the third neural network, feature extraction is performed on the cropped ultrasound image to obtain defect features. The image semantic features, temperature distribution features, abnormal region features, and defect features constitute multimodal features. The multimodal features are fused using a channel attention mechanism to obtain a multimodal feature map.

[0013] Furthermore, the step of predicting the multimodal feature map to obtain several defect types, several defect locations, and several confidence levels, and evaluating the defect level based on the several defect types, several defect locations, and several confidence levels, includes: The multimodal feature map is predicted using an object detection network or a fully connected network. Several defect types are obtained based on the cross-entropy loss function, and several defect locations and several confidence levels are obtained based on the mean square error loss function. Several defect area algorithms are determined based on several defect types, and a defect area index is obtained based on several defect area algorithms, several defect locations, and several confidence levels. The sound barrier thickness is obtained by calculating the defect depth index based on several defect types, several defect locations, several confidence levels, the clipped multimodal atlas, and the sound barrier thickness. Based on several defect locations and several confidence levels, the number of defects per unit area is determined, and a defect quantity index is obtained based on the number of defects per unit area. The key locations of the sound barrier are obtained, and based on the key locations of the sound barrier, several defect locations, and several confidence levels, a defect distribution index is obtained; The defect level is assessed based on the defect area index, the defect depth index, the defect quantity index, and the defect distribution index.

[0014] A sound barrier surface defect assessment system, employing the sound barrier surface defect assessment method as described in the above technical solution, the system comprising: The acquisition module is used to acquire initial visible light images, initial infrared thermal images, and initial ultrasonic images of the sound barrier surface to form an initial multimodal atlas. A denoising module is used to perform image denoising processing on the initial multimodal image set to obtain a denoised multimodal image set; An enhancement module is used to perform image enhancement processing on the denoised multimodal image set to obtain an enhanced multimodal image set; The registration and cropping module is used to perform image registration on the enhanced multimodal image set to obtain a registered multimodal image set, and to crop the registered multimodal image set into a cropped multimodal image set. The fusion module is used to extract and fuse multimodal features based on the cropped multimodal map set to obtain a multimodal feature map; An evaluation module is used to predict the multimodal feature map to obtain several defect types, several defect locations and several confidence levels, and to evaluate the defect level based on the several defect types, several defect locations and several confidence levels.

[0015] A computer includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the sound barrier surface defect assessment method as described above.

[0016] Compared with existing technologies, the advantages of this invention are as follows: By acquiring the initial multimodal image set, it is beneficial to comprehensively and clearly display the defect features of the sound barrier surface, avoiding the problem of difficulty in identifying defects in a single modal image due to changes in lighting and weather; By selectively denoising and enhancing visible light images, infrared thermal imaging images, and ultrasonic images respectively, the quality of multimodal images is improved, the detail information of the images is enhanced, and the defect features are made more obvious, which is beneficial to increasing the accuracy of defect identification; By registering and cropping the multimodal images, the effective area of ​​the sound barrier in the image is preserved, which is beneficial to reducing the amount of data for subsequent processing and improving the efficiency and speed of identifying defects on the sound barrier surface; By using different types of neural networks, considering the properties of images of different modalities, features with different preferences are extracted in a targeted manner, and feature fusion with an attention mechanism is performed to make full use of the complementary information of images of different modalities and improve the expressive power of features; By analyzing the fused features through a target detection algorithm, multiple information about defects is identified, and the defects on the sound barrier surface are comprehensively evaluated from multiple dimensions such as defect area, defect depth, defect number, and defect distribution. Attached Figure Description

[0017] Figure 1 This is a flowchart of the sound barrier surface defect assessment method in the first embodiment of the present invention; Figure 2 This is a structural block diagram of the sound barrier surface defect assessment system in the second embodiment of the present invention; The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0018] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0019] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0021] Please see Figure 1 The sound barrier surface defect assessment method in the first embodiment of the present invention includes the following steps: Step S10: Acquire initial visible light images, initial infrared thermal images, and initial ultrasonic images of the sound barrier surface to form an initial multimodal atlas; The surface defects of sound barriers are diverse in form. Under different lighting and weather conditions, single-modal images often fail to fully and clearly display the defect features. For example, in backlit environments, visible light images are prone to producing shadows, obscuring some defects; while at night, the quality of visible light images drops significantly, making defects difficult to identify. Acquiring the initial infrared thermal image and the initial ultrasonic image, combined with the initial visible light image, is beneficial for fully and clearly displaying the defect features of the sound barrier surface, avoiding the problem of difficulty in identifying defects in single-modal images due to changes in lighting and weather.

[0022] Step S10 includes: S110: Acquire an initial visible light image of the sound barrier surface in the vertical direction using a camera, acquire an initial infrared thermal image of the sound barrier surface using an infrared thermal imager, and acquire an initial ultrasonic image of the sound barrier surface based on ultrasonic waves with a frequency of 1MHz to 10MHz.

[0023] Preferably, an industrial-grade high-definition camera is used, mounted on an adjustable-angle bracket, ensuring the camera lens is perpendicular to the sound barrier surface and taken from a distance of 2m to 3m. The infrared thermal imager has a temperature measurement range of -20℃ to 200℃. Direct sunlight and other heat sources should be avoided when installing the infrared thermal imager. The focal length and viewing angle of the infrared thermal imager should be adjusted to cover the detection area on the sound barrier surface. The initial ultrasonic image is acquired using an ultrasonic testing instrument with high-resolution imaging capabilities. The ultrasonic testing instrument generates ultrasonic waves with a frequency of 1MHz to 10MHz to detect defects at different depths within the sound barrier. During ultrasonic testing, an appropriate amount of ultrasonic coupling agent is applied to the surface of the sound barrier to ensure that the ultrasonic waves can effectively penetrate into the sound barrier. The ultrasonic probe of the ultrasonic testing instrument is then moved evenly across the sound barrier surface for a comprehensive scan. Understandably, the high-definition camera can capture the appearance features of the sound barrier surface, such as color and texture, and is suitable for detecting visible surface defects such as cracks and peeling. The initial infrared thermal imaging image, based on the infrared radiation field of the object, can reflect the temperature distribution of the sound barrier surface and can be used to detect temperature anomaly areas caused by internal defects. Ultrasonic detection generates the initial ultrasonic image by emitting and receiving ultrasonic waves and based on the reflection of the ultrasonic waves, which is beneficial for detecting defects such as cavities and delamination in deeper layers of the sound barrier. By acquiring the initial multimodal image set, comprehensive defect information of the sound barrier can be obtained, providing a rich and comprehensive data foundation for subsequent defect detection.

[0024] Step S20: Perform image denoising processing on the initial multimodal image set to obtain a denoised multimodal image set; Understandably, removing noise interference from images helps improve image quality, enhances the identifiability of defects, and thus improves the accuracy of defect detection.

[0025] In step S20, the denoised multimodal image set includes denoised visible light images, denoised infrared thermal imaging images, and denoised ultrasound images. Step S20 includes: S210: Remove Gaussian noise from the initial visible light image and smooth the image edges to obtain the denoised visible light image; Preferably, a 3*3 or 5*5 Gaussian kernel can be used for filtering. Specifically, in this embodiment, a 3*3 Gaussian kernel is used. In the OpenCV library of Python, the function cv2.GaussianBlur() is used, with the ksize parameter set to (3,3) and sigmaX and sigmaY both set to 1. The image pixel values ​​are weighted and averaged to effectively remove Gaussian noise from the image, smooth the image edges, and improve image clarity.

[0026] S220: Remove the pulse noise in the initial infrared thermal imaging image by median filtering to obtain the denoised infrared thermal imaging image; Preferably, the cv2.medianBlur() function from the OpenCV library is used, with the kernel size set to 3 or 5. Specifically, in this embodiment, the kernel size is set to 3, which replaces the gray value of each pixel with the median value of its neighboring pixels, effectively removing impulse noise while preserving the edge and detail information of the image.

[0027] S230: Perform wavelet decomposition on the initial ultrasound image, remove noise components and perform wavelet reconstruction to obtain the denoised ultrasound image, and combine the denoised visible light image, the denoised infrared thermal imaging image and the denoised ultrasound image into a denoised multimodal image set.

[0028] Preferably, in the PyWavelets library of Python, wavelet basis functions are selected to perform wavelet decomposition on the initial ultrasound image, thresholding of high-frequency coefficients at different scales to remove noise components, and then wavelet reconstruction is performed to recover the denoised image, thereby obtaining the denoised ultrasound image.

[0029] Step S30: Perform image enhancement processing on the denoised multimodal image set to obtain an enhanced multimodal image set; Understandably, image enhancement processing is beneficial for enhancing the contrast and detail of an image, making defect features more apparent.

[0030] In step S30, the enhanced multimodal atlas includes enhanced visible light images, enhanced infrared thermal imaging images, and enhanced ultrasound images. Step S30 includes: S310: Convert the denoised visible light image into YCrCb format to obtain a first transition visible light image; process the brightness channel Y of the first transition visible light image through a histogram equalization algorithm to obtain a second transition visible light image; convert the second transition visible light image into RGB format to obtain a third transition visible light image. Preferably, the color format of the denoised visible light image is RGB. The first transition visible light image is processed using the cv2.equalizeHist() function in the OpenCV library. Compared with the denoised visible light image, the grayscale distribution of the third transition visible light image is stretched, and the details in the image are more clearly visible.

[0031] S320: The enhanced infrared thermal imaging image is obtained by stretching the grayscale distribution of the denoised infrared thermal imaging image using a histogram equalization algorithm. Preferably, the denoised infrared thermal imaging image is also processed using the cv2.equalizeHist() function to stretch the grayscale distribution.

[0032] S330: Based on several initial gray values ​​of the denoised ultrasound image, an initial gray range is obtained; a target gray range is set based on the initial gray range; several initial gray values ​​are linearly mapped to the target gray range to obtain several target gray values, so as to form a transition ultrasound image. Preferably, the target grayscale range is larger than the initial grayscale range. Specifically, in this embodiment, the target grayscale range is [0, 255]. Based on the minimum and maximum grayscale values ​​in the denoised ultrasound image, a mapping relationship is calculated, and the grayscale of each pixel in the denoised ultrasound image is recalculated, thereby enhancing the contrast of different tissue structures and defects in the denoised ultrasound image.

[0033] S340: Construct a sharpening filter based on the Laplacian operator, and enhance the high-frequency components of the third transition visible light image and the transition ultrasound image through the sharpening filter to obtain the enhanced visible light image and the enhanced ultrasound image.

[0034] Preferably, the image is convolved using the Laplacian operator to enhance its high-frequency components. Sharpening is achieved using a custom convolution kernel in the OpenCV library; specifically, in this embodiment, [[0,-1,0],[-1,5,-1],[0,-1,0]] is selected as the convolution kernel. Understandably, sharpening helps to highlight defects such as cracks and peeling on the surface of the sound barrier.

[0035] Step S40: Perform image registration on the enhanced multimodal image set to obtain a registered multimodal image set, and crop the registered multimodal image set into a cropped multimodal image set; Understandably, aligning images of different modalities to the same coordinate system can ensure the spatial consistency of multimodal images. By cropping the image, redundant areas in the image can be removed, retaining only the effective area containing the sound barrier, reducing the amount of data for subsequent processing, and improving the efficiency of subsequent feature extraction and defect identification.

[0036] In step S40, the registration multimodal image set includes a registered visible light image, a registered infrared thermal imaging image, and a registered ultrasound image. Step S40 includes: S410: Extract several feature points from the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image respectively, and perform feature matching on the several feature points of the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image using a matching algorithm to obtain several matching point pairs; Preferably, the SIFT algorithm is used to register the multimodal images. Specifically, the cv2.xfeatures2d.SIFT_create() function in the OpenCV library is used to extract several SIFT feature points from the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image. Feature point matching is performed using the BFMatcher() function, and the KNN matching algorithm is used with K set to 2 to select several reliable matching point pairs. The transformation matrix is ​​calculated using the cv2.findHomography() function.

[0037] S420: Calculate several transformation matrices based on several matching point pairs, and perform image registration on the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image based on several transformation matrices to obtain the visible light image, the infrared thermal imaging image, and the ultrasound image to be evaluated, so as to form a multimodal image set to be evaluated. Preferably, the enhanced multimodal image set is transformed to the same coordinate system through several transformation matrices, so that the visible light image to be evaluated, the infrared thermal imaging image to be evaluated, and the ultrasound image to be evaluated have spatial consistency, which is beneficial to providing an accurate data foundation for subsequent feature fusion and defect identification.

[0038] S430: Set a root mean square error threshold and a peak signal-to-noise ratio threshold; calculate several root mean square errors and several peak signal-to-noise ratios based on the enhanced multimodal image set and the multimodal image set to be evaluated; determine whether the multimodal image set to be evaluated meets the registration accuracy based on several root mean square errors and the root mean square error threshold; and determine whether the multimodal image set to be evaluated meets the image quality based on several peak signal-to-noise ratios and the peak signal-to-noise ratio threshold. S440: If the multimodal image set to be evaluated meets the registration accuracy and image quality requirements, then the multimodal image set to be evaluated is established as a registration multimodal image set; Preferably, the error between the registered image and the reference image is calculated to obtain the root mean square error (RMSE) and the peak signal-to-noise ratio (PSNR). The smaller the RMSE, the higher the registration accuracy; the larger the PSNR, the better the image quality and the better the registration effect. If the RMSE is less than the RMSE threshold and the PSNR is greater than the PSNR threshold, then the multimodal image to be evaluated can be judged to meet the registration accuracy and image quality requirements. Specifically, when the RMSE is less than 5 pixels and the PSNR is greater than 30dB, the registered image meets the registration accuracy and image quality requirements and satisfies the accuracy requirements for sound barrier defect detection. If not, the parameters of the SIFT algorithm are adjusted, such as the threshold for feature point detection and the dimension of the feature point descriptor, to optimize the image registration result.

[0039] S450: Determine the positions of the sound barriers in the registered visible light image, the registered infrared thermal image, and the registered ultrasound image respectively, and crop the registered visible light image, the registered infrared thermal image, and the registered ultrasound image according to several sound barrier positions to obtain a cropped multimodal atlas.

[0040] Preferably, the location of the sound barrier is determined based on its actual position and size in the image, combined with detection requirements, to determine the image cropping area. The location of the sound barrier can be determined by manual annotation or target detection algorithms, and the bounding box is established to extract the effective area containing the sound barrier. Specifically, in this embodiment, the SSD model is used to predict the position of the sound barrier and the bounding box coordinates. After determining the cropping area, an image processing library is used for image cropping. Furthermore, the cropped image can be obtained by inputting the rectangular coordinates of the image and the cropping area through the cv2.getRectSubPix() function, and the cropped multimodal image set is adjusted to a uniform pixel size.

[0041] Step S50: Based on the cropped multimodal map set, extract and fuse multimodal features to obtain a multimodal feature map; Feature extraction is performed on the preprocessed visible light image, infrared thermal imaging image, and ultrasound image to obtain the multimodal features, which are then fused to fully utilize the complementary information of different modal images and improve the expressive power of the features.

[0042] In step S50, the cropping of the multimodal image set includes cropping visible light images, cropping infrared thermal imaging images, and cropping ultrasound images. Step S50 includes: S510: Obtain the first neural network, the second neural network, and the third neural network; Preferably, the first neural network is a ResNet-50 deep convolutional neural network, the second neural network is a VGG-16 network, and the third neural network is an Inception-V3 network.

[0043] S520: Based on the first neural network, feature extraction is performed on the cropped visible light image, and residual connections are made to obtain image semantic features; Preferably, introducing residual connections into the ResNet-50 deep convolutional neural network can effectively solve the gradient vanishing problem in deep neural networks.

[0044] S530: Based on the second neural network, feature extraction is performed on the cropped infrared thermal imaging image to obtain temperature distribution features and abnormal region features; Preferably, the VGG-16 network includes 13 convolutional layers and 3 fully connected layers, which has a strong ability to extract the spatial structural features of the cropped thermal imaging image.

[0045] S540: Based on the third neural network, feature extraction is performed on the cropped ultrasound image to obtain defect features. The image semantic features, the temperature distribution features, the abnormal region features, and the defect features constitute multimodal features. Preferably, the Inception module is introduced into the Inception-V3 network, which can extract features from the image at different scales and adapt to defects of different sizes and shapes in the cropped ultrasound image.

[0046] S550: The multimodal features are fused using a channel attention mechanism to obtain a multimodal feature map.

[0047] Preferably, the multimodal features are subjected to global average pooling to compress the spatial dimension and obtain channel-dimensional feature vectors. Then, the channel feature vectors are learned through two fully connected layers to obtain the importance weight of each channel, so as to highlight the key features related to the defects on the surface of the sound barrier and suppress irrelevant information. Specifically, a channel attention mechanism module is implemented in Python through a custom layer. After processing by the channel attention mechanism module, a splicing operation is performed to achieve the fusion of multimodal features.

[0048] Step S60: Predict the multimodal feature map to obtain several defect types, several defect locations and several confidence levels, and evaluate the defect level based on the several defect types, several defect locations and several confidence levels.

[0049] Preferably, the defect types include cracks, peeling, corrosion, etc. For crack defects, their severity can be determined based on the length, width, and depth of the crack. For peeling defects, they can be evaluated based on the area and location of the peeling. Understandably, based on several defect types, several defect locations, and several confidence levels, multiple information about the defects can be identified, and the defects on the surface of the sound barrier can be comprehensively evaluated from multiple dimensions such as defect area, defect depth, defect number, and defect distribution.

[0050] Step S60 includes: S610: The multimodal feature map is predicted using an object detection network or a fully connected network, and several defect types are obtained based on the cross-entropy loss function, and several defect locations and several confidence levels are obtained based on the mean square error loss function. Preferably, if a fully connected network is used for prediction, the `Dense()` function in the Keras library can be used to construct a fully connected layer, setting an appropriate number of neurons and activation function, such as the ReLU activation function. The number of neurons in the last fully connected layer is determined according to the number of defect types, and the `Softmax` activation function is used to output the probability distribution of each defect type. Simultaneously, the regression layer outputs the defect location and the confidence score. If an object detection network is used for prediction, feature extraction, feature fusion, and object prediction are performed through the backbone, neck, and head networks of the object detection network. The defect type, defect location, and confidence score are output, where the defect location is the bounding box coordinates of the defect. Specifically, in this embodiment, the YOLOv5 object detection network is used for prediction. When using the YOLOv5 object detection network for classification tasks, the error between the predicted result and the true label is calculated using the cross-entropy loss function. Therefore, by minimizing the cross-entropy loss function, several defect types are obtained, which can make the classification prediction results as close as possible to the true labels and improve the accuracy of defect classification. When performing the regression task of the defect location and the confidence level, the mean squared error loss function can be used to calculate the mean squared error between the predicted location and the true location, and the mean squared error between the predicted confidence level and the true confidence level. The network parameters are adjusted by the backpropagation algorithm so that it can accurately predict the defect location and the confidence level. Specifically, in the TensorFlow library, the mean squared error is calculated by the tf.keras.losses.MeanSquaredError() function. Further, a prediction model is built based on the object detection network or the fully connected network, and a prediction model optimizer is built by the stochastic gradient descent algorithm to adjust the parameters of the prediction model. The learning rate of the model optimizer is set to 0.001 and the momentum is set to 0.9.

[0051] Furthermore, a large dataset of multimodal images of sound barriers was collected, including images of normal sound barriers and sound barriers with various defects. Bounding boxes were labeled on the dataset, and the types of defects were marked. 70% of the labeled image dataset was used for training, 15% for validation, and 15% for testing to train the prediction model. The performance and generalization ability of the model were evaluated through validation and testing. During the training process, a learning rate decay strategy was adopted according to the training status of the prediction model. Specifically, cosine annealing decay was used to gradually reduce the learning rate in order to improve the training stability and convergence speed of the model.

[0052] S620: Determine several defect area algorithms based on several defect types, and obtain a defect area index based on several defect area algorithms, several defect locations, and several confidence levels; Preferably, detection results with a confidence level below 0.5 are considered false detections and filtered out. For crack defects, the defect area algorithm calculates the area based on the length and width of the crack. For spalling defects, the defect area algorithm directly measures the area of ​​the spalled region. Minor defects have an area less than 10 cm². 2 The area of ​​a moderate defect is 10 cm². 2 ~50cm 2 The area of ​​a serious defect is greater than 50 cm² 2 .

[0053] S630: Obtain the sound barrier thickness, and based on several of the aforementioned defect types, several of the aforementioned defect locations, several of the aforementioned confidence levels, the aforementioned clipped multimodal atlas, and the aforementioned sound barrier thickness, obtain a defect depth index; Preferably, the depth of the defect is obtained through ultrasonic images, that is, the depth of the defect is obtained through the cut ultrasonic images in the cut multimodal image set, so as to determine the defect level according to the ratio of the depth of the defect to the thickness of the sound barrier for defects such as internal voids and delamination. The depth of a minor defect is less than one-fifth of the thickness of the sound barrier, the depth of a moderate defect is between one-fifth and one-half of the thickness of the sound barrier, and the depth of a severe defect is greater than one-half of the thickness of the sound barrier.

[0054] S640: Based on several defect locations and several confidence levels, determine the number of defects per unit area, and obtain a defect quantity index based on the number of defects per unit area; Preferably, defects with a confidence level below 0.5 are filtered out, and the number of defects per unit area is counted based on the remaining reliable defect locations. The defects are then classified into different levels based on the number of defects: minor defects have fewer than 3 defects, moderate defects have between 3 and 10 defects, and severe defects have more than 10 defects.

[0055] S650: Obtain the key locations of the sound barrier, and based on the key locations of the sound barrier, several defect locations, and several confidence levels, obtain a defect distribution index; Preferably, the key locations of the sound barrier are the key stress-bearing parts and edge areas, such as the fixing bolts. If the defects are concentrated in the key locations of the sound barrier, the defect level is increased.

[0056] S660: Evaluate the defect level based on the defect area index, the defect depth index, the defect quantity index, and the defect distribution index.

[0057] Preferably, a minor defect corresponds to a defect area index value of 1, a moderate defect corresponds to a defect area index value of 2, and a severe defect corresponds to a defect area index value of 3. The defect area index is assigned a weight of 0.3, the defect depth index is assigned a weight of 0.3, the defect quantity index is assigned a weight of 0.2, and the defect distribution index is assigned a weight of 0.2. Thus, a comprehensive score for the defect level can be calculated to assess the defect level.

[0058] Furthermore, the diagnostic results are output after evaluation. The diagnostic results include information such as the detection time, detection location, type and specifications of the sound barrier, and detection equipment model. At the same time, the diagnostic results summarize the overall situation of defects on the surface of the sound barrier, including the total number of defects, the number and corresponding proportion of different types of defects, and the number and corresponding proportion of different grades of defects. For each defect, the defect type, defect location, and confidence level are recorded, as well as its defect level and size. The corresponding trimmed multimodal atlas is attached so that maintenance personnel can accurately find the defects and perform maintenance.

[0059] Please see Figure 2 The sound barrier surface defect assessment system in the second embodiment of the present invention applies the sound barrier surface defect assessment method as described in the first embodiment above, and the system includes: The acquisition module 10 is used to acquire initial visible light images, initial infrared thermal imaging images and initial ultrasonic images of the sound barrier surface to form an initial multimodal atlas. The acquisition module 10 includes: The first unit is used to acquire an initial visible light image of the sound barrier surface in the vertical direction using a camera, acquire an initial infrared thermal image of the sound barrier surface using an infrared thermal imager, and acquire an initial ultrasonic image of the sound barrier surface based on ultrasonic waves with a frequency of 1MHz to 10MHz.

[0060] Denoising module 20 is used to perform image denoising processing on the initial multimodal image set to obtain a denoised multimodal image set; In the denoising module 20, the denoised multimodal image set includes denoised visible light images, denoised infrared thermal imaging images, and denoised ultrasound images. The denoising module 20 includes: The second unit is used to remove Gaussian noise from the initial visible light image and smooth the image edges to obtain the denoised visible light image; The third unit is used to remove pulse noise from the initial infrared thermal imaging image by median filtering to obtain the denoised infrared thermal imaging image. The fourth unit is used to perform wavelet decomposition on the initial ultrasound image, remove noise components and perform wavelet reconstruction to obtain the denoised ultrasound image, and combine the denoised visible light image, the denoised infrared thermal imaging image and the denoised ultrasound image into a denoised multimodal image set.

[0061] Enhancement module 30 is used to perform image enhancement processing on the denoised multimodal image set to obtain an enhanced multimodal image set; In the enhancement module 30, the enhanced multimodal atlas includes enhanced visible light images, enhanced infrared thermal imaging images, and enhanced ultrasound images. The enhancement module 30 includes: The fifth unit is used to convert the denoised visible light image into YCrCb format to obtain a first transition visible light image, process the brightness channel Y of the first transition visible light image through a histogram equalization algorithm to obtain a second transition visible light image, and convert the second transition visible light image into RGB format to obtain a third transition visible light image. The sixth unit is used to stretch the grayscale distribution of the denoised infrared thermal imaging image using a histogram equalization algorithm to obtain the enhanced infrared thermal imaging image. The seventh unit is used to obtain an initial grayscale range based on several initial grayscale values ​​of the denoised ultrasound image, set a target grayscale range based on the initial grayscale range, linearly map several initial grayscale values ​​to the target grayscale range, and obtain several target grayscale values ​​to form a transition ultrasound image. The eighth unit is used to construct a sharpening filter based on the Laplacian operator, and to enhance the high-frequency components of the third transition visible light image and the transition ultrasound image through the sharpening filter, so as to obtain the enhanced visible light image and the enhanced ultrasound image.

[0062] The registration and cropping module 40 is used to perform image registration on the enhanced multimodal image set to obtain a registered multimodal image set, and to crop the registered multimodal image set into a cropped multimodal image set. In the registration and cropping module 40, the registration multimodal image set includes registered visible light images, registered infrared thermal imaging images, and registered ultrasound images. The registration and cropping module 40 includes: The ninth unit is used to extract several feature points from the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image, respectively, and to perform feature matching on the several feature points of the enhanced visible light image, the enhanced infrared thermal imaging image, and the enhanced ultrasound image using a matching algorithm to obtain several matching point pairs; The tenth unit is used to calculate several transformation matrices based on several matching point pairs, and to perform image registration on the enhanced visible light image, the enhanced infrared thermal imaging image and the enhanced ultrasound image based on several transformation matrices to obtain the visible light image, the infrared thermal imaging image and the ultrasound image to be evaluated, so as to form a multimodal image set to be evaluated. The eleventh unit is used to set the root mean square error threshold and the peak signal-to-noise ratio threshold, calculate several root mean square errors and several peak signal-to-noise ratios based on the enhanced multimodal image set and the multimodal image set to be evaluated, determine whether the multimodal image set to be evaluated meets the registration accuracy based on the several root mean square errors and the root mean square error threshold, and determine whether the multimodal image set to be evaluated meets the image quality based on the several peak signal-to-noise ratios and the peak signal-to-noise ratio threshold. The twelfth unit is used to establish the multimodal image set to be evaluated as a registered multimodal image set if the multimodal image set to be evaluated meets the registration accuracy and image quality requirements. The thirteenth unit is used to determine the positions of the sound barriers in the registered visible light image, the registered infrared thermal imaging image, and the registered ultrasound image, respectively, and to crop the registered visible light image, the registered infrared thermal imaging image, and the registered ultrasound image according to several sound barrier positions to obtain a cropped multimodal atlas.

[0063] The fusion module 50 is used to extract and fuse multimodal features based on the cropped multimodal atlas to obtain a multimodal feature map; In the fusion module 50, the cropped multimodal image atlas includes cropping visible light images, cropping infrared thermal imaging images, and cropping ultrasound images. The fusion module 50 includes: The fourteenth unit is used to obtain the first neural network, the second neural network, and the third neural network; The fifteenth unit is used to extract features from the cropped visible light image based on the first neural network and perform residual connections to obtain image semantic features; The sixteenth unit is used to extract features from the cropped infrared thermal imaging image based on the second neural network to obtain temperature distribution features and abnormal area features. The seventeenth unit is used to extract features from the cropped ultrasound image based on the third neural network to obtain defect features. The image semantic features, the temperature distribution features, the abnormal region features, and the defect features constitute multimodal features. The eighteenth unit is used to fuse the multimodal features using a channel attention mechanism to obtain a multimodal feature map.

[0064] Evaluation module 60 is used to predict the multimodal feature map to obtain several defect types, several defect locations and several confidence levels, and evaluate the defect level based on the several defect types, several defect locations and several confidence levels.

[0065] The evaluation module 60 includes: The nineteenth unit is used to predict the multimodal feature map using a target detection network or a fully connected network, obtain several defect types based on the cross-entropy loss function, and obtain several defect locations and several confidence levels based on the mean square error loss function. The twentieth unit is used to determine several defect area algorithms based on several defect types, and to obtain a defect area index based on several defect area algorithms, several defect locations and several confidence levels. The twenty-first unit is used to obtain the sound barrier thickness by obtaining a defect depth index based on several defect types, several defect locations, several confidence levels, the clipped multimodal atlas, and the sound barrier thickness. The twenty-second unit is used to determine the number of defects per unit area based on several defect locations and several confidence levels, and to obtain a defect quantity index based on the number of defects per unit area. The twenty-third unit is used to obtain the key locations of the sound barrier, and based on the key locations of the sound barrier, several defect locations and several confidence levels, to obtain a defect distribution index. The twenty-fourth unit is used to evaluate the defect level based on the defect area index, the defect depth index, the defect quantity index, and the defect distribution index.

[0066] The third embodiment of the present invention provides a computer, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the sound barrier surface defect assessment method as described in the first embodiment.

[0067] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0068] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method of evaluating surface defects of a sound barrier, characterized by, The method comprises the following steps: Collecting initial visible light images, initial infrared thermal imaging images and initial ultrasonic images of a sound barrier surface to form an initial multi-modal image set; Performing image denoising processing on the initial multi-modal image set to obtain a denoised multi-modal image set; Performing image enhancement processing on the denoised multi-modal image set to obtain an enhanced multi-modal image set; Performing image registration on the enhanced multi-modal image set to obtain a registered multi-modal image set, and cutting the registered multi-modal image set into a cut multi-modal image set; Based on the cut multi-modal image set, multi-modal features are extracted and fused to obtain a multi-modal feature map; Performing prediction on the multi-modal feature map to obtain a plurality of defect types, a plurality of defect positions and a plurality of confidence levels, and evaluating the defect level based on the plurality of defect types, the plurality of defect positions and the plurality of confidence levels.

2. The sound barrier surface defect evaluation method according to claim 1, characterized by, The step of collecting initial visible light images, initial infrared thermal imaging images and initial ultrasonic images of a sound barrier surface comprises: Collecting initial visible light images of the sound barrier surface by a camera in the vertical direction of the sound barrier surface, collecting initial infrared thermal imaging images of the sound barrier surface by an infrared thermal imager, and collecting initial ultrasonic images of the sound barrier surface based on ultrasonic waves with a frequency of 1 MHz to 10 MHz.

3. The sound barrier surface defect evaluation method according to claim 1, characterized by, The denoised multi-modal image set comprises denoised visible light images, denoised infrared thermal imaging images and denoised ultrasonic images, and the step of performing image denoising processing on the initial multi-modal image set to obtain a denoised multi-modal image set comprises: Removing Gaussian noise in the initial visible light images and smoothing the image edges to obtain the denoised visible light images; Removing impulse noise in the initial infrared thermal imaging images by a median filter method to obtain the denoised infrared thermal imaging images; Performing wavelet decomposition on the initial ultrasonic images, removing noise components and performing wavelet reconstruction to obtain the denoised ultrasonic images, and combining the denoised visible light images, the denoised infrared thermal imaging images and the denoised ultrasonic images into a denoised multi-modal image set.

4. The sound barrier surface defect evaluation method according to claim 3, characterized by, The enhanced multi-modal image set comprises enhanced visible light images, enhanced infrared thermal imaging images and enhanced ultrasonic images, and the step of performing image enhancement processing on the denoised multi-modal image set to obtain an enhanced multi-modal image set comprises: Converting the denoised visible light images into YCrCb format to obtain a first transition visible light image, processing the luminance channel Y of the first transition visible light image by a histogram equalization algorithm to obtain a second transition visible light image, converting the second transition visible light image into RGB format to obtain a third transition visible light image; Stretching the gray scale distribution of the denoised infrared thermal imaging images by a histogram equalization algorithm to obtain the enhanced infrared thermal imaging images; Based on a plurality of initial gray scale values of the denoised ultrasonic images, an initial gray scale interval is obtained, a target gray scale interval is set based on the initial gray scale interval, a plurality of target gray scale values are obtained by linearly mapping a plurality of the initial gray scale values to the target gray scale interval to form a transition ultrasonic image; A sharpening filter is constructed based on a Laplacian operator, and high-frequency components of the third transition visible light image and the transition ultrasonic image are enhanced through the sharpening filter to obtain the enhanced visible light image and the enhanced ultrasonic image.

5. The sound barrier surface defect evaluation method according to claim 4, characterized by, The step of performing image registration on the enhanced multi-modal image set to obtain a registered multi-modal image set comprises: A plurality of feature points in the enhanced visible light image, the enhanced infrared thermal imaging image and the enhanced ultrasonic image are extracted respectively, and a plurality of the feature points in the enhanced visible light image, the enhanced infrared thermal imaging image and the enhanced ultrasonic image are matched by a matching algorithm to obtain a plurality of matching point pairs; A plurality of transformation matrices are calculated based on the plurality of matching point pairs, and the enhanced visible light image, the enhanced infrared thermal imaging image and the enhanced ultrasonic image are registered based on the plurality of transformation matrices to obtain an evaluated visible light image, an evaluated infrared thermal imaging image and an evaluated ultrasonic image to form an evaluated multi-modal image set; Root mean square error threshold and peak signal-to-noise ratio threshold are set, a plurality of root mean square errors and a plurality of peak signal-to-noise ratios are calculated based on the enhanced multi-modal image set and the evaluated multi-modal image set, whether the evaluated multi-modal image set meets the registration accuracy is judged based on the plurality of root mean square errors and the root mean square error threshold, and whether the evaluated multi-modal image set meets the image quality is judged based on the plurality of peak signal-to-noise ratios and the peak signal-to-noise ratio threshold; If the evaluated multi-modal image set meets the registration accuracy and the image quality, the evaluated multi-modal image set is established as a registered multi-modal image set.

6. The sound barrier surface defect evaluation method according to claim 1, characterized by, The registered multi-modal image set comprises a registered visible light image, a registered infrared thermal imaging image and a registered ultrasonic image, and the step of cropping the registered multi-modal image set into a cropped multi-modal image set comprises: Sound barrier positions in the registered visible light image, the registered infrared thermal imaging image and the registered ultrasonic image are determined respectively, and the registered visible light image, the registered infrared thermal imaging image and the registered ultrasonic image are cropped according to a plurality of the sound barrier positions to obtain a cropped multi-modal image set.

7. The sound barrier surface defect evaluation method according to claim 1, characterized by, The cropped multi-modal image set comprises a cropped visible light image, a cropped infrared thermal imaging image and a cropped ultrasonic image, and the step of extracting multi-modal features and fusing based on the cropped multi-modal image set to obtain a multi-modal feature map comprises: A first neural network, a second neural network and a third neural network are obtained; Image semantic features are extracted from the cropped visible light image based on the first neural network, and residual connection is performed to obtain the image semantic features; Temperature distribution features and abnormal area features are extracted from the cropped infrared thermal imaging image based on the second neural network; Defect features are extracted from the cropped ultrasonic image based on the third neural network, and the image semantic features, the temperature distribution features, the abnormal area features and the defect features constitute multi-modal features; The multi-modal features are fused by using a channel attention mechanism to obtain a multi-modal feature map.

8. The sound barrier surface defect evaluation method of claim 1, wherein, The step of predicting the multi-modal feature map to obtain a plurality of defect types, a plurality of defect positions, and a plurality of confidence levels, and evaluating a defect level based on the plurality of defect types, the plurality of defect positions, and the plurality of confidence levels comprises: The multi-modal feature map is predicted by using a target detection network or a fully connected network, a plurality of defect types are obtained based on a cross-entropy loss function, a plurality of defect positions and a plurality of confidence levels are obtained based on a mean square error loss function; A plurality of defect area algorithms are determined based on the plurality of defect types, a defect area index is obtained based on the plurality of defect area algorithms, the plurality of defect positions, and the plurality of confidence levels; The thickness of the sound barrier is obtained, and a defect depth index is obtained based on the plurality of defect types, the plurality of defect positions, the plurality of confidence levels, the cropped multi-modal image set, and the thickness of the sound barrier; A unit area defect quantity is determined based on the plurality of defect positions and the plurality of confidence levels, and a defect quantity index is obtained based on the unit area defect quantity; A sound barrier key position is obtained, and a defect distribution index is obtained based on the sound barrier key position, the plurality of defect positions, and the plurality of confidence levels; The defect level is evaluated based on the defect area index, the defect depth index, the defect quantity index, and the defect distribution index.

9. A sound barrier surface defect evaluation system applying the sound barrier surface defect evaluation method according to any one of claims 1 to 8, characterized by The system comprises: A collection module configured to collect an initial visible light image, an initial infrared thermal imaging image, and an initial ultrasonic image of a surface of a sound barrier to form an initial multi-modal image set; A denoising module configured to perform image denoising processing on the initial multi-modal image set to obtain a denoised multi-modal image set; An enhancement module configured to perform image enhancement processing on the denoised multi-modal image set to obtain an enhanced multi-modal image set; A registration and cropping module configured to perform image registration on the enhanced multi-modal image set to obtain a registered multi-modal image set, and crop the registered multi-modal image set into a cropped multi-modal image set; A fusion module configured to extract multi-modal features and fuse based on the cropped multi-modal image set to obtain a multi-modal feature map; An evaluation module configured to predict the multi-modal feature map to obtain a plurality of defect types, a plurality of defect positions, and a plurality of confidence levels, and evaluate a defect level based on the plurality of defect types, the plurality of defect positions, and the plurality of confidence levels.

10. A computer comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the sound barrier surface defect evaluation method of any one of claims 1-8. The processor executes the computer program to implement the sound barrier surface defect evaluation method of any one of claims 1-8.

Images