Inspection image feature detection and disease identification method

Abstract

The invention discloses a routing inspection image feature detection and disease recognition method, which comprises the following steps: firstly, matching a routing inspection image acquired by an unmanned aerial vehicle with a template image in a pre-constructed image sample library by adopting a template matching algorithm, and determining an original template image corresponding to the routing inspection image; secondly, performing feature extraction on the inspection image and the matched template image by adopting an image feature extraction algorithm to obtain image features of the inspection image and the template image; and then, according to the extracted image features, adopting an image difference algorithm to obtain difference features between the inspection image and the template image, and obtaining a change feature image of the inspection image, thereby realizing intelligent identification and marking of a disease area in the inspection image, and finally, inputting the change feature image into a trained disease classification model, thereby realizing disease classification. And classifying and identifying diseases in the inspection image through the disease classification model.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition using classification, and in particular to a method for detecting inspection image features and identifying defects. Background Art

[0002] Under the influence of natural and human factors, high slopes are prone to geological disasters such as slope scour, landslides, debris flows, and collapses. Therefore, regular inspections of high highway slopes can effectively reduce the occurrence of highway slope disasters and play a positive role in ensuring the normal operation of highways.

[0003] Highway slope safety inspections primarily rely on manual inspections. Traditional manual inspections employ visual inspections, tapping, and touch, with visual inspection being the primary method. Inspection results are typically recorded manually, leading to safety risks, a lack of image data, blind spots, and a lack of dedicated database storage and system management.

[0004] Currently, drone-based rapid slope image inspection equipment is available, enabling rapid, automated patrol capture of multiple inspection points. This significantly improves the efficiency and accuracy of slope inspections and solves the image acquisition problem. However, image processing after inspections still requires technical personnel to review and identify each image individually, increasing the workload. As the number of slopes inspected increases, the image recognition workload increases dramatically, reducing not only recognition precision and accuracy but also manual error-proneness and significantly reducing work efficiency. Therefore, intelligent recognition and judgment after image acquisition have become key issues that need to be addressed in current highway slope inspections. Summary of the Invention

[0005] In response to the shortcomings of existing technologies, this paper proposes a method for inspection image feature detection and disease identification, which can realize automatic disaster identification and labeling of drone inspection images. The specific technical solution is as follows: In a first aspect, a method for detecting inspection image features is provided. In a first possible implementation of the first aspect, the method includes: Perform template matching on the acquired inspection image to determine a template image that matches the inspection image; Performing feature extraction on the inspection image and the template image respectively to obtain image features of the inspection image and the template image; According to the image features of the inspection image and the template image, an image difference algorithm is used to obtain a change feature image of the inspection image.

[0006] In combination with the first implementable manner of the first aspect, in a second implementable manner of the first aspect, performing template matching on the acquired inspection image includes: A pyramid sampling template matching algorithm is used to match the template image of the inspection image.

[0007] In combination with the first implementable manner of the first aspect, in a third implementable manner of the first aspect, performing feature extraction on the inspection image and the template image includes: The inspection image and the template image are subjected to image noise filtering processing, color restoration processing, image enhancement processing and / or dual-channel histogram equalization processing.

[0008] In combination with the third possible implementation of the first aspect, a fourth possible implementation of the first aspect includes: Use statistical-based color restoration algorithms to perform color restoration on inspection images and template images; And / or, an image enhancement algorithm based on guided filtering Retinex is used to perform image enhancement processing on the inspection image and the template image.

[0009] In combination with the first implementable manner of the first aspect, in a fifth implementable manner of the first aspect, feature extraction is performed on the inspection image and the template image respectively, including: Extract edge features and / or macro deformation features of the inspection image and the template image.

[0010] In conjunction with the fifth possible implementation of the first aspect, a sixth possible implementation of the first aspect includes: The Canny edge detection algorithm is used to extract edge features of the inspection image and the template image; And / or, a detection algorithm based on Lucas-Kanade optical flow is used to extract macroscopic deformation features of the inspection image and the template image.

[0011] In combination with the first implementable manner of the first aspect, in a seventh implementable manner of the first aspect, using an image difference algorithm to obtain a change feature image of the inspection image includes: An image difference algorithm is used to obtain a difference image between the inspection image and the template image, and the difference image is subjected to an adaptive threshold binarization process; An image erosion and dilation operation is performed on the result of the adaptive threshold binarization process to obtain a change feature image of the inspection image.

[0012] In a second aspect, a method for identifying defects in inspection images is provided. In a first possible implementation of the second aspect, the method includes: Adopting the inspection image feature detection method as described in any one of the first to seventh possible implementations of the first aspect to obtain a change feature image of the inspection image; Based on the change feature image, the diseases in the inspection image are classified and identified using a trained disease classification model.

[0013] In combination with the first implementable manner of the second aspect, in a second implementable manner of the second aspect, a ConvNext model is used as the disease classification model.

[0014] In combination with the first implementable manner of the second aspect, in a third implementable manner of the second aspect, a transfer learning strategy is adopted to train the ConvNext model.

[0015] Beneficial Effects: The inspection image feature detection and defect identification method of the present invention can determine a template image that matches the inspection image by performing template matching on the acquired inspection image against a pre-built template image database. Based on the image features of the inspection image and the matched template image, an image difference algorithm is then used to effectively segment the changing feature image in the inspection image, enabling intelligent identification and labeling of defect areas in the inspection image, laying the foundation for subsequent defect identification and classification in the inspection image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present invention, the following briefly introduces the drawings required for use in the specific embodiments. In all the drawings, each element or part is not necessarily drawn according to the actual scale.

[0017] Figure 1 A flowchart of a method for detecting inspection image features according to an embodiment of the present invention; Figure 2 A flowchart of a method for identifying defects in inspection images provided by one embodiment of the present invention; Figure 3 This is a comparison chart of the inspection image before and after mean filtering denoising; Figure 4 This is a comparison chart of the inspection image before and after Gaussian filtering denoising; Figure 5 This is a comparison chart of the inspection image before and after Retinex image enhancement; Figure 6 Schematic diagram for color restoration; Figure 7 Inspection images and matching template images; Figure 8 For Figure 7 The inspection image and template image shown, and the edge information diagrams detected by three edge detection algorithms; Figure 9 It is the grayscale image of the inspection image during macro deformation detection; Figure 10 For Figure 9 The inspection image shown uses the Lucas-Kanade optical flow algorithm to detect macroscopic deformation areas; Figure 11 This is a comparison chart of the inspection image before and after histogram equalization; Figure 12 This is the effect diagram of post-processing of the difference image; Figure 13 This is the ConvNext model structure diagram. DETAILED DESCRIPTION

[0018] The following embodiments of the technical solution of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention and are therefore only examples and are not intended to limit the scope of protection of the present invention.

[0019] like Figure 1 The flowchart of the inspection image feature detection method shown in FIG. 1 includes: Step 1: Perform template matching on the acquired inspection image to determine a template image that matches the inspection image; Step 2: extracting features from the inspection image and the template image to obtain image features of the inspection image and the template image; Step 3: Based on the image features of the inspection image and the template image, an image difference algorithm is used to obtain a change feature image of the inspection image.

[0020] Specifically, first, existing template matching algorithms can be used to match inspection images captured by drones with template images in a pre-built image sample library to determine the original template image corresponding to the inspection image. Then, existing image feature extraction algorithms can be used to extract features from the inspection image and the matched template image to obtain the image features of the inspection image and the template image. Finally, based on the extracted image features, an image difference algorithm can be used to determine the difference features between the inspection image and the template image, resulting in a change feature image of the inspection image. This allows for intelligent identification and labeling of diseased areas in the inspection image, laying the foundation for subsequent disease identification and classification in the inspection image.

[0021] The image differencing algorithm effectively identifies millimeter-level structural changes, such as 0.2mm cracks or insulator damage. Its lightweight nature (single-frame processing time <50ms) requires no model training, significantly reducing computing power requirements. Compared to end-to-end deep learning solutions, this method significantly suppresses environmental interference such as lighting changes and seasonal vegetation changes through feature alignment, while also filtering out non-structural noise such as cloud shadows. The resulting binary difference map provides high-purity input for the subsequent disease classification network.

[0022] In this embodiment, when constructing the image sample library, typical disease samples can be collected from highway slope structures, drainage facilities, and slope deformations using drones, the internet, and other means. Because these data collection methods can only collect a limited number of samples, noise can be added to the collected samples to simulate the noise and interference found in real scenes, generating more disease samples and expanding the image sample library.

[0023] For example, OpenCV can be used to add noise of different types and densities to generate more training data, thereby improving the performance and robustness of the model. Alternatively, OpenCV can be used to randomly replace pixel values ​​in some related images and adjust the pixel values ​​to the pixel values ​​of the diseased samples to generate diseased images.

[0024] In addition, image contrast enhancement algorithms, such as gamma correction and histogram equalization, can be used to enhance the contrast of newly generated image samples to hide the differences between tampered and non-tampered areas and erase traces of local image tampering operations.

[0025] In this embodiment, optionally, template matching is performed on the acquired inspection image, including: A pyramid sampling template matching algorithm is used to match the template image of the inspection image.

[0026] Specifically, there are many methods for performing template matching on the acquired inspection images, such as brute force matching algorithm, normalized cross-correlation matching algorithm, square difference matching algorithm and pyramid sampling template matching algorithm.

[0027] Among them, the brute force matching algorithm is a pixel-based template matching algorithm. Its basic principle is to traverse every pixel in the target image and compare it with the template image to find the most similar position. Specifically, the pixels of the template image and the inspection image are first compared. For each pixel in the template image, the corresponding pixel is found in the inspection image, and the error between them is calculated. The error can be expressed using methods such as distance between pixels, Euclidean distance, and square error. Then, the position with the smallest error is found and marked as the best matching position. Finally, the best matching position can be marked in the target image, or a metric value of the matching degree can be calculated. The time complexity of the brute force matching algorithm is very high, and every pixel in the inspection image needs to be compared. Therefore, its matching efficiency is relatively low for large-sized images.

[0028] The Normalized Cross Correlation (NCC) algorithm is a commonly used template matching algorithm that finds the best matching location by calculating the cross-correlation coefficient between each possible location in the template image and the original image. NCC effectively handles issues such as illumination variations, scale changes, and rotation changes, and is therefore widely used in image processing and computer vision.

[0029] The NCC algorithm compares the pixel values ​​at every possible location in the template image and the original image and calculates the similarity between them. The present invention uses the cross correlation coefficient to measure similarity. The cross correlation coefficient is a value between -1 and 1 that represents the linear correlation between two vectors. When the cross correlation coefficient is 1, the two vectors are completely identical; when the cross correlation coefficient is -1, the two vectors are completely opposite; and when the cross correlation coefficient is 0, the two vectors have no correlation at all.

[0030] In the NCC algorithm, for each possible position in the inspection image, the correlation coefficient between it and the template image can be calculated, and the position with the highest score is taken as the matching result. Specifically, the NCC algorithm can be divided into the following steps: First, you need to choose a small target template, which is the object or pattern you want to find in the image. The target template can be an image patch of any size, usually a square or rectangle.

[0031] Then, for the corresponding areas of the target template and the image to be matched, the mean values ​​of the pixel values ​​of the corresponding areas of the target template and the image to be matched are calculated respectively.

[0032] After that, the standard deviation of the pixel values ​​of the corresponding areas of the target template and the image to be matched is calculated respectively, and the following calculation steps are performed for each pixel position in the image to be matched: First, subtract the pixel values ​​at the corresponding positions in the target template from those in the image to obtain a difference sequence.

[0033] The difference sequence is then multiplied by the mean pixel value of the target template.

[0034] Finally, the above result is divided by the product of the standard deviation of the pixel values ​​of the target template and the standard deviation of the pixel values ​​of the corresponding area of ​​the image.

[0035] Then, for each pixel location, a normalized cross-correlation coefficient is obtained. A larger cross-correlation coefficient indicates that the target template is more similar to a region in the image. You can optionally set a threshold to filter out matches with cross-correlation coefficients below the threshold.

[0036] Finally, some post-processing steps can be performed to improve the accuracy of the matching results according to specific needs. For example, non-maximum suppression can be applied to eliminate overlapping matching results.

[0037] The calculation of the NCC coefficient r(x,y) is usually divided into two types: removing the mean and not removing the mean.

[0038] Remove the mean: ; Without removing the mean: ; in, and are the grayscale values ​​of the pixels in the template graphic and image matching window, is the horizontal direction (i.e. The number of pixels involved in the calculation or the width of the window in the horizontal direction indicates the pixel range involved in the relevant calculation in the horizontal direction; is the vertical direction (i.e. The number of pixels involved in the calculation or the height of the window in the vertical direction, and the pixel range involved in the relevant calculation in the vertical direction; is the coordinate on the image plane, marking the starting position of the correlation calculation in the image; and The two images to be matched are Gray value at. NCC coefficient for , the larger the value, the greater the correlation between the two. When , it means there is no correlation between the two images. , it means that the correlation between the two images is the greatest.

[0039] The Pyramid Sampling template matching algorithm is a multi-scale template matching method used to locate the target template in an image. It constructs an image pyramid and performs template matching at different scales to adapt to the scale changes of inspection images.

[0040] The algorithm's speed is ensured by selecting a stable, reliable, and easily computable similarity metric function, fast feature extraction, and the use of an image pyramid algorithm. When considering translation and selection issues with the target being searched, the template is translated and rotated to find one or more targets that are identical or similar to the template in the template image based on the similarity metric. This translation vector and rotation transformation are the desired translation vectors. In a two-dimensional image plane, the translation variables and rotation transformations are as follows: ; in, is the rotation matrix, is the rotation angle.

[0041] The similarity metric function not only affects the accuracy of the final matching result, but also determines the processing time of the algorithm to a large extent. A fast geometric template matching algorithm can be used to calculate the similarity metric value by extracting the edge information of the template image and the inspection image as matching parameters. After the template image is processed by the edge detection algorithm, it can be converted into a series of points. , constitute, is the number of points, corresponding to each point there is a gradient direction vector . 、 are the gradient direction vectors exist 、 The component in the direction reflects the change of the image grayscale in the vertical direction at that point.

[0042] The image being searched can also be converted into a vector consisting of each point (x, y) and its corresponding direction vector In the matching process, the similarity measure is calculated between the transformed template image and the sub-image at a certain position of the inspection image. The transformation expression of the gradient vector of the edge point in the template image is as follows: ; At a certain position in the searched image At the sub-image, the similarity metric function is as follows: ; When the gradient directions of a pair of corresponding edge points are the same, the dot product reaches its maximum value. When the gradient directions are orthogonal, the dot product is zero. When the gradient directions are opposite, the dot product reaches its negative maximum value. The calculated similarity metric value is not convenient for judging whether the sub-image meets the requirements. In other words, it is difficult to determine a threshold to filter the required results. Therefore, the similarity metric function is normalized: ; in, is the normalized gradient direction vector, 、 Respectively in 、 The gradient component after adjustment in the direction, is a feature vector related to another feature vector of the image, such as a feature direction vector that may represent a sub-image area in template matching; 、 are the components of the eigenvector in different directions, is the modulus of the gradient direction vector, is the eigenvector The module length, is the similarity measure after normalization, Represents a vector and The dot product is used to measure the similarity between the two vectors. A mark indicating the format of the formula.

[0043] Since the direction vectors are normalized, the similarity metric is invariant to any illumination changes, regardless of whether there is noise in the template or the search image. Since the direction vectors caused by the noise are random, on average they do not contribute to the above summation formula, and thus have good anti-interference properties.

[0044] When actually calculating the similarity metric, in order to minimize the square root operation, the gradient vector is normalized when calculating the template and the searched image, that is, the gradient vector is a unit vector. and They are 、 The unit vector of . Then the formula for calculating the similarity measure becomes: ; During the search of the entire target image, there are only one or a few true target locations, and the vast majority of locations are non-target locations. This means that during the entire traversal process, the similarity metric values ​​at non-target locations do not need to be fully calculated. By setting a certain threshold and adopting a certain termination condition, the calculation can be terminated early. Non-target locations are not of concern to the algorithm, and similarity metric calculations can be terminated under appropriate conditions. In this embodiment, the termination condition for the similarity metric value can be set as: ; in, Indicates that in the process of traversing the target image, the first The similarity metric value at a position is used to measure the similarity between the position and the target template; Represents the pre-set minimum similarity threshold, Indicates the position number of the current similarity calculation, Indicates the total number of locations where similarity needs to be calculated, that is, the total number of traversals.

[0045] Considering that the inspection image may be partially blocked or hidden, the termination condition can be set as: ; in, In order to avoid misjudgment, the parameter g is usually set to 0.9.

[0046] The pyramid sampling template matching algorithm effectively handles changes in the scale of the target template. By matching images at different scales, it can find the matching position that best matches the target template scale. Furthermore, pyramid sampling improves the algorithm's robustness, making it more tolerant to factors such as illumination changes and rotation.

[0047] By comparing the matching results of various matching algorithms, the pyramid sampling template matching algorithm has the highest matching degree and is more adaptable to noise than other matching algorithms. Therefore, the present invention adopts the pyramid sampling template matching algorithm to match the template image of the inspection pattern.

[0048] In this embodiment, optionally, feature extraction is performed on the inspection image and the template image, including: The inspection image and the template image are subjected to image noise filtering processing, color restoration processing, image enhancement processing and / or dual-channel histogram equalization processing.

[0049] Specifically, after template matching, the matched template image and inspection image are subjected to image preprocessing, and the images are subjected to denoising, color restoration, image enhancement and / or dual-channel histogram equalization, so as to reduce the factors affecting the surrounding environment on the images taken by the drone inspection at different time points.

[0050] Specifically, when using drones to capture images for slope inspections, the unstable and fluid atmosphere can cause optical pixel discrepancies, corrupting the recorded images with noise. Channel noise, measurement errors, and the quantization process of digital storage all contribute to the noise introduced into the transmission medium.

[0051] There are many methods for filtering noise. In this embodiment, mean filtering or Gaussian filtering can be used to perform image noise filtering on the template image and the inspection image.

[0052] Mean filtering is also called linear filtering, and its main method is neighborhood averaging. The basic principle of linear filtering is to replace the pixel values ​​in the original image with the mean, that is, the current pixel to be processed , select a template, which is composed of several pixels of its neighbors, calculate the mean of all pixels in the template, and then assign the mean to the current pixel , as the grayscale of the processed image at that point ,Right now , is the total number of pixels in the template including the current pixel, and f(x, y) is the coordinate of the original image. The grayscale value of the pixel.

[0053] Use mean filtering to remove noise from slope inspection images, such as Figure 3 As shown, the left side is the inspection image before denoising, and the right side is the inspection image after denoising. Figure 3 It can be clearly seen that the noise in the image has been reduced a lot, the image is clearer, and the error is reduced for subsequent processing.

[0054] Gaussian filtering is a linear smoothing filter that is suitable for removing Gaussian noise and is widely used in image denoising. Simply put, Gaussian filtering denoising is a weighted average of the pixel values ​​of the entire image. The value of each pixel is the weighted average of its own value and the values ​​of other pixels in its neighborhood.

[0055] Use a specified template (also called convolution, mask) to scan each pixel in the image, and replace the value of the center pixel of the template with the weighted average grayscale value of the pixels in the neighborhood determined by the template.

[0056] One-dimensional Gaussian distribution: ; in, Indicates The Gaussian distribution function value at the coordinate position, is the standard deviation of pixel values In coordinates The two-dimensional Gaussian distribution function value at is: .

[0057] In image processing, Gaussian filtering is generally implemented in two ways: one is to use a discretized window sliding window convolution, and the other is to use Fourier transform. The first sliding window implementation is the most common. Only when the discretized window is very large and the sliding window is very computationally intensive (i.e., when using a separable filter implementation), the Fourier transform implementation may be considered.

[0058] The specific effect is shown in Figure 4. The left side is the image before denoising, and the right side is the image after denoising. It can be seen that a large amount of noise can be removed, and the image becomes clearer, which reduces the error in the subsequent processing results.

[0059] In this embodiment, the optional steps include: Use statistical-based color restoration algorithms to perform color restoration on inspection images and template images; And / or, an image enhancement algorithm based on guided filtering Retinex is used to perform image enhancement processing on the inspection image and the template image.

[0060] Specifically, in foggy or high-light conditions, inspection images captured by drones during fixed-point patrols may exhibit low quality, excessive brightness, or severe fogging. To address this, an image enhancement algorithm based on guided filtering (Retinex) can be used to enhance the inspection and template images, making them clearer.

[0061] Specifically, the image is first read in by the imread function, and the image is inverted. Then the image is divided into three color channels, RGB, and the image of each channel is normalized. Each image is convolved with pre-defined Gaussian surround functions of different scales to obtain processing results of different scales for different color channels. Then, the processing results of different scales for the same color channel are weightedly added. After grayscale transformation, the image processing of the Retinex algorithm is completed. The images of the three color channels are synthesized by the cat function, and the processing results of the inverted image are inverted again. The obtained result is linearly weighted superpositioned with the fog processing image, and finally the image is displayed. The enhanced effect is as follows: Figure 5 The left side is the image before enhancement, and the right side is the image after enhancement. It can be seen that the image processed by the improved Retinex algorithm is more natural, with better color fidelity and richer details.

[0062] Capturing object color involves three key elements: a light source, the object to be measured, and the device used to capture the color. A complete color reproduction process also includes the image display. Factors that influence color distortion in images captured by image acquisition equipment are the light source and the device itself.

[0063] Taking a CCD image acquisition device as an example, the working principle of the image acquisition device is as follows: uniform light emitted by a light source is irradiated onto the image to be acquired. After reflection or transmission through the image, reflected light or transmitted light representing the color characteristics of the image is generated. After passing through the optical system, it is irradiated onto a color filter. After being separated by the red, yellow, and blue color filters, the R, G, and B components are obtained. These are irradiated onto the corresponding CCD photosensitive elements and converted into analog signals. After signal processing, they are converted into RGB digital signals and stored to obtain an RGB image. At the same time, the image acquisition device uses the RGB color space associated with the device. Therefore, to improve the accuracy of color restoration, it is necessary to find a more accurate correspondence between the input color information and the output color information and apply this mapping relationship to the image to be measured.

[0064] Specifically, such as Figure 6 As shown, by setting the chromaticity value to The standard target color is collected by CCD image acquisition equipment and the relevant color value is obtained , to obtain and The exact mapping relationship between them is the conversion algorithm M. The chromaticity value is The color original to be tested is collected by the CCD image acquisition device to obtain the image color record value RGB. RGB is then corrected by the conversion algorithm M to obtain the accurate restoration color , making and As close as possible. 、 、 The red component, green component and blue component respectively In this embodiment, optionally, feature extraction is performed on the inspection image and the template image respectively, including: Extract edge features and / or macro deformation features of the inspection image and the template image.

[0065] Specifically, after performing image noise filtering, color restoration, image enhancement, and / or dual-channel histogram equalization on the inspection and template images, image features extracted from them include edge features and macro-deformation features. Edge features are used to identify defects such as cracks and clogged drains. Macro-deformation features are used to identify slope deformation, as well as the nature, trend, and extent of the deformation.

[0066] In this embodiment, the optional steps include: The Canny edge detection algorithm is used to extract edge features of the inspection image and the template image; And / or, a detection algorithm based on Lucas-Kanade optical flow is used to extract macroscopic deformation features of the inspection image and the template image.

[0067] Specifically, there are many methods for extracting edge features, such as the Canny edge detection algorithm, the Laplacian edge detection algorithm, the Gini edge detection algorithm, and the like.

[0068] Among them, the Canny edge detection algorithm includes the following steps: 1) Use a Gaussian filter to smooth the image to remove noise. A Gaussian filter is a linear filter that can smooth the image and reduce noise.

[0069] 2) Use the Sobel operator to calculate the gradient of the smoothed image. The gradient can help us find obvious edges in the image.

[0070] 3) On the gradient image, non-maximum suppression is performed to eliminate redundant responses caused by edge detection. This process can make the edges more refined.

[0071] 4) Use dual-threshold detection to determine which edges are true edges and which are noise. Dual-threshold detection divides the gradient image into three components: strong edges, weak edges, and noise. If the gradient value of a pixel is greater than a high threshold, it is considered a strong edge. If the gradient value of a pixel is between the high and low thresholds, it is considered a weak edge. If the gradient value of a pixel is less than the low threshold, it is considered noise. Typically, the difference between the high and low thresholds is called the threshold difference.

[0072] 5) Connect strong edges and weak edges to form a complete edge line. The connection method is usually to link adjacent weak edges to the strong edge by tracing the path of the strong edge.

[0073] Canny edge detection can detect edges with high precision and high recall, and can eliminate noise and unnecessary edge responses.

[0074] The Laplacian edge detection algorithm consists of the following steps: 1) Convert the image to grayscale image; 2) Perform Laplace filtering on the grayscale image to obtain the Laplace response image. You can use the following template to filter the image: ; Among them, the template center point is 4, which means that the pixel value change rate around this pixel point is large, which may be an edge position.

[0075] 3) Binarize the Laplace response image, mark pixels with values ​​higher than a certain threshold as edge pixels, and mark pixels with values ​​lower than the threshold as non-edge pixels.

[0076] 4) Perform non-maximum suppression on the pixels marked as edge pixels to remove redundant edge pixels. The following methods can be used to suppress edge pixels: Specifically, for each pixel, calculate the gradient magnitude of the pixels in its neighborhood along the gradient direction and find the maximum and second-largest values. If the gradient magnitude of the current pixel is greater than the maximum value in the neighborhood, it means that the pixel may be an edge pixel. Otherwise, the pixel is marked as a non-edge pixel.

[0077] The Laplace edge detection algorithm can detect details and textures in images, which has certain advantages for complex image scenes. In addition, the Laplace edge detection algorithm can control the number and quality of edges by adjusting the threshold, which has a certain degree of flexibility.

[0078] The Gini edge detection algorithm consists of the following steps: 1) Read the image and convert it to grayscale.

[0079] 2) Calculate the histogram of the grayscale image to obtain the distribution of pixel values.

[0080] 3) Normalize the histogram and scale the pixel values ​​to between 0 and 1.

[0081] 4) Calculate the Gini coefficient, which is used to describe the uneven distribution of pixel values. The calculation formula of the Gini coefficient is as follows: ; in, Indicates that the pixel value is The proportion of pixels in the image indicates the unevenness of the pixel value distribution. The Gini coefficient ranges from 0 to 1. A larger value indicates a more uneven distribution of pixel values, which may indicate an edge location.

[0082] The Gini coefficient is thresholded and pixels with a Gini coefficient higher than a certain threshold are marked as edge pixels, and pixels with a Gini coefficient lower than the threshold are marked as non-edge pixels.

[0083] Non-maximum suppression is performed on pixels marked as edge pixels to remove redundant edge pixels.

[0084] The Gini edge detection algorithm can detect details and textures in images, which has certain advantages for complex image scenes. In addition, the Gini edge detection algorithm can control the number and quality of edges by adjusting the threshold, which has a certain degree of flexibility.

[0085] First, the template matching algorithm is used, and then the template image and the inspection image are respectively matched with the three edge detection algorithms mentioned above. Figure 7 The inspection image and template image shown in the figure are used for edge detection, and the detection results are as follows: Figure 8 As shown in the figure. From top to bottom, the edge information maps are obtained by using the Canny edge detection algorithm, the Laplacian edge detection algorithm, and the Gini edge detection algorithm. The left side is the edge information map corresponding to the template image, and the right side is the edge information map corresponding to the inspection image. Figure 8 Among the effect diagrams of the three different methods, it can be clearly seen from the changes in contour information and target area that the target change area of ​​Canny edge detection is the most obvious and the contour information is also the clearest.

[0086] Considering that the inspection images in actual engineering applications are not ideal, a series of weather factors and shooting factors are added to the disease simulation images to analyze their effects on the captured images, such as shadow, exposure, image blur, shadow + blur, and shadow + exposure.

[0087] Through comparative testing experiments on the Gini edge detection algorithm, the Laplace edge detection algorithm and the Canny edge detection algorithm, it was determined that the target area can be seen between the binary images for subsequent operations. The results are shown in the following table.

[0088] Table 1 Comparative experiment of edge detection algorithms (drain blockage) Table 2 Comparative experiment of edge detection algorithms (surface cracks) A total of 18 images were used for testing, and the algorithm implemented using Canny edge detection had a higher accuracy rate of 0.83. Therefore, this embodiment uses the Canny edge detection algorithm for edge feature extraction.

[0089] Similarly, there are many methods for extracting macro deformation features, such as the detection algorithm based on Hough linear transform, the detection algorithm based on SIFT feature matching, and the detection algorithm based on Lucas-Kanade optical flow.

[0090] Based on the Hough line transform detection algorithm, morphological operations, such as opening and closing, are first applied to smooth the image and remove noise. The image is then binarized to facilitate edge detection. Edges in the image are detected using the Canny edge detection algorithm. Morphological operations are performed on the edges to fill in any missing parts and remove unnecessary details. Straight lines in the image are detected using the Hough line transform algorithm. Based on the line detection results, the deformation of the frame beam is calculated and the corresponding metrics are generated.

[0091] The detection algorithm based on Scale-Invariant Feature Transform (SIFT) feature matching is a commonly used computer vision method for detecting and matching scale-invariant feature points in an image. The SIFT algorithm extracts key points and descriptors from an image and performs feature point matching. The main steps include: 1) By constructing a Gaussian pyramid, the image is blurred at multiple scales and the extreme points in the scale space, i.e., key point candidates, are found at different scales.

[0092] 2) Near the extreme points of the scale space, the interpolation method is used to accurately locate the key points to obtain the accurate position and scale of the key points.

[0093] 3) For each keypoint, a dominant direction is assigned to it by calculating the direction histogram of the local image gradient to enhance the rotation invariance of the feature.

[0094] 4) With the key point as the center, a feature descriptor is constructed in the local area around it. The descriptor is usually based on information such as the gradient direction and amplitude near the key point. 5) Match the feature descriptors extracted from the two images. Common methods include nearest neighbor matching and ratio testing. Nearest neighbor matching compares each feature descriptor with the feature descriptor in the other image and selects the most similar one as the matching pair. Ratio testing compares the distance ratio between the most similar feature descriptor and the second most similar feature descriptor to determine the reliability of the match.

[0095] 6) Screen and eliminate feature point matching results. Use some criteria and techniques to eliminate false matches.

[0096] The Lucas-Kanade optical flow algorithm is a classic computer vision algorithm used to estimate the temporal motion of objects in an image sequence. The algorithm is based on a set of assumptions: the position of pixels in an image varies little between frames, and neighboring pixels have similar motion. The Lucas-Kanade optical flow algorithm estimates the motion vectors of pixels by solving a least-squares problem to obtain the trajectory of the object in the image. The main steps include: 1) Select some stable feature points from the image sequence as tracking targets. Common feature point detection methods include Harris corner detection and Shi-Tomasi corner detection.

[0097] 2) In the first frame, the selected feature points are marked or their feature descriptors are extracted. These feature points are then tracked in subsequent frames, i.e., corresponding feature points are found between adjacent frames.

[0098] 3) At each feature point, a brightness error equation is constructed to describe the pixel motion. This equation is based on a first-order approximation of the grayscale value and assumes that adjacent pixels have similar motion.

[0099] 4) By minimizing the sum of squares of the brightness error equation, a linear system of equations is obtained. This system of linear equations is solved using the least squares method to estimate the motion vector of each feature point.

[0100] 5) Calculate the motion vector (displacement) of each feature point based on the solved linear equations. Various methods can be used to solve linear equations, such as singular value decomposition (SVD) or Gaussian elimination.

[0101] 6) Based on certain criteria and thresholds, abnormal or unreliable motion vectors are filtered out. Common filtering methods include removing points with excessive or insufficient motion speed, or eliminating false matches based on the consistency of motion vectors.

[0102] Because color images require a lot of computation, the color of each pixel in a color image is determined by three components: R, G, and B. The value range of each component is between 0-255. So for a computer, a pixel in a color image has a range of 256*256*256=16777216 colors. A grayscale image is a special color image with the same R, G, and B components. For a computer, the range of a pixel is only 256, 0-255. The information content of a color image is too large, and when performing image recognition, it is actually sufficient to use the information in the grayscale image. Therefore, in order to increase the computing speed, the image is first converted into a grayscale image, such as Figure 9 shown.

[0103] Then, the feature corner points detected on the original image are obtained by calling the feature location function in OpenCV, and then the Lucas-Kanade optical flow algorithm is implemented using the CalcOpticalFlowPyrLK method. The Lucas-Kanade optical flow algorithm is used to detect the offset changes between the original image and the diseased image in macroscopic diseases, such as Figure 10 Show.

[0104] In this embodiment, optionally, an image difference algorithm is used to obtain a change feature image of the inspection image, including: An image difference algorithm is used to obtain a difference image between the inspection image and the template image, and the difference image is subjected to an adaptive threshold binarization process; An image erosion and dilation operation is performed on the result of the adaptive threshold binarization process to obtain a change feature image of the inspection image.

[0105] Specifically, the inspection image and the template image can be set as a continuous frame video sequence, and the difference between the inspection image and the template image can be calculated using an inter-frame difference algorithm. The specific steps include: 1) Read the video sequence and convert each frame into a grayscale image.

[0106] 2) For each pixel, calculate the difference in pixel values ​​between adjacent frames to obtain a differential image. The difference in each pixel can be calculated using the following formula: ; in, Indicates that the coordinates are at Frame and The difference in pixel values ​​between frames, Indicates in The coordinates in the frame are The pixel value of is the absolute value function.

[0107] 3) Binarize the difference image to remove unnecessary noise and detail information, and convert the difference in pixel values ​​into binary 0 or 1.

[0108] 4) Perform motion detection on the binarized differential image. Methods such as connected domain analysis, edge detection, and template matching can be used to detect and track moving objects.

[0109] The advantage of the inter-frame difference algorithm is that it is simple and easy to implement. It does not require complex algorithms and models and can perform motion detection and tracking in real time. The inter-frame difference algorithm is sensitive to lighting changes and noise. To improve the stability and robustness of the algorithm, the pixel values ​​less than 10 after the difference can be set to 0 to remove noise points. The result is as follows: Figure 12 shown. Figure 12 From left to right in the figure are the difference image, the result of the adaptive threshold binarization, the image erosion and dilation result, and the final change feature image. Through the inter-frame difference algorithm, it can be found that in addition to the area where the simulated disease occurs, there are other transformed areas. These other areas will have a certain impact on the final detection results. Therefore, the difference image can be further processed by adaptive threshold binarization, and the part of the image threshold setting value below 60 is set to zero. The effect is as follows Figure 12 As shown. At this time, only the target area remains, but the target area is not clear enough. The result after the adaptive threshold binarization process can be further processed by image erosion and dilation. The effect is shown in Figure 12. Figure 12 It can be clearly seen that the white area has expanded a lot, which makes subsequent processing more convenient. Finally, the obtained white area can be used as a change feature image for disease identification.

[0110] like Figure 2 The flowchart of the inspection image defect recognition method shown in FIG. 1 includes: Step S1: using the above inspection image feature detection method to obtain a change feature image of the inspection image; Step S2: Based on the change feature image, the diseases in the inspection image are classified and identified using a trained disease classification model.

[0111] Specifically, first, the above-mentioned inspection pattern feature detection method can be used to extract a change feature image from the currently acquired inspection pattern. Then, the change feature image can be input into a trained disease classification model to classify and identify diseases in the inspection image.

[0112] In this embodiment, optionally, a ConvNext model is used as the disease classification model.

[0113] Specifically, there are many classification algorithms currently, such as the AlexNet neural network classification algorithm, the ResNet classification algorithm, and the ConvNext classification algorithm.

[0114] The AlexNet neural network classification algorithm is an improvement on the LeNet network, with a deeper network structure, enabling the algorithm to better learn high-dimensional image features. Its feature extraction network structure consists of five convolutional layers, and its classification structure consists of three fully connected layers. Image features are extracted through multiple convolutional layers. Within a single convolutional layer, there are typically many kernels of the same size. The kernel width and height are typically the same, and the depth is the same as the number of channels.

[0115] The first two convolutional layers are followed by overlapping max pooling layers. The third, fourth, and fifth convolutional layers are directly connected. The fifth convolutional layer is followed by an overlapping max pooling layer, whose output is fed into a series of two fully connected layers. The second fully connected layer feeds a softmax classifier with 1000 class labels. A nonlinear ReLU activation function is applied after all convolutional and fully connected layers. The nonlinear ReLU activation functions of the first and second convolutional layers are followed by a local normalization step before pooling.

[0116] Using a simple approach to gradually deepen the network layers can cause many problems, such as vanishing and exploding gradients. The ResNet classification algorithm, however, introduces a residual network structure. The main functions of this residual structure include shortcut connections and identity mapping. Quick connections enrich the possibilities for residuals, while identity mapping allows the network to become deeper. The identity mapping is designed to facilitate the effects of skip connections and activation functions.

[0117] ConvNext is a convolutional neural network (CNN) model for image classification tasks. This model is based on the ideas of ResNet and SENet models, combining the residual network (ResNet) and the attention mechanism (SENet) to improve the performance of the model. The ConvNext model structure is shown in the figure below. Figure 13 shown.

[0118] The basic structure of the ConvNext model consists of a series of convolutional layers, batch normalization layers, and activation functions, as well as the residual blocks of ResNet and the attention modules of SENet. The residual blocks help solve the vanishing gradient problem in deep neural networks, while the attention modules help the model better focus on important features in the image.

[0119] The ConvNext model's training consists of two phases. In the first phase, the model preprocesses the training data using data augmentation techniques such as random cropping and horizontal flipping, and trains using the cross-entropy loss function. In the second phase, the model is fine-tuned using a larger image resolution to further improve model performance.

[0120] In the prediction phase, the ConvNext model extracts features from the input image through convolutional layers and residual blocks, then uses an attention module to weight the extracted features to focus on the most important features. Finally, the model maps the features to category probabilities through global average pooling and fully connected layers for classification prediction.

[0121] In general, the ConvNext model combines the advantages of ResNet and SENet models and is trained through techniques such as data augmentation and fine-tuning to achieve higher performance in image classification tasks.

[0122] To verify the classification effectiveness of the ConvNext model, we used a custom slope disease dataset, pre-trained weights trained on Imagenet1000 on ConvNext_tiny, and then trained the entire model using ConvNext. We then classified and identified the change regions, and finally counted the number of each of the three categories. We then calculated the length, width, area, and aspect ratio of each change region.

[0123] The results of AlexNet, ResNet50, and ConvNext training are compared, as shown in the following table. Whether comparing the training loss results or the test accuracy, ConvNext has the best effect. Therefore, the present invention adopts the ConvNext model for disease classification and recognition.

[0124] In this embodiment, optionally, a transfer learning strategy is adopted to train the ConvNext model.

[0125] Specifically, since the primary prerequisite for training slope surface defect recognition and detection models is a sufficient set of images, and currently there are insufficient datasets with manually annotated defect features, a transfer learning strategy can be used to train the ConvNext model to address the overfitting problem faced by convolutional neural networks when training with small samples. This effectively extracts deep features from images and reduces the problem of model overfitting.

[0126] Transfer learning involves retraining a pre-trained model using the image dataset of the detection area to obtain a classification model specific to that area. This typically involves freezing the parameters of different layers and then fine-tuning the parameters of other layers. Depending on the layers frozen, various transfer strategies can be implemented. For example, training only the last three layers of the model, training the last three layers and an adjacent convolutional block, or training all layers of the network model can be employed.

[0127] Among them, only the last three layers of the model are trained. During the secondary training process, the convolutional layer and pooling layer of the pre-trained model are fixed. That is, these two layers are only used for image feature extraction and do not participate in the full training of the model. Only the last three fully connected layers are adjusted and trained.

[0128] Training the final three layers and the adjacent convolutional block during secondary training fixes the parameters of the structural layer and trains only the final three layers of the model. Extracting deep, abstract features specific to inspection images is difficult. Therefore, training the final three layers and the adjacent convolutional block improves the model.

[0129] Training all layers of the network model is to open all structural layers of the model during the secondary training process, and train and adjust the parameters of all structural layers of the pre-trained model.

[0130] While keeping other parameters unchanged, we selected different migration strategies to compare the ConvNext model. Scheme 1 involves training only the last three layers of the model, Scheme 2 involves training the last three layers and an adjacent convolutional block, and Scheme 3 involves training all layers of the network model. The training results are shown in the following table: Compared to the two migration methods of Schemes 1 and 2, which continuously fine-tune the parameters of all layers, the validation set accuracy of the ConvNext model has been greatly improved, reaching 98.9%. The validation set accuracy of the migration strategies of Schemes 1 and 2 has also reached 93.4% and 90.9%, respectively. Therefore, the present invention adopts a transfer learning strategy for all layers of the trained network model to train the ConvNext model.

[0131] To validate the robustness of the defect recognition method provided by this invention, we tested slope cracks captured under three conditions: normal, distorted, and blurred images. A total of 88 images of slope cracks were tested, with 66 correct detections, 4 false detections, and 18 missed detections, for an accuracy rate of 75%. Detailed information is shown in the table below.

[0132] The proposed defect recognition method was then used to test slope drainage ditches under normal conditions, varying shade, varying exposure, and blurred images. The method accurately captured the areas and information related to defect changes. A total of 84 drainage ditch images were tested, with 59 correct detections, 19 false detections, and 19 missed detections, for an accuracy rate of 70.24%. Detailed information is shown in the table below.

[0133] The proposed defect recognition method was used to test macroscopic slope deformation under normal conditions, varying shade, varying exposure, and blurred images. The method accurately captured the areas and information related to defect changes. A total of 92 images were tested for macroscopic deformation, with 80 correct detections, 5 false detections, and 7 missed detections, for an accuracy rate of 86.96%. Detailed information is shown in the table below.

[0134] The above test results show that the proposed fault identification method can maintain a high fault identification accuracy for different facilities under different interference conditions. It can effectively identify typical faults such as macroscopic slope deformation, structural faults, and drainage facility defects in inspection images.

[0135] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A method for detecting features of inspection images, characterized in that: include: Perform template matching on the acquired inspection image to determine a template image that matches the inspection image; Performing feature extraction on the inspection image and the template image respectively to obtain image features of the inspection image and the template image; According to the image features of the inspection image and the template image, an image difference algorithm is used to obtain a change feature image of the inspection image.

2. The inspection image feature detection method according to claim 1, characterized in that: Perform template matching on the acquired inspection images, including: A pyramid sampling template matching algorithm is used to match the template image of the inspection image.

3. The inspection image feature detection method according to claim 1, characterized in that: Extracting features from the inspection image and the template image includes: The inspection image and the template image are subjected to image noise filtering processing, color restoration processing, image enhancement processing and / or dual-channel histogram equalization processing.

4. The inspection image feature detection method according to claim 3, characterized in that: include: Use statistical-based color restoration algorithms to perform color restoration on inspection images and template images; And / or, an image enhancement algorithm based on guided filtering Retinex is used to perform image enhancement processing on the inspection image and the template image.

5. The inspection image feature detection method according to claim 1, characterized in that: Feature extraction is performed on the inspection image and the template image respectively, including: Extract edge features and / or macro deformation features of the inspection image and the template image.

6. The inspection image feature detection method according to claim 5, characterized in that: include: The Canny edge detection algorithm is used to extract edge features of the inspection image and the template image; And / or, a detection algorithm based on Lucas-Kanade optical flow is used to extract macroscopic deformation features of the inspection image and the template image.

7. The inspection image feature detection method according to claim 1, characterized in that: The image difference algorithm is used to obtain the change feature image of the inspection image, including: An image difference algorithm is used to obtain a difference image between the inspection image and the template image, and the difference image is subjected to an adaptive threshold binarization process; An image erosion and dilation operation is performed on the result of the adaptive threshold binarization process to obtain a change feature image of the inspection image.

8. A method for identifying defects in inspection images, characterized in that: include: Adopting the inspection image feature detection method according to any one of claims 1 to 7 to obtain a change feature image of the inspection image; Based on the change feature image, the diseases in the inspection image are classified and identified using a trained disease classification model.

9. The inspection image defect recognition method according to claim 8, characterized in that: The ConvNext model was used as the disease classification model.

10. The inspection image defect recognition method according to claim 8, characterized in that: The ConvNext model is trained using a transfer learning strategy.

Images