Wind turbine blade image accurate splicing and defect positioning method based on deep learning

Abstract

This invention discloses a deep learning-based method for accurate image stitching and defect localization of wind turbine blades, belonging to the field of image stitching and defect localization technology. It uses a U-Net model for semantic segmentation of wind turbine blade images to avoid background interference. Then, the SIFT algorithm is used for keypoint detection, achieving high-precision feature point matching, significantly improving matching accuracy. Simultaneously, a KD-Tree structure is combined to match the foreground image of the blade area, accelerating the matching process of high-dimensional features and improving matching efficiency. This method is suitable for processing large-scale image data and significantly improves computational efficiency and real-time performance. Finally, the RANSAC algorithm is used to estimate the homography matrix, and stitching is performed based on the homography matrix to avoid mismatches, improving stitching quality and facilitating application and promotion.

Description

Technical Field

[0001] This invention belongs to the field of image stitching and defect localization technology, specifically involving a method for accurate image stitching and defect localization of wind turbine blades based on deep learning. Background Technology

[0002] A wind turbine is an electrical device that converts wind energy into mechanical work, which drives a rotor to rotate and ultimately outputs alternating current. A wind turbine generally consists of a rotor, generator, deflector, tower, speed limiting safety mechanism, and energy storage device. When the rotor rotates under the influence of wind, it converts the kinetic energy of the wind into the mechanical energy of the rotor shaft, which in turn drives the generator to rotate and generate electricity. The rotor consists of blades and a hub. Because the blades are exposed to harsh environments such as sandstorms for extended periods, they are prone to defects such as protective layer peeling and cracks. Traditional manual inspection methods suffer from high labor intensity, low efficiency, and limited coverage.

[0003] In recent years, with the advancement and development of technology, drone technology has been introduced into wind turbine inspection. By acquiring images of the blades using high-definition cameras, inspection efficiency and defect identification capabilities have been improved. However, drones capture multiple local images, and to obtain global information about the entire blade, these local images need to be stitched together to obtain a complete panoramic image. However, the blade surface has poor texture and uniform color, and the drone's wide field of view during flight results in complex backgrounds, making image stitching challenging. Currently, commonly used image stitching methods include the LORB algorithm and the SURF (Speed ​​Up Robust Feature) algorithm. The LORB algorithm combines the ORB (Oriented Fast and Rotated Brief) algorithm with FAST feature point detection and the BRIEF descriptor, offering advantages such as fast computation speed and suitability for real-time applications. However, the ORB algorithm suffers from low matching accuracy when processing images with simple textures or large lighting variations, easily leading to mismatches and affecting stitching quality. The SURF algorithm has good robustness in feature point detection and description, capable of handling certain scale and rotation changes, but its computational load is high when processing high-resolution images, making it difficult to meet real-time requirements.

[0004] Therefore, how to provide an effective technical solution to address the problems of low matching accuracy, easy mismatch, and poor stitching quality in existing technologies, as well as the large computational load and difficulty in meeting real-time requirements when processing high-resolution images, has become an urgent technical challenge to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide a method for accurate image stitching and defect localization of wind turbine blades based on deep learning, in order to solve the above-mentioned problems existing in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for accurate image stitching and defect localization of wind turbine blades based on deep learning, comprising: Acquire several images of the wind turbine blades, including the fan blade portion; The U-Net model is used to perform semantic segmentation on each wind turbine blade image to obtain multiple foreground images of the blade area; The YoLov8 algorithm is used to detect defects in the foreground image of each fan blade to obtain the final detection result. The final detection result includes the defect category and the coordinates of the defect bounding box corresponding to each foreground image of the fan blade. The SIFT algorithm is used to detect key points in each foreground image of the fan blade, and the key points corresponding to each foreground image of the fan blade are obtained. Then, a key point descriptor is generated based on the key points corresponding to each foreground image of the fan blade. A KD-Tree is constructed based on key point descriptors. In the KD-Tree, foreground images of multiple fan blade parts are matched to obtain fan blade part matching images, which include multiple sets of matching point pairs. The RANSAC algorithm is used to correct each pair of matching points to obtain the homography matrix; The foreground images of the fan blade area are stitched together based on the homography matrix to obtain a stitched image; Based on the preset geometric transformation relationship, the coordinates of the defect boundary box are transformed to the coordinate system of the stitched image, and the coordinates of the defect boundary box are combined with the stitched image to obtain a panoramic defect fan-shaped image.

[0007] In one possible design, the SIFT algorithm is used to detect keypoints in each foreground image of the fan blade area, obtaining the keypoints corresponding to each foreground image of the fan blade area. Then, a keypoint descriptor is generated based on the keypoints corresponding to each foreground image of the fan blade area, including: Based on the Gaussian kernel function, Gaussian kernels of different scales are convolved with each foreground image of the fan blade to obtain the scale space representation of each foreground image of the fan blade. Based on the foreground images of the fan blade corresponding to adjacent scale space representations, a Gaussian difference pyramid is constructed. In the difference of Gaussian pyramid, each pixel in the foreground image of the fan blade is compared with its multiple neighboring pixels in the same layer of the same scale and its multiple neighboring pixels in the adjacent layers at the corresponding positions in adjacent scales. If the pixel is greater than each neighboring pixel in the same layer and the neighboring pixels in the adjacent layers, or if the pixel is less than each neighboring pixel in the same layer and the neighboring pixels in the adjacent layers, then the pixel is regarded as a candidate key point. Calculate the Hessian matrix of each candidate keypoint, and obtain the trace and determinant of each candidate keypoint based on the Hessian matrix. If the ratio of the trace and determinant of a candidate keypoint is less than a preset edge threshold, then retain the candidate keypoint to obtain multiple keypoints. On the scale image of each key point, a circular neighborhood is selected with the key point as the center and based on a preset radius, and the first gradient magnitude and first direction of all pixels in the circular neighborhood are calculated. A gradient direction histogram is constructed based on the first gradient magnitude and first direction of all pixels, and the gradient values ​​are weighted based on the Gaussian weighting function to obtain a weighted gradient direction histogram. The direction corresponding to the peak in the weighted gradient direction histogram is taken as the main direction of the key point. Centered on the keypoint, a preset first number of pixel neighborhoods are extracted, and all pixel neighborhoods are divided into a preset second number of sub-regions. Within each sub-region, the gradient magnitude and direction of each pixel relative to the main direction of the keypoint are calculated to obtain the second gradient magnitude and second direction. The second direction is quantized into eight directions, and the second gradient magnitude is weighted using a Gaussian weighting function to obtain multiple sub-region orientation histograms. Based on the sub-region orientation histogram of each sub-region, multiple feature descriptor vectors are obtained. Each feature descriptor vector is normalized to obtain multiple keypoint descriptors.

[0008] In one possible design, a KD-Tree is constructed based on keypoint descriptors. Within the KD-Tree, foreground images of multiple fan blade regions are matched to obtain a fan blade region matching image. This matching image includes multiple sets of matching point pairs, including: Select a foreground image of a fan blade as the target image, and extract all key point descriptors from the target image. Iterate through the dimensions of all key point descriptors and calculate the variance of the dimensions. Select the dimension with the largest variance as the segmentation dimension. Recursively segment all key point descriptors based on the segmentation dimension until the preset segmentation termination condition is reached to obtain the KD-Tree. Select another foreground image of the fan blade as the query image, and perform a nearest neighbor search in the KD-Tree based on all key point descriptors in the query image to obtain the nearest neighbor distance and the second nearest neighbor distance. The distance ratio is calculated based on the nearest neighbor distance and the second nearest neighbor distance. If the distance ratio is less than a preset distance threshold, the matching point pair is retained. If the distance ratio is greater than or equal to the preset distance threshold, the matching point pair is removed. The process involves reselecting a foreground image of the fan blade area as the target image until all key point descriptors in the foreground images of the fan blade area are matched, thus obtaining a matched image of the fan blade area.

[0009] In one possible design, the RANSAC algorithm is used to refine each pair of matching points, resulting in a homography matrix, including: Randomly select a preset number of matching point pairs from each set of matching point pairs to obtain an initial point pair sample set; An initial homography matrix is ​​obtained based on an initial point pair sample set. The initial point pair sample set is then transformed into the target image based on the initial homography matrix to obtain predicted coordinates. The reprojection error is calculated based on the predicted coordinates and the coordinates in the target image. The reprojection error is compared with the preset error threshold. If the reprojection error is less than the preset error threshold, the point corresponding to the predicted coordinates is taken as an interior point. If the reprojection error is greater than or equal to the preset error threshold, the point corresponding to the predicted coordinates is taken as an exterior point. The number of interior points corresponding to the initial homography matrix is ​​calculated. Repeat the above steps until the preset iteration termination condition is reached, and obtain a set of interior point numbers corresponding to multiple initial homography matrices. Select the initial homography matrix with the largest number of interior points, and obtain the homography matrix based on the interior points of the selected initial homography matrix.

[0010] In one possible design, the foreground images of the fan blade area are stitched together based on the homography matrix to obtain a stitched image, including: The foreground image of the fan blade is selected as the first image, and the target image is selected as the second image; Based on the homography matrix, the corner points of the first image are transformed into the coordinate system of the second image, and the transformed corner points of the first image and the corner points of the second image are merged. Calculate the merged bounding box and use it as the canvas boundary; The first image is then subjected to homography transformation again based on the homography matrix, and then the transformed first image is translated using the translation matrix to obtain the translated first image. The overlapping areas between the translated first and second images are weighted and fused, and the non-overlapping areas between the translated first and second images are stitched together to obtain a stitched image.

[0011] In one possible design, a weighted fusion is performed on the overlapping region between the translated first and second images, including: Based on the translated first and second images, the overlapping area between the two is obtained; Within the overlapping region, calculate the weight mask for each pixel in the translated first image and the weight mask for each pixel in the second image, respectively. A weighted fusion is performed based on the pixel values ​​of the overlapping regions in the first image, the pixel values ​​of the overlapping regions in the second image, the weight mask of each pixel in the translated first image, and the weight mask of each pixel in the second image.

[0012] In one possible design, based on a preset geometric transformation relationship, the coordinates of the defect bounding box are transformed to the coordinate system of the stitched image, and the coordinates of the defect bounding box are combined with the stitched image to obtain a panoramic defect fan-shaped image, including: Extract the coordinates of multiple corner points of the defect bounding box, and transform each corner point coordinate based on the homography matrix to obtain multiple mapped coordinates; The location bounding boxes in the stitched image are obtained based on multiple mapped coordinates. The location bounding boxes and defect categories are drawn on the stitched image to obtain a panoramic defect fan-shaped image.

[0013] Secondly, the present invention provides a deep learning-based system for precise image stitching and defect localization of wind turbine blades, comprising: The image acquisition module is used to acquire several images of wind turbine blades, including the fan blade portion; The image segmentation module is used to perform semantic segmentation on each wind turbine blade image using the U-Net model to obtain multiple foreground images of the blade parts; The defect detection results are used to perform defect detection on each foreground image of the fan blade using the YoLov8 algorithm to obtain the final detection results. The final detection results include the defect category and defect bounding box coordinates corresponding to each foreground image of the fan blade. The key point detection module is used to perform key point detection on each foreground image of the fan blade using the SIFT algorithm, obtain the key points corresponding to each foreground image of the fan blade, and generate key point descriptors based on the key points corresponding to each foreground image of the fan blade. The image matching module is used to construct a KD-Tree based on key point descriptors. In the KD-Tree, foreground images of multiple fan blade parts are matched to obtain fan blade part matching images, which include multiple sets of matching point pairs. The matching correction module is used to correct each pair of matching points using the RANSAC algorithm to obtain the homography matrix; The image stitching module is used to stitch together the foreground image of the fan blade area based on the homography matrix to obtain a stitched image; The transformation and combination module is used to transform the coordinates of the defect boundary box to the coordinate system of the stitched image according to the preset geometric transformation relationship, and combine the coordinates of the defect boundary box with the stitched image to obtain a panoramic defect fan-shaped image.

[0014] Thirdly, the present invention provides a computer device comprising a memory, a processor, and a transceiver connected in sequence and communication, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the deep learning-based method for accurate image stitching and defect localization of wind turbine blades as described in the first aspect above.

[0015] Fourthly, the present invention provides a computer-readable storage medium storing instructions that, when executed on a computer, perform the deep learning-based method for precise image stitching and defect localization of wind turbine blades as described in the first aspect above.

[0016] Fifthly, the present invention provides a computer program product containing instructions that, when the instructions are executed on a computer, cause the computer to perform the deep learning-based method for precise image stitching and defect localization of wind turbine blades as described in the first aspect above.

[0017] The beneficial effects of this invention are as follows: This invention discloses a method for accurate image stitching and defect localization of wind turbine blades based on deep learning. First, several images of wind turbine blades containing the blade portion are acquired. Then, the U-Net model is used to perform semantic segmentation on each wind turbine blade image, resulting in multiple foreground images of the blade portion. The YoLov8 algorithm is used to detect defects in each foreground image of the blade portion, obtaining the defect category and defect bounding box coordinates. The SIFT algorithm is used to detect key points in each foreground image of the blade portion, obtaining the key points corresponding to each foreground image of the blade portion. Finally, based on the foreground images of each blade portion... Keypoint descriptors are generated for the corresponding keypoints. A KD-Tree is constructed based on the keypoint descriptors. In the KD-Tree, foreground images of multiple fan blade parts are matched to obtain matching images of the fan blade parts. The matching images of the fan blade parts include multiple sets of matching point pairs. The RANSAC algorithm is used to correct each set of matching point pairs to obtain a homography matrix. The foreground images of the fan blade parts are stitched together based on the homography matrix to obtain a stitched image. According to the preset geometric transformation relationship, the coordinates of the defect bounding box are transformed to the coordinate system of the stitched image. The coordinates of the defect bounding box are combined with the stitched image to obtain a panoramic defect fan blade image. Semantic segmentation is performed using the U-Net model to avoid interference from the background. Then, the SIFT algorithm is used for keypoint detection, which significantly improves matching accuracy and precision, achieving high-precision matching of feature points. At the same time, the KD-Tree structure is combined to match the foreground image of the fan blade area, which can accelerate the matching process of high-dimensional features and improve matching efficiency. This method is suitable for processing large-scale image data and significantly improves computational efficiency and real-time performance. Subsequently, the RANSAC algorithm is used to estimate the homography matrix, and the image is stitched based on the homography matrix to avoid mismatches, improve stitching quality, and facilitate application and promotion. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a deep learning-based method for precise image stitching and defect localization of wind turbine blades, as provided in an embodiment of the present invention. Figure 2 A block diagram of a deep learning-based wind turbine blade image precision stitching and defect localization system provided in an embodiment of the present invention; Figure 3 A structural block diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0020] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0021] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0022] Example: like Figure 1 As shown, the first aspect of this embodiment provides a method for accurate image stitching and defect localization of wind turbine blades based on deep learning. This method can be executed, but is not limited to, by a computer device or virtual machine with sufficient computing resources, such as a personal computer or smartphone, or by a virtual machine. The method for accurate image stitching and defect localization of wind turbine blades includes, but is not limited to, the following steps: S1. Obtain several images of the wind turbine blades, including the fan blade section; In practice, high-definition cameras mounted on drones are used to take multiple images of the wind turbine blades along their surface during the inspection flight. The drones are designed to capture images of key areas of the wind turbine blades, such as the leading edge, trailing edge, blade root, and blade tip. Each image of the wind turbine blade contains sufficient information about the blade surface texture to provide the raw data foundation for subsequent operations.

[0023] S2. Use the U-Net model to perform semantic segmentation on each wind turbine blade image to obtain multiple foreground images of the blade area; It should be noted that the U-Net model is a deep convolutional neural network architecture designed specifically for image segmentation tasks. It includes an encoder and a decoder. The principle of semantic segmentation using the U-Net model is to extract contextual information by shrinking the path and to achieve precise localization by expanding the path. Finally, multi-scale information is fused through feature concatenation to complete pixel-level classification, thereby obtaining multiple foreground images of the fan blade area.

[0024] S3. Use the YoLov8 algorithm to perform defect detection on the foreground image of each fan blade to obtain the final detection result. The final detection result includes the defect category and defect bounding box coordinates corresponding to the foreground image of each fan blade. It should be noted that the YoLov8 algorithm is a single-stage object detection algorithm that can directly predict defect bounding boxes and defect categories from the input image. After obtaining the defect bounding boxes and defect categories, the NMS (Non-maximum suppression) algorithm is used to remove overlapping defect bounding boxes to obtain the final detection result, which significantly improves the accuracy and efficiency of defect detection. Both the YoLov8 algorithm and the NMS algorithm are existing technologies and will not be described in detail here.

[0025] S4. Use the SIFT algorithm to detect key points in each foreground image of the fan blade area, obtain the key points corresponding to each foreground image of the fan blade area, and generate key point descriptors based on the key points corresponding to each foreground image of the fan blade area. Specifically, in step S4, the SIFT algorithm is used to detect key points in each foreground image of the fan blade area, obtaining the key points corresponding to each foreground image of the fan blade area, and generating key point descriptors based on the key points corresponding to each foreground image of the fan blade area, including: S41. Based on the Gaussian kernel function, convolve Gaussian kernels of different scales with the foreground image of each fan blade part to obtain the scale space representation of the foreground image of each fan blade part. Based on the foreground images of the fan blade parts corresponding to adjacent scale space representations, construct the Gaussian difference pyramid. S42. In the difference of Gaussian pyramid, each pixel in the foreground image of the fan blade is compared with its multiple neighboring pixels in the same layer of the same scale and its multiple neighboring pixels in the adjacent layers at the corresponding positions in adjacent scales. If the pixel is greater than each neighboring pixel in the same layer and the neighboring pixels in the adjacent layers, or if the pixel is less than each neighboring pixel in the same layer and the neighboring pixels in the adjacent layers, then the pixel is regarded as a candidate key point. S43. Calculate the Hessian matrix of each candidate key point, and obtain the trace and determinant of each candidate key point based on the Hessian matrix. If the ratio of the trace and determinant of a candidate key point is less than a preset edge threshold, then retain the candidate key point to obtain multiple key points. S44. On the scale image of each key point, select a circular neighborhood centered on the key point and based on a preset radius, and calculate the first gradient magnitude and first direction of all pixels in the circular neighborhood; S45. Construct a gradient direction histogram based on the first gradient magnitude and first direction of all pixels, and weight the gradient values ​​based on the Gaussian weighting function to obtain a weighted gradient direction histogram. Take the direction corresponding to the peak in the weighted gradient direction histogram as the main direction of the key point. S46. Using the keypoint as the center, extract a preset first number of pixel neighborhoods, divide all pixel neighborhoods into a preset second number of sub-regions, calculate the gradient magnitude and direction of each pixel relative to the main direction of the keypoint within each sub-region, obtain the second gradient magnitude and second direction, quantize the second direction into eight directions, use a Gaussian weighting function to weight the second gradient magnitude, obtain multiple sub-region orientation histograms, obtain multiple feature descriptor vectors based on the sub-region orientation histograms of each sub-region, normalize each feature descriptor vector to obtain multiple keypoint descriptors.

[0026] It should be noted that the expression for the Gaussian kernel function in this embodiment is: In the formula, and For the input vector, For scale parameters, It is an exponential function. The foreground image of the fan blade is pi. Each foreground image of the fan blade is convolved with Gaussian kernels of different scales to obtain a scale-space representation, where the expression for the scale-space representation is: Then, by calculating the difference between the foreground images of the fan blade region in adjacent scale-space representations, a Gaussian difference pyramid is constructed. The expression for the Gaussian difference pyramid is: In the formula, k is the scale multiplication factor. s is the number of layers within each octave. An octave represents the set of images obtained by using different Gaussian kernels at a specific image size. In this embodiment, in the difference-of-gaussian pyramid, each pixel has 8 neighboring pixels in the same scale image, and 9 neighboring pixels in the upper layer and 9 neighboring pixels in the lower layer, for a total of 26 neighboring pixels. Each pixel is compared with 26 neighboring pixels to detect local extrema. Local extrema are used as candidate keypoints. If a pixel has more than 26 neighboring pixels or less than 26 neighboring pixels, then the pixel is used as a candidate keypoint.

[0027] Furthermore, the Hessian matrix of the difference of Gaussians function at each candidate keypoint is calculated. Based on the edge response criterion, the Hessian matrix is ​​used to eliminate unstable candidate keypoints located at the edges. The expression for the Hessian matrix is ​​as follows: In the formula, Let be the second partial derivative of the Gaussian difference function in the x-direction. Let be the mixed partial derivative of the Gaussian difference function. Let be the second-order partial derivative of the Gaussian difference function in the y-direction. Based on the Hessian matrix, the trace and determinant of each candidate keypoint are obtained. The expressions for the trace and determinant are as follows: ,in, For traces, For determinant, and All are eigenvalues ​​of the Hessian matrix, and Greater than In this embodiment, the expression for the edge response criterion is: In the formula, If the ratio of trace to determinant is less than a preset edge threshold (i.e. If the ratio of trace to determinant is greater than or equal to a preset edge threshold, then the candidate key point is retained, and it is considered that the candidate key point is relatively stable. If the ratio of trace to determinant is greater than or equal to a preset edge threshold, then the candidate key point is removed, thus obtaining the key point set.

[0028] For each keypoint in the keypoint set, a circular neighborhood is selected centered on the keypoint in the Gaussian difference image at its scale, based on a preset radius. In this embodiment, the preset radius is [missing information]. It calculates the first gradient magnitude and first direction of all pixels in the circular neighborhood, and the expression for the first gradient magnitude is: In the formula, Let be the gradient magnitude at pixel (x, y). and Let (x, y) be the values ​​of the pixels adjacent to each other in the horizontal direction. and Let (x, y) be the value of the pixel's adjacent pixels in the vertical direction. The expression for the first direction is: In the formula, This represents the gradient direction at pixel (x, y), i.e., the orientation of the edge. The arctangent function is used. Then, a 36-bar gradient direction histogram is constructed based on the first gradient magnitude and first direction of all pixel values. The gradient values ​​are then weighted using a Gaussian weighting function. The expression for the resulting weighted gradient direction histogram is: In the formula, Let be the Gaussian weighting function, and its expression is: The direction corresponding to the peak value in the weighted gradient direction histogram is taken as the main direction of the key point. In one possible design, if there is a secondary peak that exceeds 80% of the peak energy, an additional key point is created, and the direction of the secondary peak is assigned to the additional key point as the main direction of the additional key point.

[0029] In this embodiment, the preset first quantity is 16. 16, the preset second quantity is 4 4. A total of 16 sub-regions were obtained. Within each of these 16 sub-regions, the second gradient magnitude and second direction of each pixel relative to the principal direction of the keypoint were calculated. The second direction was quantized into eight directions: 0°, 45°, 90°, 135°, 180°, 225°, 270°, and 315°. A Gaussian weighting function was used to weight the second gradient magnitudes, resulting in multiple sub-region orientation histograms. The expression for the Gaussian weighting function is as follows: In the formula, For each sub-region, the position and orientation of the orientation histogram are interpolated to improve the robustness of the processing, based on the window size. In this embodiment, the sub-region orientation histogram is an eight-dimensional orientation histogram, meaning that each of the 16 sub-regions generates a corresponding eight-dimensional orientation histogram, resulting in a total of 128-dimensional feature descriptor vectors. These 128-dimensional feature descriptor vectors are normalized, and then bounded. This bounded feature descriptor vector is then normalized again to obtain the final keypoint descriptors. The purpose of this second normalization is to improve robustness to changes in illumination. The normalization expression is as follows: In the formula, The normalized feature descriptor For feature descriptor vectors, The modulus of the feature descriptor vector is used to limit the normalized feature descriptor vector based on a preset maximum value, which is 0.2.

[0030] S5. Construct a KD-Tree based on key point descriptors. In the KD-Tree, match the foreground images of multiple fan blade parts to obtain a fan blade part matching image. The fan blade part matching image includes multiple sets of matching point pairs. Specifically, in step S5, a KD-Tree is constructed based on key point descriptors. Within the KD-Tree, foreground images of multiple fan blade regions are matched to obtain a fan blade region matching image. This matching image includes multiple sets of matching point pairs, including: S51. Select a foreground image of a fan blade as the target image, extract all key point descriptors of the target image, traverse the dimensions of all key point descriptors, calculate the variance of the dimensions, select the dimension with the largest variance as the segmentation dimension, recursively segment all key point descriptors based on the segmentation dimension until the preset segmentation termination condition is reached, and obtain KD-Tree. S52. Select another foreground image of the fan blade as the query image, and perform a nearest neighbor search in the KD-Tree based on all key point descriptors in the query image to obtain the nearest neighbor distance and the second nearest neighbor distance; S53. Calculate the distance ratio based on the nearest neighbor distance and the second nearest neighbor distance. If the distance ratio is less than the preset distance threshold, retain the matching point pair. If the distance ratio is greater than or equal to the preset distance threshold, discard the matching point pair. S54. Reselect a foreground image of the fan blade area as the target image until the key point descriptors in all foreground images of the fan blade area are matched, and obtain the matched image of the fan blade area.

[0031] It should be noted that, for a set of keypoint descriptors in a foreground image of the fan blade area, multiple KD-Trees are constructed to establish an efficient search structure. All dimensions of the 128-dimensional keypoint descriptors are traversed, and the variance of each dimension is calculated. The dimension corresponding to the largest variance is used as the segmentation dimension. The keypoint descriptors are segmented according to the segmentation dimension to divide the keypoint descriptors into left and right subtrees. The segmentation termination condition in this embodiment is that all keypoint descriptors have been segmented, thus stopping the segmentation operation after reaching the preset segmentation termination condition, resulting in the constructed KD-Tree. In this embodiment, the number of KD-Trees constructed is 5. The preset segmentation termination condition includes, but is not limited to, the number of keypoint descriptors contained in a node being less than a set value. For each keypoint descriptor in the query image, a parallel search is performed in the KD-Tree. Specifically, starting from the root node, the search proceeds down to the leaf node, then backtracks to check other branches, checking if there are closer neighbors in adjacent regions to find the nearest and second-nearest keypoint descriptors in the target image by Euclidean distance. The distance uses Euclidean distance, and the specific calculation expression is: In the formula, To query keypoint descriptors in an image, For key point descriptors in the target image, This represents the current dimension number. It is a Euclidean distance.

[0032] Find the nearest and second nearest neighbor keypoint descriptors in the target image for each keypoint descriptor in the query image. Calculate the ratio of the distance between each keypoint descriptor in the query image and its corresponding nearest neighbor, and the distance between each keypoint descriptor in the query image and its corresponding second nearest neighbor. If the distance ratio is less than a preset distance threshold, the query image is discarded; if the distance ratio is greater than or equal to the preset distance threshold, the query image is retained. The expression for calculating the distance ratio is as follows: ,in, This represents the distance between a keypoint descriptor and its corresponding nearest neighbor keypoint descriptor. The distance between a keypoint descriptor and its corresponding next nearest neighbor keypoint descriptor is used to treat the query image as a fan-blade region matching image, wherein the fan-blade region matching image includes multiple sets of matching point pairs.

[0033] S6. Use the RANSAC algorithm to correct each pair of matching points to obtain the homography matrix; In step S6, the RANSAC algorithm is used to correct each pair of matching points to obtain the homography matrix, including: S61. Randomly select a preset number of matching point pairs from each set of matching point pairs to obtain an initial point pair sample set; S62. Obtain the initial homography matrix based on the initial point pair sample set, transform the initial point pair sample set into the target image based on the initial homography matrix to obtain the predicted coordinates, and calculate the reprojection error based on the predicted coordinates and the coordinates in the target image; S63. Based on the comparison between the preset error threshold and the reprojection error, if the reprojection error is less than the preset error threshold, the point corresponding to the predicted coordinates is taken as an interior point; if the reprojection error is greater than or equal to the preset error threshold, the point corresponding to the predicted coordinates is taken as an exterior point. Calculate the number of interior points corresponding to the initial homography matrix. S64. Repeat the above steps until the preset iteration termination condition is reached, and obtain a set of interior point counts corresponding to multiple initial homography matrices. Select the initial homography matrix with the largest number of interior points, and obtain the homography matrix based on the interior points of the selected initial homography matrix.

[0034] It should be noted that since the homography matrix has 8 degrees of freedom, at least 4 pairs of points are required for the solution. Therefore, the preset number of point pairs is 4, and the initial homography matrix is ​​a 3x3 matrix. A 3x3 matrix is ​​used to describe the projection transformation relationship between two planes. Based on the initial homography matrix, the initial point pair sample set is transformed into the target image to obtain the predicted coordinates. The specific transformation process is expressed as follows: In the formula, Let be the initial homography matrix. Let the coordinates of the initial point be relative to the sample set. For the predicted coordinates, where , The reprojection error is calculated by calculating the predicted coordinates and key point coordinates. The expression for the reprojection error is as follows: In the formula, Given the coordinates in the target image, the reprojection error is compared with a preset error threshold to determine whether the point corresponding to the predicted coordinate is an interior point or an exterior point. If the reprojection error is less than the preset error threshold, the point corresponding to the predicted coordinate is taken as an interior point; if the reprojection error is greater than or equal to the preset error threshold, the point corresponding to the predicted coordinate is taken as an exterior point. This selects the initial homography matrix with the most interior points, and then re-estimates the homography matrix based on the initial homography matrix.

[0035] In one possible design, the preset iteration termination condition is reaching the required number of iterations, which is determined by the following expression: In the formula, The confidence level is 0.99 in this embodiment. The proportion of interior points is estimated a priori, and m is the minimum number of samples. In this embodiment, m=4. The number of iterations is obtained through the above expression, thereby terminating the repetition of steps S61-S63.

[0036] S7. Based on the homography matrix, the foreground images of the fan blade area are stitched together to obtain a stitched image; Specifically, in step S7, the foreground image of the fan blade area is stitched together based on the homography matrix to obtain a stitched image, including: S71. Select the foreground image of the fan blade area as the first image, and the target image as the second image; S72. Based on the homography matrix, transform the corner points of the first image to the coordinate system of the second image, and merge the transformed corner points of the first image and the corner points of the second image; S73. Calculate the merged bounding box and use it as the canvas boundary; S74. Perform homography transformation on the first image again based on the homography matrix, and then use the translation matrix to translate the transformed first image to obtain the translated first image; S75. Perform weighted fusion on the overlapping area between the translated first image and the second image, and stitch the non-overlapping area between the translated first image and the second image to obtain a stitched image.

[0037] It should be noted that before performing weighted fusion on the overlapping area between the translated first image and the second image in step S75, the translated first image is uniformly adjusted to 640. A resolution of 512 was used to make the final stitched image 1024. 1024 resolution.

[0038] In practice, a homography matrix is ​​used to transform the four corner points of the first image to the coordinate system of the second image. The four corner points of the first image are... The expression after transformation to the coordinate system of the second image is: The four corner points of the second image are By merging the four corner points of the first image and the four corner points of the second image, we obtain... Therefore, the calculation expression for the merged bounding box is: This yields the canvas boundaries. The first image is then transformed using a homography matrix, and the transformed first image is translated using a translation matrix to obtain the translated first image. The expression for the translation matrix is: The overlapping areas between the translated first and second images are weighted and fused, while the non-overlapping areas between the translated first and second images directly use the pixel values ​​of the corresponding images. For example, if only the pixel values ​​of the first image are available, the pixel values ​​of the first image are used; if only the pixel values ​​of the second image are available, the pixel values ​​of the second image are used, thus obtaining the stitched image.

[0039] Furthermore, in step S75, a weighted fusion is performed on the overlapping area between the translated first image and the second image, including: S75.1. Based on the translated first and second images, obtain the overlapping area between them; S75.2. Within the overlapping region, calculate the weight mask for each pixel in the translated first image and the weight mask for each pixel in the second image, respectively; S75.3. Perform weighted fusion based on the pixel values ​​of the overlapping regions in the first image, the pixel values ​​of the overlapping regions in the second image, the weight mask of each pixel in the translated first image, and the weight mask of each pixel in the second image.

[0040] It should be noted that the expression for the weight mask of each pixel in the translated first image is: In the formula, Given a Gaussian weighting function, the expression for the weight mask of each pixel in the second image is: In the formula, Given a Gaussian weighting function, the expression for weighted fusion is: In the formula, Let be the pixel value of the first translated image at position (x, y). The pixel value of the second image at position (x, y).

[0041] S8. Based on the preset geometric transformation relationship, transform the coordinates of the defect boundary box to the coordinate system of the stitched image, and combine the coordinates of the defect boundary box with the stitched image to obtain a panoramic defect fan-shaped image.

[0042] Specifically, in step S8, according to a preset geometric transformation relationship, the coordinates of the defect boundary box are transformed to the coordinate system of the stitched image, and the coordinates of the defect boundary box are combined with the stitched image to obtain a panoramic defect fan-shaped image, including: S81. Extract the coordinates of multiple corner points of the defect bounding box, and transform each corner point coordinate based on the homography matrix to obtain multiple mapped coordinates; S82. Based on multiple mapped coordinates, obtain the location bounding box in the stitched image, and draw the location bounding box and defect category on the stitched image to obtain a panoramic defect fan-shaped image.

[0043] In practice, the coordinates of the defect bounding box are obtained, and the corner coordinates of the defect bounding box are extracted. The corner coordinates are transformed based on the homography matrix to obtain the mapped coordinates. The smallest horizontal rectangle formed by the mapped coordinates is calculated in the stitched image. This smallest horizontal rectangle is the location bounding box of the defect on the stitched image. All defect bounding box coordinates and defect categories are drawn on the stitched image to obtain the final panoramic defect fan blade image, realizing the location of the defect on the complete fan blade, which facilitates timely defect elimination by maintenance personnel and makes defect detection more intuitive.

[0044] like Figure 2 As shown, the second aspect of this embodiment provides a deep learning-based system for precise image stitching and defect localization of wind turbine blades, including: The image acquisition module is used to acquire several images of wind turbine blades, including the fan blade portion; The image segmentation module is used to perform semantic segmentation on each wind turbine blade image using the U-Net model to obtain multiple foreground images of the blade parts; The defect detection results are used to perform defect detection on each foreground image of the fan blade using the YoLov8 algorithm to obtain the final detection results. The final detection results include the defect category and defect bounding box coordinates corresponding to each foreground image of the fan blade. The key point detection module is used to perform key point detection on each foreground image of the fan blade using the SIFT algorithm, obtain the key points corresponding to each foreground image of the fan blade, and generate key point descriptors based on the key points corresponding to each foreground image of the fan blade. The image matching module is used to construct a KD-Tree based on key point descriptors. In the KD-Tree, foreground images of multiple fan blade parts are matched to obtain fan blade part matching images, which include multiple sets of matching point pairs. The matching correction module is used to correct each pair of matching points using the RANSAC algorithm to obtain the homography matrix; The image stitching module is used to stitch together the foreground image of the fan blade area based on the homography matrix to obtain a stitched image; The transformation and combination module is used to transform the coordinates of the defect boundary box to the coordinate system of the stitched image according to the preset geometric transformation relationship, and combine the coordinates of the defect boundary box with the stitched image to obtain a panoramic defect fan-shaped image.

[0045] The working process, working details and technical effects of the deep learning-based wind turbine blade image precision stitching and defect localization system provided in the second aspect of this embodiment can be found in the deep learning-based wind turbine blade image precision stitching and defect localization method described in the first aspect, and will not be repeated here.

[0046] like Figure 3 The third aspect of this embodiment provides a computer device, including a memory, a processor, and a transceiver connected in sequence for communication. The memory stores a computer program, the transceiver sends and receives messages, and the processor reads the computer program and executes the deep learning-based wind turbine blade image precision stitching and defect localization method described in the first aspect. Specifically, the memory may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; the processor may include, but is not limited to, an STM32F105 series microprocessor. Furthermore, the computer device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0047] The working process, working details and technical effects of the aforementioned computer equipment provided in the third aspect of this embodiment can be found in the method for accurate image stitching and defect localization of wind turbine blades based on deep learning described in the first aspect, and will not be repeated here.

[0048] The fourth aspect of this embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a computer, the method for accurate image stitching and defect localization of wind turbine blades based on deep learning as described in the first aspect is performed. The computer-readable storage medium refers to a carrier for storing data, and may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or Memory Sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0049] The working process, working details and technical effects of the aforementioned computer-readable storage medium provided in the fourth aspect of this embodiment can be found in the method for accurate image stitching and defect localization of wind turbine blades based on deep learning as described in the first aspect, and will not be repeated here.

[0050] The fifth aspect of this embodiment provides a computer program product, including a computer program or instructions, which, when executed by a computer, are used to implement the deep learning-based method for accurate image stitching and defect localization of wind turbine blades as described in the first aspect.

[0051] The working process, working details and technical effects of the aforementioned computer program product provided in this embodiment can be found in the method for accurate image stitching and defect localization of wind turbine blades based on deep learning as described in the first aspect, and will not be repeated here.

[0052] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for accurate image stitching and defect localization of wind turbine blades based on deep learning, characterized in that, include: Acquire several images of the wind turbine blades, including the fan blade portion; The U-Net model is used to perform semantic segmentation on each wind turbine blade image to obtain multiple foreground images of the blade area; The YoLov8 algorithm is used to detect defects in the foreground image of each fan blade to obtain the final detection result. The final detection result includes the defect category and the coordinates of the defect bounding box corresponding to each foreground image of the fan blade. The SIFT algorithm is used to detect key points in each foreground image of the fan blade, and the key points corresponding to each foreground image of the fan blade are obtained. Then, a key point descriptor is generated based on the key points corresponding to each foreground image of the fan blade. A KD-Tree is constructed based on key point descriptors. In the KD-Tree, foreground images of multiple fan blade parts are matched to obtain fan blade part matching images, which include multiple sets of matching point pairs. The RANSAC algorithm is used to correct each pair of matching points to obtain the homography matrix; The foreground images of the fan blade area are stitched together based on the homography matrix to obtain a stitched image; Based on the preset geometric transformation relationship, the coordinates of the defect boundary box are transformed to the coordinate system of the stitched image, and the coordinates of the defect boundary box are combined with the stitched image to obtain a panoramic defect fan-shaped image.

2. The method for accurate image stitching and defect localization of wind turbine blades based on deep learning according to claim 1, characterized in that, The SIFT algorithm is used to detect keypoints in the foreground image of each fan blade area, obtaining the keypoints corresponding to each foreground image of the fan blade area. Based on these keypoints, keypoint descriptors are generated, including: Based on the Gaussian kernel function, Gaussian kernels of different scales are convolved with each foreground image of the fan blade to obtain the scale space representation of each foreground image of the fan blade. Based on the foreground images of the fan blade corresponding to adjacent scale space representations, a Gaussian difference pyramid is constructed. In the difference of Gaussian pyramid, each pixel in the foreground image of the fan blade is compared with its multiple neighboring pixels in the same layer of the same scale and its multiple neighboring pixels in the adjacent layers at the corresponding positions in adjacent scales. If the pixel is greater than each neighboring pixel in the same layer and the neighboring pixels in the adjacent layers, or if the pixel is less than each neighboring pixel in the same layer and the neighboring pixels in the adjacent layers, then the pixel is regarded as a candidate key point. Calculate the Hessian matrix of each candidate keypoint, and obtain the trace and determinant of each candidate keypoint based on the Hessian matrix. If the ratio of the trace and determinant of a candidate keypoint is less than a preset edge threshold, then retain the candidate keypoint to obtain multiple keypoints. On the scale image of each key point, a circular neighborhood is selected with the key point as the center and based on a preset radius, and the first gradient magnitude and first direction of all pixels in the circular neighborhood are calculated. A gradient direction histogram is constructed based on the first gradient magnitude and first direction of all pixels, and the gradient values ​​are weighted based on the Gaussian weighting function to obtain a weighted gradient direction histogram. The direction corresponding to the peak in the weighted gradient direction histogram is taken as the main direction of the key point. Centered on the keypoint, a preset first number of pixel neighborhoods are extracted, and all pixel neighborhoods are divided into a preset second number of sub-regions. Within each sub-region, the gradient magnitude and direction of each pixel relative to the main direction of the keypoint are calculated to obtain the second gradient magnitude and second direction. The second direction is quantized into eight directions, and the second gradient magnitude is weighted using a Gaussian weighting function to obtain multiple sub-region orientation histograms. Based on the sub-region orientation histogram of each sub-region, multiple feature descriptor vectors are obtained. Each feature descriptor vector is normalized to obtain multiple keypoint descriptors.

3. The method for accurate image stitching and defect localization of wind turbine blades based on deep learning according to claim 1, characterized in that, A KD-Tree is constructed based on keypoint descriptors. Within the KD-Tree, foreground images of multiple fan blade regions are matched to obtain matching images of the fan blade regions. These matching images include multiple pairs of matching points, including: Select a foreground image of a fan blade as the target image, and extract all key point descriptors from the target image. Iterate through the dimensions of all key point descriptors and calculate the variance of the dimensions. Select the dimension with the largest variance as the segmentation dimension. Recursively segment all key point descriptors based on the segmentation dimension until the preset segmentation termination condition is reached to obtain the KD-Tree. Select another foreground image of the fan blade as the query image, and perform a nearest neighbor search in the KD-Tree based on all key point descriptors in the query image to obtain the nearest neighbor distance and the second nearest neighbor distance. The distance ratio is calculated based on the nearest neighbor distance and the second nearest neighbor distance. If the distance ratio is less than a preset distance threshold, the matching point pair is retained. If the distance ratio is greater than or equal to the preset distance threshold, the matching point pair is removed. The process involves reselecting a foreground image of the fan blade area as the target image until all key point descriptors in the foreground images of the fan blade area are matched, thus obtaining a matched image of the fan blade area.

4. The method for accurate image stitching and defect localization of wind turbine blades based on deep learning according to claim 3, characterized in that, The RANSAC algorithm is used to correct each pair of matching points to obtain the homography matrix, including: Randomly select a preset number of matching point pairs from each set of matching point pairs to obtain an initial point pair sample set; An initial homography matrix is ​​obtained based on an initial point pair sample set. The initial point pair sample set is then transformed into the target image based on the initial homography matrix to obtain predicted coordinates. The reprojection error is calculated based on the predicted coordinates and the coordinates in the target image. The reprojection error is compared with the preset error threshold. If the reprojection error is less than the preset error threshold, the point corresponding to the predicted coordinates is taken as an interior point. If the reprojection error is greater than or equal to the preset error threshold, the point corresponding to the predicted coordinates is taken as an exterior point. The number of interior points corresponding to the initial homography matrix is ​​calculated. Repeat the above steps until the preset iteration termination condition is reached, and obtain a set of interior point numbers corresponding to multiple initial homography matrices. Select the initial homography matrix with the largest number of interior points, and obtain the homography matrix based on the interior points of the selected initial homography matrix.

5. The method for accurate image stitching and defect localization of wind turbine blades based on deep learning according to claim 3, characterized in that, The foreground images of the fan blade region are stitched together based on the homography matrix to obtain a stitched image, including: The foreground image of the fan blade is selected as the first image, and the target image is selected as the second image; Based on the homography matrix, the corner points of the first image are transformed into the coordinate system of the second image, and the transformed corner points of the first image and the corner points of the second image are merged. Calculate the merged bounding box and use it as the canvas boundary; The first image is then subjected to homography transformation again based on the homography matrix, and then the transformed first image is translated using the translation matrix to obtain the translated first image. The overlapping areas between the translated first and second images are weighted and fused, and the non-overlapping areas between the translated first and second images are stitched together to obtain a stitched image.

6. The method for accurate image stitching and defect localization of wind turbine blades based on deep learning according to claim 5, characterized in that, Weighted fusion is performed on the overlapping region between the translated first and second images, including: Based on the translated first and second images, the overlapping area between the two is obtained; Within the overlapping region, calculate the weight mask for each pixel in the translated first image and the weight mask for each pixel in the second image, respectively. A weighted fusion is performed based on the pixel values ​​of the overlapping regions in the first image, the pixel values ​​of the overlapping regions in the second image, the weight mask of each pixel in the translated first image, and the weight mask of each pixel in the second image.

7. The method for accurate image stitching and defect localization of wind turbine blades based on deep learning according to claim 1, characterized in that, Based on a preset geometric transformation relationship, the coordinates of the defect boundary box are transformed to the coordinate system of the stitched image, and the coordinates of the defect boundary box are combined with the stitched image to obtain a panoramic defect fan-shaped image, including: Extract the coordinates of multiple corner points of the defect bounding box, and transform each corner point coordinate based on the homography matrix to obtain multiple mapped coordinates; The location bounding boxes in the stitched image are obtained based on multiple mapped coordinates. The location bounding boxes and defect categories are drawn on the stitched image to obtain a panoramic defect fan-shaped image.

8. A deep learning-based system for precise image stitching and defect localization of wind turbine blades, used to implement the method described in any one of claims 1 to 7, characterized in that, include: The image acquisition module is used to acquire several images of wind turbine blades, including the fan blade portion; The image segmentation module is used to perform semantic segmentation on each wind turbine blade image using the U-Net model to obtain multiple foreground images of the blade parts; The defect detection results are used to perform defect detection on each foreground image of the fan blade using the YoLov8 algorithm to obtain the final detection results. The final detection results include the defect category and defect bounding box coordinates corresponding to each foreground image of the fan blade. The key point detection module is used to perform key point detection on each foreground image of the fan blade using the SIFT algorithm, obtain the key points corresponding to each foreground image of the fan blade, and generate key point descriptors based on the key points corresponding to each foreground image of the fan blade. The image matching module is used to construct a KD-Tree based on key point descriptors. In the KD-Tree, foreground images of multiple fan blade parts are matched to obtain fan blade part matching images, which include multiple sets of matching point pairs. The matching correction module is used to correct each pair of matching points using the RANSAC algorithm to obtain the homography matrix; The image stitching module is used to stitch together the foreground image of the fan blade area based on the homography matrix to obtain a stitched image; The transformation and combination module is used to transform the coordinates of the defect boundary box to the coordinate system of the stitched image according to the preset geometric transformation relationship, and combine the coordinates of the defect boundary box with the stitched image to obtain a panoramic defect fan-shaped image.

9. A computer device, characterized in that, The device includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the deep learning-based method for accurate image stitching and defect localization of wind turbine blades as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or the instructions are executed by the computer, they implement the deep learning-based method for accurate image stitching and defect localization of wind turbine blades as described in any one of claims 1 to 7.

Images