High-resolution fan blade image small defect detection method and related equipment

Abstract

The application discloses a kind of high-resolution fan blade image small defect detection method and related equipment, wherein method includes: obtaining high-resolution blade image, and the image is preprocessed, and initial dataset is constructed;Defect classification network is constructed in combination with field prior knowledge, and defect classification network is trained;Defect classification network after training is used, whether the image region exists defect is judged and defect category probability is calculated, and defect saliency picture is generated;Defect detection network is constructed, and initial dataset is used to train defect detection network;Defect saliency picture and the down-sampling image of original drawing are input into defect detection network, respectively utilize detection head to obtain the detection frame coordinates of defect corresponding category, then according to original drawing coordinate, detection result fusion is carried out.The present application is based on field prior knowledge, and saliency image can be screened in high-resolution blade image the area with highest defect probability in priority, facilitate the initial positioning of small defect, improve detection efficiency and precision.

Description

Technical Field

[0001] This invention relates to the field of defect detection, and in particular to a method and related equipment for detecting small defects in high-resolution wind turbine blade images. Background Technology

[0002] Wind energy, as a clean energy source, generates electricity through the rotation of wind turbine blades and is widely distributed in my country. To maximize the utilization of wind energy, large wind turbines are often deployed in remote and open areas, and can also be deployed offshore. However, changes in the natural environment can cause various types of defects on the surface of wind turbine blades, affecting the power generation efficiency of the turbines. If detection and repair are not timely, it can lead to turbine failure or blade breakage. Traditional blade inspection mainly relies on manual labor, using the inspector's experience and equipment such as telescopes to determine the location and type of defects. However, manual inspection is inefficient and cannot detect small cracks and localized corrosion. In addition, there are sensor detection methods. For example, during turbine shutdown, acoustic emission sensors or optical sensors are fixed to the blades to collect acoustic and optical signals for diagnosis. However, sensors are susceptible to vibration and airflow, resulting in noise interference in the collected data and causing diagnostic inaccuracies. In contrast, using drones with cameras to directly photograph wind turbine blades and extracting high-resolution blade images through image processing can more quickly determine the location of defects.

[0003] In recent years, the development of machine vision and deep learning has provided a new approach for detecting small defects in wind turbine blades. Most wind turbine blade detection networks are built based on convolutional neural networks (CNNs) or Transformers. However, convolutional computations limit the model's receptive field, leading to detail loss in high-resolution blade images with unclear defect boundaries. Transformers, on the other hand, can dynamically adjust the receptive field, acquiring global relevant information and positional encoding capabilities. However, as the pixel count increases, the computational complexity of Transformers increases quadratically, and since most defects are small, the computation often results in redundant information. A commonly used method is the sliding window approach, which segments the image before detection. However, this method leads to excessive computation and can cause detection errors by cutting a complete defect in half.

[0004] Patent application CN119067917 A discloses a vision-based intelligent wind turbine blade damage detection method, but the infrared thermal imaging images it uses differ fundamentally in color distribution from images captured by traditional cameras, and it also fails to address the impact of defect boundary features on the network. Patent application CN118762179 A discloses a method, system, equipment, and medium for detecting leading-edge corrosion of wind turbine blades, but it fails to address the detection of other blade defects such as surface scratches, cracks, and contamination. Patent application CN119107533A discloses a method for detecting surface defects in offshore wind turbine blades based on an improved YOLOV8, but it only considers wind turbine blades with low-resolution images and fails to address defect detection in high-resolution blade images.

[0005] Meanwhile, processing high-resolution blade images for defect detection presents numerous challenges. The complex background interference from wind turbines situated in diverse geographical environments (such as reduced contrast and masking of defect information due to lighting variations) makes it difficult for networks to accurately distinguish small defects from blade boundaries. Wind turbine blades, due to their location, are also susceptible to extreme weather conditions, resulting in diverse defect sizes and shapes, making defect feature extraction difficult and requiring comprehensive consideration of both local defects and overall texture information. While high-resolution images contain a wealth of detailed information, which aids in defect detection, it also increases the computational complexity of image processing. Furthermore, excessively high image resolution and large image data volumes lead to processing time delays, directly impacting real-time detection performance. Training corresponding defect detection models using complex networks also consumes significant computational resources. Summary of the Invention

[0006] In order to at least partially solve one of the technical problems existing in the prior art, the purpose of this invention is to provide a method and related equipment for detecting small defects in high-resolution wind turbine blade images.

[0007] The first technical solution adopted in this invention is:

[0008] A method for detecting small defects in high-resolution wind turbine blade images includes the following steps:

[0009] High-resolution leaf images were acquired and preprocessed to construct an initial dataset;

[0010] A defect classification network is constructed by combining prior domain knowledge, and the network is trained using an initial dataset.

[0011] The trained defect classification network is used to determine whether there are defects in the image region and calculate the probability of the defect category, thereby generating a defect saliency image;

[0012] Construct a defect detection network and train it using the initial dataset;

[0013] The defect saliency image and the downsampled image of the original image are input into the defect detection network. The detection head is used to obtain the detection box coordinates of the corresponding defect category. Then, the detection results are fused according to the coordinates of the original image to calculate the defect confidence.

[0014] Furthermore, the process of acquiring high-resolution leaf images and preprocessing the images to construct an initial dataset includes:

[0015] High-resolution images of wind turbine blades were obtained by using drones.

[0016] A contrast-based spatial channel attention mechanism is used to process high-resolution blade images to generate images that highlight defect boundaries.

[0017] The Labelme software was used to annotate the images that highlighted the defect boundaries, and the image data was expanded using a preset method to obtain an initial dataset.

[0018] Furthermore, the defect classification network operates as follows:

[0019] The high-resolution leaf image is divided into multiple small image patches. Feature extraction is performed on each image patch to obtain local feature information, as shown in the following expression:

[0020] x0=[x cls ||x patch ]+x pos

[0021] In the formula, x0 represents local feature information, x cls For feature category information, x patch For block image information, x pos Embedded at a local location;

[0022] Based on prior knowledge of the domain, image patches are matched and filtered, and the filtered image patches are then stitched together in order.

[0023] The local feature information of the selected image patches is input into the Channel Region Attention (CRA) mechanism to obtain region feature information, as shown in the following expression:

[0024] y k =x k-1 +CRA(LN(x k-1 ))

[0025] In the formula, LN represents normalization calculation, CRA represents multi-attention calculation of the channel region on the normalized features; k represents the one-dimensional position index of the feature; x k-1 This represents the local feature information of the (k-1)th image patch;

[0026] Using the feature information of each region as input, a channel-local attention mechanism is used to calculate the local features, as shown in the following expression:

[0027] z (i,j) =y (i,j) +CLA(LN(y (i,j) ))

[0028] In the formula, (i,j) represents the spatial location index of the local feature within the region, and z (i,j) Indicates local feature output; y (i,j) Represents the input region feature information; CLA represents the multi-attention calculation of the normalized channel local features;

[0029] All local features are concatenated to obtain encoded global feature information, which is then sent to the Transformer decoder and fully connected layer for defect classification to obtain a defect saliency image.

[0030] Furthermore, the step of matching and filtering image patches based on domain prior knowledge includes:

[0031] A large number of images related to blade defects are collected through prior knowledge, the features of the prior knowledge are extracted, and the extracted prior knowledge features are stored in a feature library.

[0032] The similarity between the features of the image patch and the features in the feature library is calculated. If the similarity value is greater than a preset threshold, the image patch is determined to match the prior knowledge of the domain and is retained. Otherwise, the image patch is determined not to match the prior knowledge of the domain and is not retained.

[0033] Furthermore, the defect classification network is implemented using the ER-ViT model, and the training steps of the defect classification network include:

[0034] The initial dataset is downsampled and fed into the ER-ViT model for training.

[0035] Organize prior knowledge of wind turbine blade defects, use the defect categories inferred from the model as a reference, compare with the labeled defects to make category corrections, and combine with prior knowledge to predict the coordinates of the defect area.

[0036] Each training image is segmented into defects, and a salient image is generated by combining the defect probability. The optimal defect region map is obtained through multiple iterations.

[0037] Furthermore, the defect detection network includes a high-resolution feature extraction module, a multi-scale micro-defect feature fusion module, a conventional detection head, and a high-resolution detection head;

[0038] The input to the high-resolution feature extraction module is a downsampled image of the original image and a defect saliency image; the features output by the high-resolution feature extraction module are transmitted to the multi-scale micro-defect feature fusion module for feature fusion to enhance the feature representation of small defects.

[0039] The defect area image is sent to a conventional detection head to determine the defect location and confidence level. Then, the downsampled image of the original image is sent to a high-resolution detection head for secondary detection. The high-resolution detection head first upsamples the input of the conventional detection head and then repeats the process of using a two-dimensional convolutional network to determine the defect location and confidence level.

[0040] Design a confidence threshold, select two image detection results with high confidence and save them, and then fuse the detection results.

[0041] Furthermore, the high-resolution feature extraction module includes dynamically deformable convolution and conventional convolution; the dynamically deformable convolution consists of three parts: dynamic convolution, fully connected layers, and a region bias module.

[0042] The dynamic convolution is used to generate basic feature maps, highlighting key features of the image; the dynamic convolution learns the features of the input image and dynamically generates convolution kernels to capture important information in the image.

[0043] The fully connected layer maps the input features to a high-dimensional space, generating parameters for calculating the region bias;

[0044] The region biasing module uses the output of the fully connected layer to generate spatial aggregation weights;

[0045] The formula for calculating dynamically deformable convolution is as follows:

[0046]

[0047] In the formula, x g As input features for the image, y g Dynamically deformable output features for each region; m gk This represents the spatial aggregation weights used in bias calculation, where p0 is the starting position of a predefined region in the image, and p k For the k-th position of the predefined region dynamic window, Δp gk The offset position of the dynamic window is set; the dynamic deformable output features of all regions are dynamically deformable. For different regions of the image, a dynamic window is set, and all features within the dynamic window are fully connected to obtain the total dynamic deformable output features of a single image.

[0048] The second technical solution adopted in this invention is:

[0049] An electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, and the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement a high-resolution wind turbine blade image small defect detection method as described above.

[0050] The third technical solution adopted in this invention is:

[0051] A computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement a high-resolution wind turbine blade image small defect detection method as described above.

[0052] The fourth technical solution adopted in this invention is:

[0053] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method for detecting small defects in high-resolution wind turbine blade images.

[0054] The beneficial effects of this invention are: based on prior knowledge of the domain, this invention uses saliency images to preferentially screen the areas with the highest defect probability in high-resolution blade images, which facilitates the initial location of small defects and improves detection efficiency and accuracy. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of the present invention or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating the high-resolution wind turbine blade image small defect detection method in an embodiment of the present invention;

[0057] Figure 2 This is a flowchart of image preprocessing in an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of a domain prior knowledge-guided defect classification network in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram of the defect detection network in an embodiment of the present invention;

[0060] Figure 5 This is a schematic diagram of the structure of the multi-scale micro-defect feature fusion module in an embodiment of the present invention;

[0061] Figure 6 This is a flowchart of the steps of the high-resolution wind turbine blade image small defect detection method in an embodiment of the present invention. Detailed Implementation

[0062] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0063] The terminology used in the embodiments of this application is for the purpose of describing specific embodiments only and is not intended to limit the embodiments of this application. The singular forms "a," "described," and "the" used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, unless otherwise expressly limited, terms such as "set," "install," and "connect" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this invention in conjunction with the specific content of the technical solution.

[0064] In the description of this application, it should be understood that the orientation descriptions, such as up, down, front, back, left, right, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0065] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0066] In the description of this application, "and / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the related objects before and after it are in an "or" relationship.

[0067] Terminology Explanation:

[0068] ViT: Vision Transformer, is a computer vision model based on a pure Transformer architecture. By segmenting images into block sequences and processing them using a self-attention mechanism, it demonstrates performance advantages over traditional convolutional neural networks (CNNs) after large-scale data pre-training.

[0069] ER-ViT: Enhanced Region Vision Transformer (Region ViT) is an improved visual Transformer model designed to enhance the performance of image sentiment analysis by enhancing effective region and context awareness.

[0070] To address the challenge of accurately classifying and locating small defects in high-resolution blade images, this invention proposes a method for detecting small defects in high-resolution wind turbine blade images. The method includes: (1) high-resolution blade image preprocessing; (2) high-resolution multi-scale feature extraction and fusion; (3) generation of defect saliency maps based on domain prior knowledge; (4) precise coordinate regression and confidence calculation for small defects; and (5) model training and prediction. Through the above image processing and defect detection process design, the features of high-resolution blade images can be effectively utilized to achieve rapid detection of small defects in the image. With the rapid development of defect detection technology, this invention provides in-depth research into blade small defect detection technology, enabling rapid and accurate classification and location of small defects, which is of significant practical importance for the safe and stable operation of wind turbines.

[0071] Specifically, the core components of this invention are a high-resolution image preprocessing method, a prior knowledge-guided defect classification network, and a defect detection network. Based on these three core components, the model is designed holistically. First, a high-resolution leaf dataset is constructed, and a contrast-based spatial channel attention mechanism image preprocessing method is used to highlight defect edge features. The images are then labeled to construct training, validation, and test sets. Second, a defect classification network is designed, incorporating prior knowledge fusion, saliency image generation, and ER-ViT (Enhanced Region Vision Transformer). Finally, a defect detection network is designed, including a high-resolution image feature extraction method, a multi-scale micro-defect feature fusion method, and a high-resolution detection head. The high-resolution image feature extraction method employs a combination of multi-layer dynamically deformable convolutions and conventional convolutions, making the model backbone more lightweight and reducing memory usage. The multi-scale micro-defect feature fusion method uses multi-scale features and fast pooling to improve the network's defect perception ability, thereby increasing defect prediction confidence. The detection head includes four scale detection heads, capable of regressing the accurate coordinates of defects, calculating the corresponding defect detection accuracy, and training the weights of the entire model.

[0072] Example 1

[0073] like Figure 1 and Figure 6 As shown, this embodiment provides a method for detecting small defects in high-resolution wind turbine blade images, including the following steps:

[0074] S1. Obtain high-resolution leaf images and preprocess the images to construct the initial dataset;

[0075] S2. Construct a defect classification network by combining domain prior knowledge, and train the defect classification network using the initial dataset;

[0076] S3. Using the trained defect classification network, determine whether there are defects in the image region and calculate the probability of the defect category to generate a defect saliency image;

[0077] S4. Construct a defect detection network and train it using the initial dataset;

[0078] S5. Input the defect saliency image and the downsampled image of the original image into the defect detection network, use the detection head to obtain the detection box coordinates of the corresponding defect category, and then fuse the detection results according to the coordinates of the original image to obtain the final detection result.

[0079] The following detailed explanation of this embodiment is provided in conjunction with the accompanying drawings and specific examples.

[0080] like Figure 1As shown in the figure, this embodiment provides a method for detecting small defects in high-resolution wind turbine blade surface images. The specific steps are as follows:

[0081] Step 1: Preprocess the acquired high-resolution leaf images and re-label them to construct the initial dataset.

[0082] As one implementation method, step 1 includes the following specific processes:

[0083] Step 1.1: Use a drone to photograph the wind turbine blades to obtain high-resolution images of the blades.

[0084] Step 1.2: Generate images that highlight boundaries using a contrast-based spatial channel attention mechanism.

[0085] Furthermore, a contrast-based spatial channel attention mechanism is used to weightedly fuse the original high-resolution image and the contrast-enhanced image.

[0086] For example, the matrix after reading the high-resolution image is high*width*channel(H*W*C). The contrast-enhanced image is obtained by operating on the read matrix using a spatial channel attention mechanism to obtain the spatial and channel information of the image (the result of (1*C*H*W) in batch 1). The result is then transformed from (1*C*H*W) to (H*W*C), which is the contrast-enhanced image. Then, through weighted calculation: 0.85*high-resolution image + 0.15*contrast-enhanced image = image with highlighted boundaries.

[0087] Step 1.3: Use Labelme software to annotate defects in the preprocessed images, and expand the dataset by flipping and mirroring. The process is as follows: Figure 2 As shown. Integrate the current data into the initial dataset.

[0088] Step 2: Construct ER-ViT and combine it with the domain prior knowledge of wind turbine blades to guide the network to pre-divide the areas where defects may exist, use the improved model to calculate the probability of defects in the area, and generate a saliency image.

[0089] In some embodiments, step 2 specifically includes the following steps:

[0090] Step 2.1: Construct ERVIT by combining domain prior knowledge, including region construction, local encoding, region encoding, local encoding, and global decoding.

[0091] Before inputting the image into ERVIT, the image is divided into n small image patches (e.g., n=128). These small image patches are input into ERVIT as local content, while the original high-resolution image is input into ERVIT as the original global content. Multiple local contents can be aggregated and constructed into region content through a stitching method. Guided by prior knowledge, local images are selected for stitching. The specific number of aggregations needs to be adjusted according to the actual situation. The process is as follows: Figure 3 As shown.

[0092] Step 2.2: Local Encoding: Encode small patches of the image using a self-attention mechanism to extract features. The specific expression is as follows:

[0093] x0=[x cls ||x patch ]+x pos

[0094] In the formula, x0 represents local feature information, x cls For feature category information, x patch For block image information, x pos Embedded at a local location.

[0095] Step 2.3: Organize prior knowledge about wind turbine blade defects, focusing on the blade locations where certain types of defects are likely to occur, the blade shooting angle, and the size distribution of blade defects, etc., for feature matching of local images. Local content that meets the matching requirements (i.e., small image patches) is retained, and the images are stitched together sequentially. The result of reading the stitched image here is the region label, while reading the original high-resolution image provides the global label, such as... Figure 3 As shown.

[0096] For example, the collection of domain prior knowledge takes wind turbine blades as an example, including information on the shape, size, color, and location of defects in the wind turbine blades, focusing on the blade locations where certain types of defects are likely to occur, the blade shooting angle, and the size distribution of blade defects, etc. Each small image patch is treated as local content, and feature matching is performed between the local content and the domain prior knowledge. Local content that meets the matching requirements is retained and the images are stitched together in sequence.

[0097] As one implementation method, feature matching is as follows:

[0098] 1) Prior knowledge refers to the information mentioned above. A large number of images related to blade defects are collected based on prior knowledge, and feature descriptions of the prior knowledge are extracted using the same method as local image feature extraction. These prior knowledge features are stored in a feature library.

[0099] 2) The BFMatcher algorithm iterates through each feature descriptor in the local image and searches for the most similar match in the prior knowledge feature descriptor set. Similarity is measured by calculating the Euclidean distance between feature descriptors. The smaller the distance, the more similar the features.

[0100] 3) Set a matching threshold to determine whether the features match. For each local image, if its matching degree with the prior knowledge features is lower than the set threshold, then the local image is considered not to match the prior knowledge; otherwise, local images with a matching degree higher than the threshold are retained.

[0101] Step 2.4: Feed all the selected and matched local features into the Channel Region Attention (CRA) mechanism to obtain the region feature information output. The specific expression is as follows:

[0102] y k =x k-1 +CRA(LN(x k-1 ))

[0103] In the formula, LN represents normalization calculation, and CRA represents multi-attention calculation of the channel region on the normalized features. k represents the one-dimensional position index of the feature. The feature information of each region is used as input to the local feature, and the channel local attention mechanism is calculated. The specific expression is as follows:

[0104] z (i,j) =y (i,j) +CLA(LN(y (i,j) ))

[0105] In the formula, (i,j) represents the spatial location index of the local feature within the region, and z (i,j) This indicates the output of local features.

[0106] Step 2.5: Concatenate all local features to obtain encoded global feature information, and then send it to the Transformer decoder and fully connected layer for defect classification.

[0107] Specifically, the Transformer decoder uses the encoder's output as keys and values, while the decoder's output serves as a query for attention computation. This allows the decoder to generate target sequences based on the encoder's information. The decoder's output is mapped to the target dimension through a linear layer, and finally, a Softmax function is used to classify defect features, calculating the probability of all possible defective ontology features.

[0108] Step 2.6: Downsample and segment the initial dataset into images, then feed it into ER-ViT for training to obtain ER-ViT based on blade defects. Using the defect category probability distribution of the images inferred by this model as a reference, the model compares the inferred defects with the labeled defects to correct the category. Combined with prior knowledge, the model then predicts the coordinates of the defect region. For example, corrosion often occurs at the leading edge of the blade, corresponding to the right side of the blade image. If the model determines that the image has corrosion defects, the network calculates and saves the coordinates of the right-side region and the defect category probability.

[0109] Step 2.7: Divide each training image into defects, generate salient images based on defect probabilities, and obtain the optimal defect region map through multiple iterations.

[0110] Step 3: The coordinates of the potentially defective regions shown in the saliency image and the downsampled image of the original image are sent to the high-resolution feature extraction module, and the multi-scale micro-defect feature fusion module is used to enable the model to learn multi-scale defect features. The defective regions and the original image are mapped to each other, and the detection head is used to obtain the coordinates of the detection boxes corresponding to the defect categories. Then, the results are fused based on the coordinates of the original image to calculate the defect confidence score.

[0111] In some embodiments, step 3 specifically includes the following steps:

[0112] Step 3.1: Construct a defect detection network, such as... Figure 4 As shown, the defect detection network consists of three key parts: a high-resolution feature extraction module, a multi-scale micro-defect feature fusion module, and a detection head.

[0113] Step 3.2: Transfer the initial dataset of downsampling (referring to image segmentation and image compression) and the defect region map obtained in Step 2.7 (containing coordinates mapped to the original image) to the high-resolution feature extraction module respectively.

[0114] Step 3.3: As Figure 4 As shown, the high-resolution feature extraction module consists of two parts: dynamically deformable convolution and regular convolution. The dynamically deformable convolution comprises three parts: dynamic convolution, fully connected layers, and a region biasing module. The output feature map obtained by the dynamic convolution is multiplied by the spatial aggregation weights obtained by the region biasing module, and then normalized along the dimension based on the image input features using the Softmax function.

[0115] Specifically, dynamic convolution is used to generate basic feature maps, highlighting key features of the image. It dynamically generates convolutional kernels by learning the features of the input image to capture important information. Fully connected layers map the input features to a high-dimensional space, generating parameters for calculating region biases. It receives the feature vector of the input image and outputs a set of high-dimensional feature representations through matrix multiplication and non-linear activation functions (such as ReLU). The region bias module utilizes the output of the fully connected layers to generate spatial aggregation weights. These weights are used to adjust the contribution of different regions to the final features, emphasizing important regions and suppressing irrelevant regions.

[0116] Specifically, the detailed process of dynamically deformable convolution is as follows:

[0117] 1) Dynamic convolution processing: The input image is first passed through a dynamic convolutional layer to generate a basic feature map.

[0118] 2) Feature mapping: The basic feature map is fed into the fully connected layer and mapped to a high-dimensional space to generate parameters for calculating the region bias.

[0119] 3) Spatial aggregation weights: The region bias module generates a spatial aggregation weight matrix based on the output of the fully connected layer, with each weight corresponding to a position in the feature map.

[0120] 4) Feature map weighting: The output feature map of dynamic convolution is multiplied by the spatial aggregation weight matrix to obtain the weighted feature map.

[0121] 5) Normalization: The weighted feature map is normalized along a specified dimension using the Softmax function to ensure that the weights sum to 1. This step makes the weight values ​​between 0 and 1, representing the relative importance of each region.

[0122] 6) Output: The normalized feature map is used as the final output of the dynamically deformable convolution for subsequent feature extraction.

[0123] The calculation formula is as follows:

[0124]

[0125] In the formula, x g As input features for the image, y g For each region, dynamically deformable output features, m gk This represents the spatial aggregation weights used in bias calculation, where p0 is the starting position of a predefined region in the image, and p k For the k-th position of the predefined region dynamic window, Δp gk The offset position of the dynamic window is defined. Dynamically deformable output features of all regions are dynamically determined. For different regions of the image, a dynamic window is set, and all features within the dynamic window are fully connected to obtain the total dynamically deformable output features of a single image.

[0126] Step 3.4: The dynamically deformable output features are processed through a 1×1 convolution to linearly combine the channel information of all pixels while maintaining the feature map size. Feature information at different receptive fields is obtained through 3×3 and 5×5 convolutions and then concatenated in a concat layer. Another 1×1 convolution is used to reduce the feature dimensionality, obtaining the output features of the high-resolution feature extraction module. The obtained features are then uniformly fed into a fast pooling layer to remove redundant information from the module output features at different scales. This layer has no parameters to learn, which can prevent overfitting to some extent.

[0127] Step 3.5: Transfer the features of the fast pooling layer to the multi-scale small defect feature fusion module (including convolutional layers, upsampling layers and skip connection layers) for feature fusion to enhance the feature representation capability of small defects.

[0128] Specifically, see Figure 5 The multi-scale micro-defect feature fusion module adopts a model structure of bottom-up connections, lateral connections, and skip connections to ensure effective information transmission during multi-scale feature fusion. Taking layer P2 as an example, the calculation formula is as follows:

[0129]

[0130] In the formula, ω represents different learning weights for different layers, CONV represents 3×3 convolution calculation, and Upsample represents upsampling operation. The resulting multi-scale fusion feature P2″ can be used as input to the detection head for the prediction of defect target boxes.

[0131] Step 3.6: Employ a method that uses both conventional and high-resolution detection heads separately. The detection head is a two-dimensional convolutional network. First, the defect region image is fed to the conventional detection head to determine the defect location and confidence level. Then, a downsampled image of the original image is sent to the high-resolution detection head for secondary detection.

[0132] For example, a high-resolution detection head needs to first upsample the input of a conventional detection head, and then repeat the process using a two-dimensional convolutional network to determine the defect location and confidence level. A confidence threshold is designed; if two image detection results with higher confidence are selected, they are saved, and the detection results are then fused.

[0133] In summary, by utilizing feature extraction, feature fusion, and high-resolution detection methods, the model learns the feature information of small defects at different scales, which can significantly improve the detection accuracy of small defects compared to traditional networks.

[0134] Step 4: Train the model using the training set and dynamically update the model parameters. Verify the model's generalization ability using the test set and analyze and predict the inference results.

[0135] In one optional embodiment, step 4 specifically includes the following steps:

[0136] Step 4.1: Use pre-trained convolutional weights to train the ER-ViT model and defect detection network obtained in Step 2 and Step 3 on the training set, and adjust the weight parameters to obtain the training model results, including the defect category probability distribution and regression coordinates.

[0137] Step 4.2: Evaluate the model using metrics such as prediction rate, recall rate, and inference time. Understand the overall predictive performance of the trained model, as well as its performance across different categories.

[0138] Step 4.3: Calculate the predictive value and recall for each defect category, and analyze the changes in prediction performance and confidence for each defect category during model training. Statistically analyze the model's prediction changes in small sample categories, dynamically adjust class weights, and reduce the impact of class imbalance in the model.

[0139] Step 4.4: Use the trained model to perform inference verification on the test set, count and analyze the samples of inference errors, and determine the direction in which the model needs to be improved.

[0140] Step 4.5: Optimize the model based on the results analysis. Possible optimization methods include adjusting model parameters and expanding training data. Continuous results analysis and model improvement can continuously enhance the model's performance and generalization ability.

[0141] In summary, compared with the prior art, the method of this embodiment has at least the following advantages and beneficial effects:

[0142] 1) A special attention mechanism image preprocessing method is designed to highlight the edge features of small defects in the blade, which facilitates the network to learn the defect features.

[0143] 2) Compared to conventional sliding window image cropping, using salient images can prioritize the selection of areas with the highest defect probability in high-resolution blade images, which facilitates the initial location of small defects.

[0144] 3) The ER-ViT (Regional ViT) model pre-calculates the probability of defects in the regions where the target may exist, guiding the location of defects.

[0145] 4) Compared to conventional target detection networks, the defect detection network in this embodiment adopts a lightweight network structure, with smaller model memory and shorter inference time.

[0146] 5) Compared with conventional convolutional networks for feature extraction, the defect network in this embodiment uses a method of cascading dynamic deformable convolution and conventional convolution, which can quickly extract defects of various shapes.

[0147] 6) A multi-scale micro-defect feature fusion method was designed to integrate defect edge features, morphological features and texture features to improve defect perception capabilities.

[0148] 7) A high-resolution detection head based on confidence thresholds calculates the location of defects in multi-scale images and fuses the defect detection results of conventional detection heads to improve the detection accuracy of defects.

[0149] 8) Adaptively increase the weight of small sample classes in the dataset to reduce the probability of false positives and false negatives.

[0150] Example 2

[0151] This invention also provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, a code set, or an instruction set. The at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to achieve the following: Figure 1 and / or Figure 6 This paper presents a method for detecting small defects in high-resolution wind turbine blade images.

[0152] It is understood that the memory may include random access memory (RAM) or read-only memory. Optionally, the memory may include non-transitory computer-readable storage medium. The memory can be used to store instructions, programs, code, code sets, or instruction sets. The memory may include a stored program area and a stored data area, wherein the stored program area may store instructions for implementing an operating system, instructions for at least one function, instructions for implementing the various method embodiments described above, etc.; the stored data area may store data created according to the use of the server, etc.

[0153] A processor may include one or more processing cores. The processor connects to various parts of the server via various interfaces and lines, executing instructions, programs, code sets, or instruction sets stored in memory, and accessing data stored in memory to perform various server functions and process data. Optionally, the processor may be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor may integrate one or more of the following: Central Processing Unit (CPU) and Modem. The CPU primarily handles the operating system and applications; the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor.

[0154] Since this electronic device is the electronic device corresponding to the high-resolution wind turbine blade image small defect detection method of the present invention, and the principle of the electronic device in solving the problem is similar to that of the method, the implementation of this electronic device can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0155] Example 3

[0156] This invention also provides a computer-readable storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to achieve the following: Figure 1 and / or Figure 6 This paper presents a method for detecting small defects in high-resolution wind turbine blade images.

[0157] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0158] Since the storage medium is the storage medium corresponding to the high-resolution wind turbine blade image small defect detection method of the present invention, and the principle of the storage medium in solving the problem is similar to that of the method, the implementation of the storage medium can refer to the implementation process of the above method embodiment, and the repeated parts will not be described again.

[0159] Example 4

[0160] In some possible implementations, various aspects of the methods of the embodiments of the present invention can also be implemented as a program product comprising program code that, when run on a computer device, causes the computer device to perform the steps of a high-resolution wind turbine blade image small defect detection method according to various exemplary embodiments of the present application described above. The executable computer program code or "code" used to perform the various embodiments can be written in high-level programming languages ​​such as C, C++, Python, Smalltalk, Java, JavaScript, Visual Basic, Structured Query Language (e.g., Transact-SQL), Perl, or in various other programming languages.

[0161] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0162] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0163] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A high resolution fan blade image small defect detection method, characterized in that, The method comprises the following steps: obtaining high-resolution blade images and preprocessing the images to construct an initial data set; combining domain prior knowledge to construct a defect classification network, and training the defect classification network using the initial data set; using the trained defect classification network to determine whether a defect exists in the image region and calculate the defect category probability to generate a defect saliency picture; constructing a defect detection network and training the defect detection network using the initial data set; inputting the defect saliency picture and the down-sampled image of the original picture into the defect detection network to obtain the detection box coordinates of the defect category using the detection head respectively, and then fusing the detection results according to the original picture coordinates; the working mode of the defect classification network is as follows: dividing the high-resolution blade image into multiple picture blocks, extracting features of the picture blocks to obtain local feature information, and the expression is as follows: wherein, represents local feature information, is feature category information, is block image information, is local position embedding; matching and screening the picture blocks according to the domain prior knowledge, and splicing the screened picture blocks in order; inputting the local feature information of the screened picture blocks into the channel region attention mechanism to obtain region feature information, and the expression is as follows: In the formula, This indicates normalized calculation. This indicates that multi-attention calculation is performed on the channel regions of the normalized features; This represents the one-dimensional location index of the feature; Indicates the first Local feature information of small patches in an image; taking each region feature information as input to perform channel local attention mechanism calculation to obtain local features, and the expression is as follows: wherein, denotes the spatial position index of local features within the region, denotes the local feature output; denotes the input region feature information; denotes the multi-attention computation on the normalized channel local features; splicing all the local features to obtain encoded global feature information, and then sending the global feature information to a Transformer decoder and a fully connected layer for defect classification to obtain a defect saliency picture; the defect classification network is implemented by using an ER-ViT model, and the training steps of the defect classification network comprise: down-sampling the initial data set and sending the initial data set to the ER-ViT model for training; organizing fan blade defect prior knowledge, correcting the defect categories by comparing the labeled defects, combining the prior knowledge, and re-estimating the defect region coordinates; dividing each training picture according to defects, generating a saliency picture by combining defect probabilities, and obtaining an optimal defect region map through multiple iterations.

2. The method of claim 1, wherein the method is characterized by: The method for obtaining high-resolution blade images and preprocessing the images to construct an initial data set comprises: obtaining high-resolution blade images by taking pictures of fan blades using a drone; processing the high-resolution blade images using a contrast-based spatial channel attention mechanism to generate pictures highlighting defect boundaries; labeling defects on the pictures highlighting defect boundaries, and expanding the image data in a preset manner to obtain an initial data set.

3. The method of claim 1, wherein the method is characterized by: The method for matching and screening picture blocks according to domain prior knowledge comprises: collecting a large number of images related to blade defects through prior knowledge, extracting features of the prior knowledge, and storing the extracted prior knowledge features in a feature library; calculating the similarity of the features of the picture blocks and the features in the feature library, if the similarity value is greater than a preset threshold, it is determined that the picture blocks match the domain prior knowledge, and the picture blocks are retained; otherwise, it is determined that the picture blocks do not match the domain prior knowledge, and the picture blocks are not retained.

4. The method of claim 1, wherein the method is characterized by: The defect detection network comprises a high-resolution feature extraction module, a multi-scale micro-defect feature fusion module, a conventional detection head and a high-resolution detection head; The input of the high-resolution feature extraction module is a down-sampled image of the original image and a defect saliency image; the output of the high-resolution feature extraction module is transmitted to a multi-scale micro-defect feature fusion module for feature fusion to enhance the feature expression of small defects; The defect area image is sent to a conventional detection head to determine the defect position and confidence, and the down-sampled image of the original image is sent to a high-resolution detection head for secondary detection; A confidence threshold is designed, and two image detection results with high confidence are selected for saving and detection result fusion.

5. The method of claim 4, wherein the method further comprises: The high-resolution feature extraction module includes dynamic deformable convolution and conventional convolution; the dynamic deformable convolution is composed of a dynamic convolution, a fully connected layer, and a region bias module; The dynamic convolution is used to generate a basic feature map to highlight the key features of the image; The fully connected layer maps the input features to a high-dimensional space to generate parameters for calculating the region bias; The region bias module uses the output of the fully connected layer to generate spatial aggregation weights; The calculation formula of the dynamic deformable convolution is as follows: wherein, is an image input feature, is a dynamically deformable output feature for each region; represents a spatial aggregation weight used in bias computation, is a starting position of a predefined region of an image, is a first position of a dynamic window of a predefined region, is a bias position of a dynamic window; All region dynamic deformable output features are set with a dynamic window for different region images, and all features in the dynamic window are fully connected to obtain the total dynamic deformable output features of a single image.

6. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, a code set or an instruction set, which are loaded and executed by the processor to implement the method of any one of claims 1-5.

8. A computer program product, characterised in that, The computer program product includes computer instructions for executing the method of any one of claims 1-5 when executed by a processor.

Images