Blade defect identification model training method and blade defect identification method

Abstract

The invention discloses a blade defect identification model training method and a blade defect identification method, which are applied to the field of blade defect detection, and are characterized in that video images and infrared images collected by a wind turbine generator blade patrolled by an unmanned aerial vehicle are fused to obtain a fused image; performing feature enhancement processing on the fused image to obtain a feature enhanced image, inputting the feature enhanced image into a blade defect recognition model for feature extraction to obtain a feature map, and performing image segmentation on the feature map to output a defect probability map to determine a model loss value under physical constraints; and updating model parameters of the blade defect identification model based on the model loss value until the trained blade defect identification model is obtained. Video images and infrared images are collected through inspection of the unmanned aerial vehicle, fusion and feature extraction are carried out on the images to train the blade defect recognition model to carry out defect recognition, various defects on the blade surface of the wind turbine generator can be rapidly and accurately detected, the detection precision is effectively improved, and the error recognition rate is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of blade defect detection, and in particular to a blade defect identification model training method, a blade defect identification method, a blade defect identification model training device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] The surface and internal defects of a fan blade will continuously deteriorate under the combined action of factors such as wind force, vibration, temperature change, etc. The structural strength of the blade will gradually decrease until it cannot bear the load during normal operation, eventually causing the blade to break, which can cause huge economic losses and even serious safety accidents. The existing wind turbine blade defect detection technology has many shortcomings. First, manual inspection is inefficient, and traditional methods such as visual inspection and knocking detection can only detect surface defects. Although ultrasonic detection and radiographic detection can detect internal defects, they are expensive, complex to operate, and inefficient, and radiographic detection has radiation hazards. Overall, the existing technology has shortcomings in terms of precision, efficiency, cost, and applicability, and cannot meet the needs of efficient and accurate detection of wind turbine blades. SUMMARY

[0003] The purpose of the present application is to provide a blade defect identification model training method and a blade defect identification method, which are applied to the field of blade defect detection. By collecting video images and infrared images through unmanned aerial vehicle inspection, the images are fused and features are extracted to train a blade defect identification model for defect identification, which can quickly and accurately detect various defects on the surface of wind turbine blades, effectively improve detection accuracy and reduce misidentification rate.

[0004] To solve the above technical problems, the present application provides a blade defect identification model training method, comprising:

[0005] Obtaining video images and infrared images collected during the process of unmanned aerial vehicle inspection of wind turbine blades, fusing the video images and the infrared images to obtain a fused image;

[0006] Performing feature enhancement processing on the fused image to obtain a feature-enhanced image, inputting the feature-enhanced image into a blade defect identification model to perform feature extraction and obtain a feature map;

[0007] Performing image segmentation on the feature map to output a defect probability map, and determining a model loss value under physical constraints based on the defect probability map;

[0008] Updating the model parameters of the blade defect identification model based on the model loss value until a trained blade defect identification model is obtained.

[0009] Optionally, determining a model loss value under physical constraints based on the defect probability map comprises:

[0010] constructing a smooth constraint loss function, a texture constraint loss function and a physical constraint loss function based on a physical constraint segmentation method;

[0011] determining a loss function weighting coefficient, and constructing a composite loss function containing the smooth constraint loss function, the texture constraint loss function and the physical constraint loss function based on the loss function weighting coefficient;

[0012] inputting the defect probability map into the composite loss function to obtain a composite model loss value.

[0013] Optionally, an expression of the smooth constraint loss function is:

[0014]

[0015] an expression of the texture constraint loss function is:

[0016]

[0017] an expression of the physical constraint loss function is:

[0018]

[0019] In the formula, L smooth is a smooth constraint loss, L texture is a texture constraint loss, L physcis is a physical constraint loss, P is the defect probability map, P gt is a true value map with a texture label, (x, y) is a pixel point coordinate, Ω is an image to be detected region, G k is the kth texture filter, K is the total number of the texture filters, σ VM (x, y) is a von Mises stress, σ yield is a material yield strength.

[0020] Optionally, performing feature enhancement processing on the fusion image to obtain a feature enhanced image, including:

[0021] performing contrast enhancement processing on the fusion image based on a dynamic contrast enhancement technology to obtain a contrast enhanced image;

[0022] performing gradient enhancement on a defect region in the contrast enhanced image based on a Soble operator to obtain a defect enhanced image.

[0023] Optionally, performing feature extraction on the feature enhanced image to obtain a feature map, including:

[0024] extracting a multi-scale feature map of the feature enhanced image through a multi-scale feature pyramid.​​​

[0025] The multi-scale feature map is weighted based on a channel attention mechanism to obtain a weighted feature map.

[0026] The weighted feature map is up-sampled and feature fused by a decoder to obtain a fused feature map.

[0027] Optionally, the video image and the infrared imaging image are image fused to obtain a fused image, comprising:

[0028] The video image and the infrared imaging image are input into an image fusion function to obtain the output fused image;

[0029] The expression of the image fusion function is:

[0030] ;

[0031] In the formula, I fused (x,y) is the fused image, I RGB (x,y) is the video image, I NIR (x,y) is the infrared image, (x,y) is the pixel point coordinate, alpha is the visible light weight coefficient, beta is the infrared enhancement factor, mu NIR is the infrared image global mean, and sigma NIR is the infrared image standard deviation.

[0032] To solve the above technical problems, the present application provides a blade defect recognition method, comprising:

[0033] Obtaining a to-be-detected video image and a to-be-detected infrared image collected by unmanned aerial vehicle inspection, and image fusing the to-be-detected video image and the to-be-detected infrared image to obtain a to-be-detected fused image;

[0034] Feature enhancement processing the to-be-detected fused image to obtain a feature enhanced image, and inputting the feature enhanced image into a blade defect recognition model to obtain a blade defect recognition result output by the model;

[0035] The blade defect recognition model is a model trained according to the blade defect recognition model training method.

[0036] To solve the above technical problems, the present application provides a blade defect recognition device, comprising:

[0037] A first module is configured to obtain a video image and an infrared image collected during unmanned aerial vehicle inspection of a wind turbine blade, and image fuse the video image and the infrared image to obtain a fused image;

[0038] The second module is configured to perform feature enhancement processing on the fusion image to obtain a feature-enhanced image, input the feature-enhanced image into a blade defect recognition model to perform feature extraction to obtain a feature map;

[0039] The third module is configured to perform image segmentation on the feature map to output a defect probability map, and determine a model loss value under physical constraints based on the defect probability map.

[0040] The fourth module is configured to update model parameters of the blade defect recognition model based on the model loss value until the blade defect recognition model is trained.

[0041] To solve the above technical problems, the present application provides an electronic device, comprising:

[0042] A memory is configured to store a computer program.

[0043] A processor is configured to implement the blade defect recognition model training method or the blade defect recognition method when executing the computer program.

[0044] To solve the above technical problems, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores computer executable instructions, and the computer executable instructions are executed by a processor to implement the blade defect recognition model training method or the blade defect recognition method.

[0045] It can be seen that, by acquiring video images and infrared images collected in the process of unmanned aerial vehicle inspection of wind turbine blades, the video images and the infrared images are fused to obtain a fusion image; the fusion image is subjected to feature enhancement processing to obtain a feature-enhanced image, and the feature-enhanced image is input into a blade defect recognition model to perform feature extraction to obtain a feature map; the feature map is subjected to image segmentation to output a defect probability map, and a model loss value under physical constraints is determined based on the defect probability map; model parameters of the blade defect recognition model are updated based on the model loss value until the blade defect recognition model is trained; and wind turbine blade defect detection is performed based on the trained blade defect recognition model. The present application can quickly and accurately detect various defects on the surface of wind turbine blades by fusing and extracting features of the video images and the infrared images collected by unmanned aerial vehicle inspection to train a blade defect recognition model for defect recognition, thereby effectively improving detection accuracy and reducing misrecognition rate. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description are only a part of the embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of the provided drawings.

[0047] Figure 1 A flow chart of a blade defect recognition model training method provided by an embodiment of the present application;

[0048] Figure 2 A structural block diagram of a blade defect recognition model training device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort fall within the protection scope of the present application.

[0050] The existing wind turbine blade defect detection technology has many shortcomings. First, manual inspection is low in efficiency, high in cost, high in risk, and easy to be affected by individual differences. Second, traditional methods such as visual inspection and knocking detection can only detect surface defects and are difficult to find internal problems. Although ultrasonic detection and ray detection can detect internal defects, they are expensive, complex to operate, low in efficiency, and ray detection has radiation hazards. Optical fiber sensor technology has good real-time monitoring effect, but is complex to install, high in cost, and easy to be disturbed by environmental factors. Infrared thermal imaging can detect internal defects, but has limited detection capability for deep defects and is easy to be disturbed by the environment. Vibration monitoring technology is difficult to directly locate damage and is greatly affected by environmental factors, and is mainly used for small wind turbines. The technology based on acoustic emission is sensitive to the environment and complex in data processing. Overall, the existing technology has deficiencies in precision, efficiency, cost and applicability, and is difficult to meet the needs of efficient and accurate detection of wind turbine blades.

[0051] The present application can quickly and accurately detect various defects on the surface of wind turbine blades by combining unmanned aerial vehicle shooting and advanced image processing technology, effectively improving detection accuracy and reducing misidentification rate. Compared with traditional methods, the present application significantly reduces labor costs and equipment investment, reduces detection costs, avoids the risk of manual high-altitude operation, and improves the safety and reliability of wind farm operation. In addition, the technology can still work stably in complex and harsh environments, discover potential problems in time and respond quickly, thereby improving the overall operation efficiency and economic benefits of the wind farm, and providing strong support for the sustainable development of the wind power industry.

[0052] The following will be described in combination with Figure 1 , Figure 1 The flowchart of the blade defect identification model training method provided by the embodiment of the present application can include:

[0053] S101: Obtain the video image and infrared image collected during the unmanned aerial vehicle inspection of the wind turbine blade, and perform image fusion on the video image and infrared image to obtain a fused image.

[0054] The embodiment can first obtain the video image and infrared image collected during the unmanned aerial vehicle inspection of the wind turbine blade, and perform image fusion on the video image and infrared image to obtain a fused image. The embodiment does not limit the specific format of the collected video image, which can be set based on actual application.

[0055] The embodiment does not limit the frequency of unmanned aerial vehicle inspection, which can be set based on actual application. After the inspection is completed, the unmanned aerial vehicle can upload the video image and infrared image to a server or cloud through a data transmission device for image processing and model training.

[0056] The embodiment does not limit the specific way of fusing the video image and infrared image, which can generally be fused by using multispectral fusion technology. Specifically, the video image and infrared image can be input into an image fusion function to obtain an output fused image; the expression of the image fusion function can be:

[0057] ;

[0058] In the formula, I fused (x,y) is the fused image, I RGB (x,y) is the video image, I NIR (x,y) is the infrared image, (x,y) is the pixel point coordinate, α is the visible light weight coefficient, β is the infrared enhancement factor, μ NIR is the global mean of the infrared image, and σ NIRis the standard deviation of the infrared image. The value range of a can be [0.6, 0.8], and the value range of β can be 1.2, which can be changed based on actual application.

[0059] For example, actual data can be taken as an example to illustrate, assuming a = 0.7, β = 1.2, μ = 100, σ = 20, I (x, y) = 150, I (x, y) = 120, the pixel value of the fusion image at the point (x, y) can be calculated by the following formula: NIR NIR RGB NIR

[0060] fused

[0061] S102: Perform feature enhancement processing on the fusion image to obtain a feature enhanced image, and input the feature enhanced image into a leaf defect recognition model to perform feature extraction to obtain a feature map.

[0062] The embodiment can perform feature enhancement processing on the fusion image to obtain a feature enhanced image. The embodiment does not limit the specific way of performing feature enhancement on the image. Generally, the dynamic contrast enhancement technology can be used to perform contrast enhancement processing on the fusion image to obtain a contrast enhanced image, and the Soble operator can be used to perform gradient enhancement on the defect region in the contrast enhanced image to obtain a defect enhanced image.

[0063] Specifically, the embodiment can perform processing on the fusion image by using the dynamic contrast enhancement (CLAHE) technology, suppress uneven illumination, and highlight the defect region. The calculation method can be as follows:

[0064] ;

[0065] ;

[0066] ;

[0067] In the formula, P(r d ) is the probability of the gray level r d in the image, n d is the number of pixels in the gray level r d , and N is the total number of pixels in the image; S(r d ) is the cumulative probability distribution of the gray level r d , that is, the cumulative sum of all gray level probabilities from gray level 0 to r d ; and T(r d ​​​​​​) is the gray level r d The new grayscale value obtained after mapping, H is the grayscale level, τ is the histogram clipping threshold, round represents the rounding operation, T(r d ) is calculated to redistribute the original grayscale according to the cumulative probability distribution to achieve contrast enhancement, while avoiding image distortion caused by over-enhancement by comparison with τ. For example, for a 3×3 grayscale image block with grayscale values ​​as shown in Table 1:

[0068] Table 1: Original grayscale values ​​of the image

[0069]

[0070] Assume that the total number of pixels in the image is N=9, the number of gray levels is H=256, and the histogram clipping threshold is τ=0.8. First, count the number of times each gray value appears n d , calculate the probability P(r d ), assuming that the statistical result is that the grayscale values ​​50, 60, 70, 80, 90, 100, 110, 120, and 130 appear once each, then the probability of each grayscale value is 1 / 9, which is approximately 0.111.

[0071] Further calculate the cumulative probability distribution S(r d ), cumulative probability distribution S(r d ) is from gray level 0 to r d The sum of the probabilities, for example: S(50)=0.111; S(60)=0.111+0.111=0.222; S(70)=0.222+0.111=0.333; S(80)=0.333+0.111=0.444; S(90)=0.444+0.111=0.555; S(100)=0.555+0.111=0.666; S(110)=0.666+0.111=0.777; S(120)=0.777+0.111=0.888; S(130)=0.888+0.111=1.000.

[0072] Finally, calculate the gray value T(r dT(50) = round(255 min(0.111, 0.8)) = round(255 0.111) = round(28.305) = 28; T(60) = round(255 min(0.222, 0.8)) = round(255 0.222) = round(56.61) = 57; T(70) = round(255 min(0.333, 0.8)) = round(255 0.333) = round(84.915) = 85; T(80) = round(255 min(0.444, 0.8)) = round(255 0.444) = round(113.22) = 113; T(90) = round(255 min(0.555, 0.8)) = round(255 0.555) = round(141.525) = 142; T(100) = round(255 min(0.666, 0.8)) = round(255 0.666) = round(169.83) = 170; T(110) = round(255 min(0.777, 0.8)) = round(255 0.777) = round(197.835) = 198; T(120) = round(255 min(0.888, 0.8)) = round(255 0.8) = round(204.0) = 204; T(130) = round(255 min(1.000, 0.8)) = round(255 0.8) = round(204.0) = 204.

[0073] That is, the gray value of the gray image after mapping can be shown in Table 2:

[0074] Table 2: Gray value of gray image after mapping

[0075]

[0076] After the dynamic contrast enhancement processing, the original gray value is mapped to a new gray range. It can be seen that the lower gray value (such as 50, 60, 70) is mapped to a lower value (28, 57, 85), thereby enhancing the contrast. The intermediate gray value (such as 80, 90, 100) is mapped to a slightly higher value (113, 142, 170), further widening the contrast. The higher gray value (such as 110, 120, 130) is mapped to a gray value close to the maximum value (198, 204, 204), while avoiding over-enhancement. This way can effectively suppress the problem of uneven illumination, highlight the defect area in the image, and improve the contrast and clarity of the image.

[0077] After the contrast enhanced image is obtained by contrast enhancement on the fusion image, a defect enhanced image can be obtained by gradient enhancement on a defect region in the contrast enhanced image based on a Soble operator (Sobel operator), and a related calculation formula of the defect region gradient enhancement is as follows:

[0078] ;

[0079] ;

[0080] ;

[0081] In the formula, G x represents a gradient component of the image in the horizontal direction, which is obtained by convolution operation of the input contrast enhanced image I contrast (x,y) and a Sobel horizontal gradient operator S x ; G y represents a gradient component of the image in the vertical direction, which is obtained by convolution operation of the input contrast enhanced image I contrast (x,y) and a Sobel vertical gradient operator S y ; Edge(x,y) is a binary function, which is used to determine whether each pixel point (x,y) in the image belongs to an edge point, and the value is 1 or 0, which is determined by the comparison result of the gradient amplitude and a set high threshold θ high .

[0082] The horizontal gradient operator in the embodiment can be:

[0083] ;

[0084] The horizontal gradient operator is mainly used for detecting the horizontal edge in the image. By performing convolution operation on the image, the region with obvious gray change in the horizontal direction, i.e., the horizontal edge feature, in the image can be highlighted. In the fan blade defect detection, it can help to identify the defect profile of the blade surface in the horizontal direction, such as the boundary of the defects such as cracks, damages, etc. in the horizontal direction.

[0085] The vertical gradient operator in the embodiment can be:

[0086] ;

[0087] The vertical gradient operator is used for detecting the vertical edge in the image. Similarly, by performing convolution operation on the image, S x can highlight the region with obvious gray change in the vertical direction, i.e., the vertical edge feature, in the image. In actual application, S y is similar to S xWhen used in combination, the edge information in the image can be fully captured, which helps to accurately locate the position and scope of defects on the blade surface.

[0088] In this embodiment, θ high The value of can be 0.3×max(√(G x ²+G y ²)). The Canny edge detection algorithm is a commonly used edge detection algorithm, in which the high threshold θ is set high Used to distinguish strong edges from weak edges, a higher threshold can ensure that only those areas with significant grayscale changes and obvious gradients are judged as edges, thereby reducing the influence of noise and false edges. high The value of can effectively retain the significant edges of blade surface defects, while suppressing noise interference in the image and improving the accuracy of defect recognition.

[0089] This embodiment can be illustrated by example, assuming that it is known that I contrast (x,y), as shown below:

[0090] ;

[0091] Among them, the grayscale values ​​of the two columns on the left are low, and the grayscale value of the column on the right is high. There is a vertical edge in the simulated image, and the horizontal gradient component and the vertical gradient component can be further calculated:

[0092] ;

[0093] ;

[0094] In summary, the gradient magnitude matrix can be calculated:

[0095] ;

[0096] Taking the maximum value of 509.9 in the gradient amplitude, the high threshold θ can be determined high The value of is:

[0097] ;

[0098] Then the value of Edge(x,y) is:

[0099] .

[0100] In this embodiment, after gradient enhancement is performed on the defect area in the contrast-enhanced image to obtain a defect-enhanced image, the defect-enhanced image may be input into a blade defect recognition model to perform feature extraction to obtain a feature map.

[0101] The embodiment does not show the specific way of feature extraction. Generally, the multi-scale feature pyramid can be used to extract the multi-scale feature map of the enhanced image; the multi-scale feature map is weighted based on the channel attention mechanism to obtain a weighted feature map; and the weighted feature map is up-sampled and feature fused by a decoder to obtain a fused feature map.

[0102] Firstly, the calculation expression of the multi-scale feature pyramid can be constructed as:

[0103] ;

[0104] In the formula, F l is the feature map of the lth layer, L is the number of feature layers, which can generally be 5, EfficientNet is a high-efficiency convolutional neural network, R is a real number space, H l is the height of the lth layer feature map, W l is the width of the lth layer feature map, D l is the number of channels of the lth layer feature map, and I defect (x, y) is the defect enhanced image, that is, the feature enhanced image.

[0105] In the deep feature extraction process, different levels of feature maps contain different scale semantic information. With the increase of network depth, the scale of the feature map gradually decreases, but the semantic information gradually increases. For example, in the fan blade defect recognition, the feature map of a shallow layer (such as l = 1, 2) can contain the texture, edge and other detailed features of the blade surface, and the feature map of a deep layer (such as l = 4, 5) can capture the overall structure of the blade, the shape of the potential defect and other global semantic information.

[0106] In the embodiment, D l can belong to the set {24, 32, 56, 112, 1792}, and the calculation method of the height of the lth layer feature map and the width of the lth layer feature map can be:

[0107] ;

[0108] ;

[0109] In the formula, H and W are the height and width of the defect enhanced image, respectively.

[0110] Further, the channel attention mechanism can be used to weight the feature map. The calculation expression is as follows:

[0111] ;

[0112] ;

[0113] In the formula, A cis the attention weight of the c-th channel, F i,j,c is the feature value at the i-th row, j-th column, c-th channel, Sigmoid is the activation function, F is the feature map before weighting, F attn is the feature map after the channel attention mechanism processing, 1 H×W is a full 1 matrix, A is a vector containing the attention weights of all channels, i.e. [A1, A2, …, A C ], which is used to weight each channel of the input feature map F, ⨀ represents element-wise weighting, which adjusts the feature value size of each channel of the original feature map with the attention coefficient, ⨂ represents dimension broadcast adaptation, which allows the channel dimension attention weight to act on the spatial dimension of the entire feature map.

[0114] The decoder gradually increases the spatial resolution of the feature map through the upsampling operation, so that it can match or approach the size of the original image. Therefore, the feature map output by the final decoder is the same size or close to the original image, so as to perform pixel-level defect recognition. In the decoder, in addition to the upsampling operation, a feature fusion operation is also needed to combine feature maps of different scales to retain more detailed information and contextual information, thereby improving the accuracy of defect recognition.

[0115] In this embodiment, the calculation formula for the upsampling operation can be as follows:

[0116] ;

[0117] In the formula, F up is the feature map after upsampling, Deconv is the transpose convolution, m = 3 represents that the transpose convolution kernel size is 3 × 3, and s = 2 represents that the step size is 2.

[0118] S103: Image segmentation is performed on the feature map to output a defect probability map, and a model loss value under physical constraints is determined based on the defect probability map.

[0119] This embodiment can perform image segmentation on the feature map to output a defect probability map, and determine a model loss value under physical constraints based on the defect probability map.

[0120] In this embodiment, the blade defect recognition model can perform image segmentation on the feature map to output a defect probability map, output a pixel-level defect probability, and realize binary classification through a logistic regression layer, i.e. defective and non-defective. Its mathematical expression can be:

[0121] ;

[0122] In the formula, p represents an output variable, takes 0 or 1, and represents a classification result; q represents an input feature vector, includes a plurality of feature values used for predicting the output, P(p=1|q) represents a probability that the output p is 1 under the condition that the feature vector of the pixel q is F(q), the learnable weight vector ω ∈ R D , the bias term b ∈ R, the feature vector F(q) of the pixel q ∈ R D , R is a real number set, and D is a feature channel number.

[0123] The physical constraint segmentation integrates physical laws into the image segmentation process. In the fan blade defect identification, a physical constraint term conforming to the material mechanics law is constructed, so that the segmentation process not only depends on the features (such as gray value, texture, etc.) of the image data itself, but also follows the failure law of the material under actual stress, which can effectively avoid the missegmentation phenomenon that may occur due to the image features alone.

[0124] Therefore, the embodiment can construct a composite loss function to optimize the segmentation result by combining data driving and physical laws. For example, a smooth constraint loss function, a texture constraint loss function and a physical constraint loss function are constructed based on the physical constraint segmentation method; a loss function weighting coefficient is determined, a composite loss function including the smooth constraint loss function, the texture constraint loss function and the physical constraint loss function is constructed based on the loss function weighting coefficient; and the defect probability map is input into the composite loss function to obtain a composite model loss value.

[0125] The expression of the composite loss function in the embodiment can be:

[0126] ;

[0127] In the formula, L total is a composite loss, L smooth is a smooth constraint loss, L texture is a texture constraint loss, L physcis is a physical constraint loss, a1 is a first weighting coefficient, a2 is a second weighting coefficient, and a3 is a third weighting coefficient. In the embodiment, a1 can be 0.5, a2 can be 0.3, and a3 can be 0.2.

[0128] The embodiment does not limit the specific forms of the smooth constraint loss function, the texture constraint loss function and the physical constraint loss function, which can be set based on actual application.

[0129] Specifically, the expression of the smooth constraint loss function can be:

[0130] ;

[0131] The expression of the texture constraint loss function can be:

[0132] ;

[0133] The expression of the physical constraint loss function can be:

[0134] ;

[0135] In the formula, L smooth is a smooth constraint loss, L texture is a texture constraint loss, L physcis is a physical constraint loss, P is a defect probability map, P gt is a ground truth map with texture labels, (x, y) is a pixel point coordinate, Ω is an image to be detected region, G k is the kth texture filter, K is the total number of texture filters, σ VM (x, y) is a von Mises stress, σ yield is the yield strength of the material.

[0136] Suppose the smooth constraint loss L smooth = 3.20, the texture constraint loss L texture = 0.18, the physical constraint loss L physcis = 0.01, and the weighting coefficients are a1 = 0.5, a2 = 0.3, and a3 = 0.2, respectively.

[0137] The composite loss function is calculated as:

[0138] L total = 0.5 x 3.20 + 0.3 x 0.18 + 0.2 x 0.01 = 1.656.

[0139] S104: Update the model parameters of the leaf defect recognition model based on the model loss value until a trained leaf defect recognition model is obtained.

[0140] The model parameters of the leaf defect recognition model can be updated based on the model loss value until a trained leaf defect recognition model is obtained.

[0141] Specifically, during the model training process, image fusion can be performed on the video image and the infrared image to obtain a fused image for label processing, and the defect area in the image is labeled for constructing a sample set.

[0142] The leaf defect recognition model is trained through the sample set, the composite loss value of the defect probability map of the sample image in the sample set is calculated, the model parameters are updated through the composite loss value, and the iteration is updated until the model converges, so as to complete the training of the model.

[0143] The application fuses the features of visible light and infrared thermal imaging through multi-modal data preprocessing technology, balances spectral information and thermal radiation information by using a multispectral fusion formula, enhances the contrast of temperature abnormal areas in the infrared image, suppresses uneven illumination by dynamic contrast enhancement, highlights the defect area, strengthens the defect edge features by defect area gradient enhancement, extracts the image edge by using a Sobel operator, retains the significant edge and suppresses the noise. A multi-scale feature pyramid is constructed by using a deep feature extraction technology, different scale semantic features are extracted, global semantic information is captured by deep features, and details and textures are retained by shallow features. A channel attention mechanism adaptively focuses on key feature channels, suppresses redundant channels and enhances effective features. The resolution of the feature map is restored by upsampling in the decoder, and spatial details are gradually restored by transposed convolution. Through physical constraint segmentation technology: a composite loss function is constructed, the segmentation result is optimized by combining data-driven and physical law, over-segmentation is suppressed by smooth constraint, defect texture is matched by texture constraint, and the segmented area is ensured to meet the mechanical failure law by physical constraint.

[0144] Based on the above embodiment, the unmanned aerial vehicle inspection collects video images and infrared images, the images are fused and feature extraction is performed to train a blade defect recognition model for defect recognition, which can quickly and accurately detect various defects on the surface of the wind turbine blade, effectively improve the detection accuracy and reduce the misidentification rate.

[0145] The following is a blade defect recognition method provided by an embodiment of the application. The method can include:

[0146] Obtaining the to-be-detected video image and the to-be-detected infrared image collected by the unmanned aerial vehicle inspection, fusing the to-be-detected video image and the to-be-detected infrared image to obtain a to-be-detected fused image;

[0147] Performing feature enhancement processing on the to-be-detected fused image to obtain a feature-enhanced image, inputting the feature-enhanced image into the blade defect recognition model, and obtaining a blade defect recognition result output by the model;

[0148] The blade defect recognition model is a model trained according to a blade defect recognition model training method.

[0149] The embodiment can obtain the to-be-detected video image and the to-be-detected infrared image collected by the unmanned aerial vehicle inspection, fuse the to-be-detected video image and the to-be-detected infrared image according to the image processing mode of the model training process to obtain a to-be-detected fused image, and perform feature enhancement processing on the to-be-detected fused image to obtain a feature-enhanced image.

[0150] The feature-enhanced image is input into the trained blade defect recognition model for defect recognition. The model can output a defect probability map of the feature-enhanced image, and output an image defect recognition result through binary classification, i.e., defective or non-defective.

[0151] The following describes the present application in detail Figure 2 , Figure 2 A structural block diagram of a blade defect recognition model training device provided by an embodiment of the present application can include:

[0152] A first module 100 is configured to acquire video images and infrared images collected during unmanned aerial vehicle inspection of wind turbine blades, and perform image fusion on the video images and the infrared images to obtain a fused image;

[0153] A second module 200 is configured to perform feature enhancement processing on the fused image to obtain a feature-enhanced image, and input the feature-enhanced image into a blade defect recognition model to perform feature extraction to obtain a feature map;

[0154] A third module 300 is configured to perform image segmentation on the feature map to output a defect probability map, and determine a model loss value under physical constraints based on the defect probability map;

[0155] A fourth module 400 is configured to update model parameters of the blade defect recognition model based on the model loss value until a trained blade defect recognition model is obtained.

[0156] Based on the above embodiments, the present application can quickly and accurately detect various defects on the surface of wind turbine blades by collecting video images and infrared images through unmanned aerial vehicle inspection, performing image fusion and feature extraction to train a blade defect recognition model for defect recognition, thereby effectively improving detection accuracy and reducing misrecognition rate.

[0157] Based on the above embodiments, the third module 300 can include:

[0158] A first unit is configured to construct a smooth constraint loss function, a texture constraint loss function, and a physical constraint loss function based on a physical constraint segmentation method;

[0159] A second unit is configured to determine a loss function weighting coefficient, and construct a composite loss function containing the smooth constraint loss function, the texture constraint loss function, and the physical constraint loss function based on the loss function weighting coefficient;

[0160] A third unit is configured to input the defect probability map into the composite loss function to obtain a composite model loss value.

[0161] Based on the above embodiments, the expression of the smooth constraint loss function is:

[0162] ;

[0163] The expression of the texture constraint loss function is:

[0164] ;

[0165] The expression of the physical constraint loss function is:

[0166] ;

[0167] In the formula, L smooth is a smooth constraint loss, L texture is a texture constraint loss, L physcis is a physical constraint loss, P is the defect probability map, P gt is a true value map with a texture label, (x, y) is a pixel point coordinate, Ω is an image to be detected region, G k is the kth texture filter, K is the total number of the texture filter, σ VM (x, y) is the von Mises stress, σ yield is the material yield strength.

[0168] Based on the above embodiments, the second module 200 can include:

[0169] The fourth unit is configured to perform contrast enhancement processing on the fused image based on a dynamic contrast enhancement technology to obtain a contrast enhanced image.

[0170] The fifth unit is configured to perform gradient enhancement on a defect region in the contrast enhanced image based on a Soble operator to obtain a defect enhanced image.

[0171] Based on the above embodiments, based on the above embodiments

[0172] The sixth unit is configured to extract a multi-scale feature map of the feature enhanced image through a multi-scale feature pyramid.

[0173] The seventh unit is configured to perform weighted processing on the multi-scale feature map based on a channel attention mechanism to obtain a weighted feature map.

[0174] The eighth unit is configured to perform up-sampling and feature fusion processing on the weighted feature map through a decoder to obtain a fused feature map.

[0175] Based on the above embodiments, the first module 100 can include:

[0176] The ninth unit is configured to input the video image and the infrared imaging image into an image fusion function to obtain the fused image output by the image fusion function.

[0177] The expression of the image fusion function is:

[0178] ;

[0179] In the formula, Ifused (x,y) is the fusion image, I RGB (x,y) is the video image, I NIR (x,y) is the infrared image, (x,y) is the pixel coordinate, α is the visible light weight coefficient, β is the infrared enhancement factor, μ NIR is the global mean of the infrared image, σ NIR is the standard deviation of the infrared image.

[0180] The following is a kind of blade defect identification device provided by the embodiment of the application, the device can include:

[0181] The fifth module is used to obtain the video image to be detected and the infrared image to be detected collected by unmanned aerial vehicle inspection, and image fusion is carried out on the video image to be detected and the infrared image to be detected to obtain the fusion image to be detected.

[0182] The sixth module is used to carry out feature enhancement processing on the fusion image to be detected to obtain a feature enhanced image, input the feature enhanced image into a blade defect identification model, and obtain the blade defect identification result output by the model.

[0183] The blade defect identification model is a model trained by the blade defect identification model training device.

[0184] Based on the above embodiment, the application further provides an electronic device, which can include a memory and a processor, wherein the memory has a computer program stored therein, and the processor can realize the steps provided by the above embodiment when calling the computer program in the memory. Of course, the device can also include various necessary network interfaces, power supplies and other components.

[0185] The application further provides a computer readable storage medium, which has a computer program stored thereon, and the computer program can realize the method provided by the embodiment of the application when executed by a terminal or a processor. The storage medium can include U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and various program code storage media.

[0186] In this document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional elements of the same name in the process, method, article, or apparatus.

Claims

1. A blade defect recognition model training method, characterized in that: include: Obtaining video images and infrared images collected during the inspection of wind turbine blades by a drone, and fusing the video images and the infrared images to obtain a fused image; Performing feature enhancement processing on the fused image to obtain a feature enhanced image, and inputting the feature enhanced image into a blade defect recognition model to perform feature extraction to obtain a feature map; Performing image segmentation on the feature map to output a defect probability map, and determining a model loss value under physical constraints based on the defect probability map; The model parameters of the blade defect recognition model are updated based on the model loss value until the trained blade defect recognition model is obtained.

2. The blade defect recognition model training method according to claim 1, characterized in that: Determining a model loss value under physical constraints based on the defect probability map includes: Based on the physical constraint segmentation method, smoothness constraint loss function, texture constraint loss function and physical constraint loss function are constructed; Determining a loss function weighting coefficient, and constructing a composite loss function including the smoothness constraint loss function, the texture constraint loss function, and the physical constraint loss function based on the loss function weighting coefficient; The defect probability map is input into the composite loss function to obtain a composite model loss value.

3. The blade defect recognition model training method according to claim 2 is characterized in that: The expression of the smooth constraint loss function is: ； The expression of the texture constraint loss function is: ； The expression of the physical constraint loss function is: ； Where, L smooth is the smooth constraint loss, L texture is the texture constraint loss, L physcis is the physical constraint loss, P is the defect probability map, P gt is the true value map with texture labels, (x, y) is the pixel coordinate, Ω is the image area to be detected, G k is the kth texture filter, K is the total number of texture filters, σ VM (x,y) is the von Mises stress, σ yield is the yield strength of the material.

4. The blade defect recognition model training method according to claim 1, characterized in that: Performing feature enhancement processing on the fused image to obtain a feature enhanced image includes: Performing contrast enhancement processing on the fused image based on a dynamic contrast enhancement technology to obtain a contrast enhanced image; The defect area in the contrast enhanced image is gradient enhanced based on the Soble operator to obtain a defect enhanced image.

5. The blade defect recognition model training method according to claim 1, characterized in that: Extracting features from the feature-enhanced image to obtain a feature map includes: Extracting a multi-scale feature map of the feature-enhanced image through a multi-scale feature pyramid; Performing weighted processing on the multi-scale feature map based on a channel attention mechanism to obtain a weighted feature map; The decoder performs upsampling and feature fusion processing on the weighted feature map to obtain a fused feature map.

6. The blade defect recognition model training method according to claim 1, characterized in that: Performing image fusion on the video image and the infrared imaging image to obtain a fused image includes: Inputting the video image and the infrared imaging image into an image fusion function to obtain the output fused image; Wherein, the expression of the image fusion function is: ； Where, I fused (x, y) is the fused image, I RGB (x, y) is the video image, I NIR (x, y) is the infrared image, (x, y) is the pixel coordinate, α is the visible light weight coefficient, β is the infrared enhancement factor, μ NIR is the global mean of infrared images, σ NIR is the standard deviation of the infrared image.

7. A blade defect identification method, characterized in that: include: Obtaining a video image to be detected and an infrared image to be detected collected by a drone inspection, and fusing the video image to be detected and the infrared image to be detected to obtain a fused image to be detected; Performing feature enhancement processing on the fused image to be detected to obtain a feature enhanced image, inputting the feature enhanced image into a blade defect recognition model to obtain a blade defect recognition result output by the model; The blade defect recognition model is a model trained according to the blade defect recognition model training method according to any one of claims 1 to 6.

8. A blade defect identification device, characterized in that: include: The first module is used to obtain video images and infrared images collected during the drone inspection of wind turbine blades, and fuse the video images and the infrared images to obtain a fused image; The second module is used to perform feature enhancement processing on the fused image to obtain a feature enhanced image, and input the feature enhanced image into the blade defect recognition model to perform feature extraction to obtain a feature map; A third module is configured to perform image segmentation on the feature map to output a defect probability map, and determine a model loss value under physical constraints based on the defect probability map; The fourth module is used to update the model parameters of the blade defect recognition model based on the model loss value until the trained blade defect recognition model is obtained.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the blade defect recognition model training method according to any one of claims 1 to 6, or the blade defect recognition method according to claim 7, when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by the processor, the blade defect recognition model training method according to any one of claims 1 to 6, or the blade defect recognition method according to claim 7 is implemented.

Images