Fan blade defect detection system and method based on multi-mode perception

Abstract

The invention discloses a multi-modal perception-based fan blade defect detection system and method, the system comprises a data fusion module and a depth feature extraction and defect detection module, the data fusion module is used for obtaining data of multiple modals of the surface of a to-be-detected fan blade in the same scene, and the depth feature extraction and defect detection module is used for extracting the data of multiple modals of the surface of the to-be-detected fan blade; preprocessing, feature extraction and feature fusion are carried out on the data of the multiple modes, a fused feature map is obtained, and the data of the multiple modes comprise a visible light image, an infrared image, laser radar data and thermal imaging data; and the depth feature extraction and defect detection module is used for introducing a dynamic multi-scale convolutional network and adaptive context convolution, performing multi-scale feature extraction on the fused feature map to obtain multi-scale feature information, and obtaining a fan blade defect detection result according to the multi-scale feature information. According to the invention, the limitation of a single-mode detection mode is solved, and the defect detection precision and adaptability of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of wind power generation, and in particular relates to a wind turbine blade defect detection system and method based on multi-modal sensing. Background Art

[0002] With the continuous growth of the number and capacity of wind power installed in my country, wind turbines are developing towards large-scale, efficient and intelligent development. With the continuous increase in the scale and size of wind turbine blades, the proportion of their cost in the total cost of wind turbines has gradually increased, making the safety inspection of wind turbine blades particularly critical. However, due to the complexity and harshness of the working environment of wind turbines, the probability of deformation, cracks, bulging, oil stains, sand holes and wind erosion of wind turbine blades increases with the extension of operation time. If the wind turbine blades crack, deform or deform during operation, if they are not immediately shut down for maintenance, but continue to operate, mechanical failures may occur, such as blade breakage or falling. Such serious accidents will cause huge property losses to users. Therefore, timely detection and discovery of blade damage is particularly urgent, which has always been one of the main challenges facing the industry.

[0003] Traditional wind turbine defect detection methods mainly rely on visible light images for defect identification. This single-modality detection method has some disadvantages: visible light images are easily affected by lighting and environmental conditions, especially in bad weather or low light conditions, defect identification may be limited; visible light images are difficult to penetrate the surface of objects, and have weak recognition capabilities for defects inside or partially hidden in the blades; visible light images may be difficult to accurately distinguish materials of similar colors or textures, leading to misjudgments; visible light images are less effective at night or in low-light environments and cannot meet all-weather monitoring needs.

[0004] In summary, relying solely on visible light images for fan defect detection has problems such as being sensitive to light and environment, insensitive to hidden defects, and difficulty in distinguishing similar materials. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a fan blade defect detection system and method based on multimodal perception. The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0006] The present invention provides a fan blade defect detection system based on multimodal perception, including a data fusion module and a deep feature extraction and defect detection module, wherein:

[0007] The data fusion module is used to obtain data of multiple modes in the same scene on the surface of the wind turbine blade to be detected, and preprocess, extract features and fuse the data of the multiple modes to obtain a fused feature map, wherein the data of the multiple modes include visible light images, infrared images, lidar data and thermal imaging data;

[0008] The deep feature extraction and defect detection module is used to introduce a dynamic multi-scale convolutional network and an adaptive context convolution, perform multi-scale feature extraction on the fused feature map, obtain multi-scale feature information, and obtain a fan blade defect detection result based on the multi-scale feature information.

[0009] In one embodiment of the present invention, the wind blade defect detection system based on multimodal perception also includes a data acquisition module, which is an unmanned aerial vehicle carrying a high-definition camera, an infrared camera, a lidar and a thermal imaging device, and is used to comprehensively scan the wind blades to obtain visible light images, infrared images, lidar data and thermal imaging data of the surface of the wind blades; the unmanned aerial vehicle also includes a flight control data acquisition module, and the flight control data acquisition module is used to perform data alignment, time and space synchronization and image matching on the collected visible light images, infrared images, lidar data and thermal imaging data.

[0010] In one embodiment of the present invention, the data fusion module includes a preprocessing unit, a feature extraction unit and a feature fusion unit, wherein:

[0011] The preprocessing unit is used to perform uniform size, histogram equalization and normalization processing on the visible light image, the infrared image and the thermal imaging data, so as to adjust them to a uniform size and scale the image pixel values ​​to [0, 1];

[0012] The preprocessing unit is further used to convert the laser radar data in the form of point cloud into a two-dimensional image, and perform uniform size, histogram equalization and normalization processing on the two-dimensional image to adjust it to a uniform size and scale the image pixel values ​​to [0, 1];

[0013] The feature extraction unit is used to extract features from data of different modalities using a CNN feature extraction network to obtain feature maps of different modalities;

[0014] The feature fusion unit is used to introduce an attention mechanism, perform weighted summation on feature maps of different modalities according to attention weights, and obtain a fused feature map.

[0015] In one embodiment of the present invention, data of different modalities are respectively input into independent CNN feature extraction networks, wherein the CNN feature extraction network includes thirteen convolutional layers, thirteen ReLU activation layers, four pooling layers and one fully connected layer, wherein:

[0016] A ReLU activation layer is added after each convolutional layer, the first pooling layer is added after the second ReLU activation layer, the second pooling layer is added after the fourth ReLU activation layer, the third pooling layer is added after the seventh ReLU activation layer, the fourth pooling layer is added after the tenth ReLU activation layer, and the fully connected layer is added after the last ReLU activation layer.

[0017] In one embodiment of the present invention, the feature fusion unit introduces an attention mechanism in the fusion process, dynamically adjusts the attention weights of data of different modalities in different regions, and performs weighted summation on feature data of different modalities according to the attention weights, wherein the calculation formula of the attention weights is:

[0018]

[0019] in, Indicates that the data of the mth mode is at position ( i,j ), Indicates that the data of the mth mode is at position ( i,j ) The learning parameters of For location ( i,j ) feature data of different modes, m∈{TIF,IRF,LiDAR,Thermal};

[0020] Then, the fused feature map is obtained based on the feature data of different modalities and the attention weights of different modalities:

[0021]

[0022] in, Indicates that the fused feature map is at position ( i,j ) is the pixel value at .

[0023] In one embodiment of the present invention, the deep feature extraction and defect detection module includes a deep feature extraction unit, an adaptive context convolution unit and a detection result output unit, wherein:

[0024] The deep feature extraction unit is based on a dynamic multi-scale convolutional network and is used to capture multi-scale feature information using the fused feature map;

[0025] The adaptive context convolution unit introduces an adaptive weight adjustment and deformable convolution kernel mechanism to automatically adjust the shape and position of the convolution kernel according to the irregular features of the input feature data, and through the context perception mechanism, comprehensively utilizes the global context and local context features to obtain a feature vector with more detailed information;

[0026] The detection result output unit is used to obtain defect information of the wind turbine blade according to the feature vector with more detailed information, and the defect information includes defect type, defect location, and defect probability.

[0027] In one embodiment of the present invention, the dynamic multi-scale convolutional network includes a plurality of convolutional layers, residual blocks and dynamic multi-scale convolutional sub-units connected in sequence, wherein:

[0028] The specific expression of the residual block is:

[0029] F residual =DMSC(ReLU(Conv(F input ))),

[0030] Among them, F input represents the input features of the residual block, Conv represents the convolution operation, DMSC represents the dynamic multi-scale convolution operation, and F residual Represents the output features of the residual block;

[0031] The expression of the dynamic multi-scale convolution operation performed by the dynamic multi-scale convolution subunit is:

[0032]

[0033] Among them, F′ input represents the input vector of the dynamic multi-scale convolution subunit, α k is the weight generated by the attention mechanism in the dynamic multi-scale convolution subunit, w k represents the original weight of the kth dynamic convolution kernel, Conv(F′ input ,w k ) represents the convolution operation using the kth dynamic convolution kernel, K represents the number of dynamic convolution kernels, and F DMSC A feature vector representing the output of a dynamic multi-scale convolutional subunit.

[0034] In one embodiment of the present invention, the adaptive context convolutional network unit is specifically used for:

[0035] A 5×5 average pooling layer and a 1×1 convolution layer are introduced to achieve adaptive selection of convolution kernels of different scales;

[0036] Use the trainable convolution weight increment Δw to dynamically adjust the weight of each convolution kernel during training to maximize the effect of feature extraction;

[0037] Introduce a deformable convolution kernel in the convolution layer, and adjust the sampling position of the convolution kernel through the learned offset to adapt to the non-uniform, deformed or irregular features in the input data;

[0038] By introducing a context-aware mechanism and integrating global and local context features, the model's ability to perceive scenes and objects is improved.

[0039] Another aspect of the present invention provides a method for detecting defects in wind turbine blades based on multimodal perception, which is performed using the wind turbine blade defect detection system described in any one of the above embodiments, and the method comprises:

[0040] S1: Acquire data of multiple modes of the same scene on the surface of a wind turbine blade to be inspected, wherein the data of multiple modes include visible light images, infrared images, lidar data, and thermal imaging data;

[0041] S2: preprocessing the data of the multiple modalities to improve data quality;

[0042] S3: Perform feature extraction and feature fusion on the preprocessed data of multiple modalities to obtain a fused feature map;

[0043] S4: Based on a dynamic multi-scale convolutional network, the fused feature map is used to capture multi-scale feature information;

[0044] S5: Adaptive weight adjustment and deformable convolution kernel mechanism are introduced to automatically adjust the shape and position of the convolution kernel according to the irregular features of the input feature data. In addition, the global context and local context features are comprehensively utilized through the context-aware mechanism to obtain feature vectors with more detailed information.

[0045] S6: Obtain defect information of the wind turbine blade according to the feature vector having more detailed information, wherein the defect information includes defect type, defect location, and defect probability.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. More accurate detection: The present invention provides a fan blade defect detection system and method based on multimodal perception. Through multimodal data fusion technology, combined with visible light images, infrared images, lidar data and thermal imaging data, it can comprehensively capture the detailed information of the fan blades, overcome the limitations of single modal detection methods, and improve the accuracy of defect identification.

[0048] 2. Strong adaptability: Through the dynamic multi-scale convolutional network (DMSCNet) and adaptive context convolution (ACC), the system can flexibly adjust the convolution kernel size and sampling position to adapt to features of different scales, enhancing the perception of complex backgrounds and diverse defects; global context connection enhances global information perception.

[0049] 3. Improved recognition of small target defects: The dynamic multi-scale convolutional network performs feature extraction based on the residual network, which significantly improves the detection capability of small target defects and ensures that more detailed information is retained during the reconstruction process, thereby improving detection accuracy.

[0050] 4. Reduce information loss: By introducing dynamic multi-scale convolution and adaptive weight adjustment, the fan blade defect detection system of this embodiment can adaptively select and adjust the convolution kernel, effectively reduce information loss, maintain the integrity and accuracy of features, and thus achieve more reliable defect detection in complex scenarios.

[0051] 5. All-round performance improvement: The present invention comprehensively uses technologies such as multimodal data fusion, dynamic multi-scale convolutional network and adaptive context convolutional network, so that the system can maintain a high level of detection performance in various complex environments, and significantly improves the all-round detection capability of wind turbine blade defects.

[0052] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a module diagram of a fan blade defect detection system based on multimodal perception provided by an embodiment of the present invention;

[0054] Figure 2 It is a specific structural diagram of a fan blade defect detection system based on multimodal perception provided by an embodiment of the present invention;

[0055] Figure 3 It is a flow chart of a fan blade defect detection method based on multimodal sensing provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a wind turbine blade defect detection system and method based on multimodal sensing proposed in accordance with the present invention in combination with the accompanying drawings and specific implementation methods.

[0057] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0058] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.

[0059] Embodiment 1

[0060] See also Figure 1 , Figure 1 It is a module diagram of a fan blade defect detection system based on multi-modal perception provided by an embodiment of the present invention. The fan blade defect detection system includes a data fusion module and a deep feature extraction and defect detection module, wherein the data fusion module is used to obtain data of multiple modes under the same scene on the surface of the fan blade to be detected, and pre-process, extract features and fuse features of the data of multiple modes to obtain a fused feature map, and the data of multiple modes include visible light images, infrared images, lidar data and thermal imaging data; the deep feature extraction and defect detection module is used to introduce a dynamic multi-scale convolutional network and an adaptive context convolution, perform multi-scale feature extraction on the fused feature map, obtain multi-scale feature information, and obtain the fan blade defect detection result based on the multi-scale feature information.

[0061] It should be noted that before performing defect detection on wind turbine blades, it is necessary to first collect data of multiple modes on the same scene on the surface of the wind turbine blades. In this embodiment, a drone carrying multiple sensors (high-definition cameras, infrared cameras, laser radars, thermal imaging equipment) is used to perform a comprehensive scan of the wind turbine blades to obtain visible light images, infrared images, laser radar data and thermal imaging data on the surface of the wind turbine blades. The drone also includes a flight control data acquisition module for data alignment, spatiotemporal synchronization and image matching of the collected visible light images, infrared images, laser radar data and thermal imaging data. Specific measures include: Spatiotemporal synchronization: ensure that all sensors on the drone collect data synchronously and obtain images and data of different modes at the same time point; image registration: use feature point matching algorithms (such as SIFT (scale-invariant feature transform) or SURF (accelerated robust features)) to register multi-modal data, align data of different modes to the same coordinate system, and facilitate the next step of data preprocessing and multi-modal data deep information fusion.

[0062] The data fusion module of this embodiment includes a preprocessing unit, a feature extraction unit and a feature fusion unit, wherein the preprocessing unit is used to perform uniform size, histogram equalization and normalization processing on visible light images, infrared images and thermal imaging data to adjust them to a uniform size and scale the image pixel values ​​to [0, 1]; the preprocessing unit is also used to convert the lidar data in the form of point clouds into a two-dimensional image, and then perform uniform size, histogram equalization and normalization processing on the two-dimensional image to adjust it to a uniform size and scale the image pixel values ​​to [0, 1]; the feature extraction unit is used to use a CNN feature extraction network to extract features from data of different modalities to obtain feature maps of different modalities; the feature fusion unit is used to introduce an attention mechanism, perform weighted summation on feature maps of different modalities according to attention weights, and obtain a fused feature map.

[0063] Specifically, the preprocessing unit involves preprocessing data of multiple modalities, including visible light images (TIF), infrared images (IRF), laser radar data (LiDAR), and thermal imaging data (Thermal). Each data type is uniformly resized to meet the network input requirements. Through histogram equalization and normalization operations, the data of different modalities are ensured to remain consistent during the processing process, thereby improving the system's adaptability to complex environments and effectively reducing noise and deviation in the input data.

[0064] Specifically, the preprocessing process for visible light images includes: unifying the size: adjusting the image size to the size required by the network input; histogram equalization: enhancing contrast; normalization: scaling pixel values ​​to [0,1]. The preprocessing process for infrared images includes: unifying the size: adjusting the image size to the size required by the network input; histogram equalization: improving contrast; normalization: scaling pixel values ​​to [0,1]. The preprocessing process for thermal imaging data includes: unifying the size: adjusting the image size to the size required by the network input; histogram equalization: enhancing contrast and highlighting temperature change areas; normalization: scaling pixel values ​​to [0,1].

[0065] LiDAR data usually exists in the form of point clouds, which need to be converted into a format suitable for CNN (convolutional neural network) processing. First, point cloud projection is performed. The 3D point cloud data is projected onto a 2D plane by projecting it onto the XY plane, XZ plane, or YZ plane. The second step is to generate a depth map. The point cloud is represented in the form of a depth map. The generation of a depth map mainly consists of three steps: (1) Generate a depth map based on the XYZ coordinates of the point cloud data. Each pixel represents the distance from a point to the sensor (HD camera, infrared camera, LiDAR, thermal imaging device); (2) For each point, calculate the distance from the point to the sensor based on its coordinates on the projection plane and map it to the corresponding position in the depth map; (3) Interpolate or filter the depth map to fill possible holes or noise to ensure that the generated image is suitable for CNN processing; finally, calculate the depth value of each projection plane. The depth value can be the distance from the point to the sensor or the height value. The purpose of converting point cloud data into a 2D image representation is to use CNN for feature extraction.

[0066] Furthermore, the feature extraction unit and feature fusion unit of this embodiment are based on the multimodal data fusion method of CNN, and achieve more efficient fusion by learning the feature relationship between different modalities. An independent CNN feature extraction network is designed for each modality of data to extract its deep features.

[0067] See also Figure 2 , the feature extraction unit includes an input layer, a convolution layer, a ReLU activation layer, a pooling layer, and a fully connected layer. The input layer can input a visible light image, an infrared image, a thermal imaging data, or a depth map converted from a lidar data after image preprocessing; multiple convolution layers are used to extract local features, and each convolution layer contains multiple convolution kernels for detecting different feature patterns. Convolution kernel size: 3×3, 5×5, etc.; a ReLU activation layer is added after each convolution layer; the pooling layer is used to reduce the dimension of the feature map and reduce the amount of calculation. Pooling method: Max Pooling, pooling kernel size: 2×2.

[0068] In this embodiment, data of different modalities are respectively input into independent CNN feature extraction networks. Preferably, the CNN feature extraction network includes thirteen convolutional layers, thirteen ReLU activation layers, four pooling layers and one fully connected layer, wherein a ReLU activation layer is added after each convolutional layer, the first pooling layer is added after the second ReLU activation layer, the second pooling layer is added after the fourth ReLU activation layer, the third pooling layer is added after the seventh ReLU activation layer, the fourth pooling layer is added after the tenth ReLU activation layer, and a fully connected layer is added after the last ReLU activation layer. Specifically, the fully connected layer uses a 1×1 convolution kernel to process the feature map. The 1×1 convolution kernel can change the number of channels of the feature map without changing the spatial dimension of the feature map, and integrate the local features extracted by different convolutional layers to achieve global combination of features and high-level feature extraction, thereby obtaining a feature map of the corresponding modality.

[0069] It should be noted that in actual use, the feature extraction unit needs to be trained. The specific training process is: collect and annotate a multimodal data set containing fan defects, which includes pre-processed visible light, infrared, lidar and thermal imaging data; initialize the parameters of the CNN feature extraction network, including the initial weights of each modal data; each modal data passes through an independent CNN feature extraction network, and the CNN feature extraction network structure of each modality is the same, consisting of several convolutional layers, pooling layers and fully connected layers. The feature vectors of each modality are concatenated into a long feature vector. Among them, the convolutional layer extracts local features and increases the complexity of the data. The pooling layer reduces the dimension of the feature map and reduces the amount of calculation. The fully connected layer combines the extracted local features to form global features.

[0070] Furthermore, the feature fusion unit of this embodiment is used to introduce an attention mechanism, perform weighted summation on the features of each modality according to the weight, and obtain a fused feature map. The feature fusion unit introduces an attention mechanism in the fusion process, and dynamically adjusts the weights of data of different modalities in different regions. The attention weight of each pixel is calculated, and the feature data of different modalities are weighted summed according to the attention weight, where the calculation formula of the attention weight is:

[0071]

[0072] in, Indicates that the data of the mth mode is at position ( i,j ), Indicates that the data of the mth mode is at position ( i,j ) The learning parameters of For location ( i,j), m∈{TIF,IRF,LiDAR,Thermal}.

[0073] Then, the fused feature map is obtained based on the feature data of different modalities and the attention weights of different modalities:

[0074]

[0075] in, Indicates that the fused feature map is at position ( i,j ), α, β, γ, δ are the attention weights obtained by adaptive learning, and F TIF 、F IRF 、F LiDAR 、F Thermal is the eigenvector of each mode.

[0076] The data fusion module of this embodiment obtains a more information-rich image by preprocessing visible light images, infrared images, lidar data and thermal imaging data, which helps to overcome the limitations of visible light images under specific environmental conditions. The data fusion module adopts a fusion method based on CNN to automatically learn the feature relationship between different modalities, thereby improving the fusion effect; the attention mechanism is used to dynamically adjust the weight of each modality in different areas to achieve a more refined fusion effect. The data fusion module of this embodiment significantly enhances the system's adaptability to environmental changes by fusing multiple perception data, and improves the consistency and stability of the detection results; the fused feature map has richer detail information, which helps to more accurately identify and locate small target defects.

[0077] Next, the deep feature extraction module of this embodiment introduces a dynamic multi-scale convolutional network (DMSCNet) and adaptive contextual convolution (ACC), and combines multi-scale feature extraction and attention mechanism to improve the detection accuracy of fan blade defects. The deep feature extraction module can more accurately learn and understand the characteristics of defects, overcoming the shortcomings of traditional single-stage target detection algorithms in perceiving small targets. The deep feature extraction module further applies deep learning technology to design the network architecture, completes the fused feature extraction, and optimizes the performance of the fan defect recognition system. This not only enables the system to learn and understand the characteristics of defects more accurately, but also improves the detection accuracy of fan blade defects and provides a more accurate description of the target spatial structure features.

[0078] Specifically, the deep feature extraction and defect detection module of this embodiment includes a deep feature extraction unit, an adaptive context convolution unit and a detection result output unit, wherein the deep feature extraction unit is based on a dynamic multi-scale convolutional network (Dynamic Multi-Scale Convolutional Network, DMSCNet), and is used to capture multi-scale feature information using the fused feature map; the adaptive context convolution unit introduces an adaptive weight adjustment and deformable convolution kernel mechanism, which can automatically adjust the shape and position of the convolution kernel according to the irregular characteristics of the input feature data, and through the context perception mechanism, comprehensively utilizes the global context and local context features to obtain a feature vector with more detailed information; the detection result output unit is used to obtain the defect information of the wind turbine blade based on the feature vector with more detailed information, and the defect information includes defect type, defect location, and defect probability.

[0079] Specifically, the deep feature extraction unit of this embodiment is based on a dynamic multi-scale convolutional network, which uses the fused feature map to capture multi-scale feature information. The dynamic multi-scale convolutional network is used to effectively capture multi-scale feature information. The dynamic multi-scale convolutional network is based on the residual block of the deep residual network (ResNet), and realizes effective training of the deep network through jump connections. Each residual block contains multiple convolutional layers, and retains and enhances features by adding input and output features. In order to improve the diversity and adaptability of features, this embodiment introduces a dynamic multi-scale convolution operation to replace the traditional static convolution layer.

[0080] The dynamic multi-scale convolution network of this embodiment includes multiple convolution layers, residual blocks and dynamic multi-scale convolution subunits connected in sequence, wherein the multiple convolution layers include 3×3 convolution layers and 5×5 convolution layers, and a ReLU activation layer is added after each convolution layer; in the residual block, one or more standard convolution layers are replaced by dynamic multi-scale convolution layers. The specific expression of the residual block is as follows:

[0081] F residual =DMSC(ReLU(Conv(F input ))),

[0082] Among them, F input Represents the input features of the residual block, which is the feature map input to the current residual block, and contains the feature information extracted from the previous network layer. Conv represents the convolution operation, DMSC represents the dynamic multi-scale convolution operation, and F residual Represents the output feature of the residual block, which is the output feature map after the dynamic multi-scale convolution operation. The output feature map contains the feature information processed by the dynamic convolution kernel and activation function.

[0083] Furthermore, the expression of the dynamic multi-scale convolution operation performed by the dynamic multi-scale convolution subunit is:

[0084]

[0085] Among them, F′ input represents the input vector of the dynamic multi-scale convolution subunit, α k is the weight generated by the attention mechanism in the dynamic multi-scale convolutional subunit, w k represents the original weight of the kth dynamic convolution kernel, Conv(F′ input , w k ) represents the convolution operation using the kth dynamic convolution kernel, k represents the kth dynamic convolution kernel, K represents the number of dynamic convolution kernels, F DMSC A feature vector representing the output of a dynamic multi-scale convolutional subunit.

[0086] Furthermore, the attention weights of the dynamic convolution kernel selection of the dynamic multi-scale convolution subunit are generated through the attention mechanism, and the specific formula is as follows:

[0087]

[0088] Among them, β k =f(F input ; θ) represents the attention weight, f is a small network, θ is its parameter, and k represents the kth dynamic convolution kernel. In other words, α k is calculated by calculating the attention weight β of the kth dynamic convolution kernel k The exponential value of all the first k attention weights β j The ratio of the sum of the exponential values ​​of α k Used to normalize the current attention weight β k The exponential value of α k Represents the relative weight of the kth dynamic convolution kernel among all dynamic convolution kernels.

[0089] Specifically, the small network f is actually a fully connected layer network for generating attention weights. It generates weights according to the input feature map, and then dynamically adjusts the use of convolution kernels in the convolution operation. This can more effectively capture the characteristics of the input data, improve the effect of feature extraction and the adaptability of the model. Each fully connected layer is usually connected with an activation function (such as ReLU or Tanh).

[0090] The input of the small network f is the expanded feature map, and the output is the weight parameter βk. That is, the last fully connected layer outputs βk, which represents the weight of the input feature map on different convolution kernels. k Normalized to α k, making sure that the sum of these weights is 1.

[0091] Furthermore, for wind turbine blade defect detection, although the traditional single-stage target detection algorithm has a fast detection speed, it has the limitation of a fixed field of view perception mechanism and cannot effectively perceive targets of different scales. To solve this problem, this embodiment proposes an adaptive context convolution (ACC) to improve the detection capability of defects of different scales. The adaptive context convolution network (ACC) unit of this embodiment is specifically used for:

[0092] (1) Multi-scale convolution kernel selection:

[0093] A 5×5 average pooling layer and a 1×1 convolution layer are introduced to achieve adaptive selection of convolution kernels of different scales. The specific steps are as follows: Use a 5×5 average pooling layer to reduce the dimensionality of the input feature map to reduce the spatial dimension and highlight important features. Use a 1×1 convolution layer to adjust the number of channels for subsequent processing. Introduce multiple convolution kernel size options, such as 3×3, 5×5, and 7×7. Each size of convolution kernel is convolved with the feature map after average pooling and convolution.

[0094] (2) Adaptive weight adjustment:

[0095] Use the trainable convolution weight increment Δw to perform adaptive adjustment using the following formula:

[0096] y=∑κ∈K(w k +Δw k )*x k

[0097] Among them, w k is the original weight of convolution kernel k, Δw k is the corresponding trainable increment, x k represents the convolution result of the input feature map on each dynamic convolution kernel κ, and represents the response of the dynamic convolution kernel to the input feature. y represents the output feature map, which is the weighted sum of all adaptively adjusted convolution operation results. This output feature map contains the comprehensive feature information after processing by each convolution kernel. In other words, y is the output feature map after adaptive adjustment, which integrates the responses of all convolution kernels. This method enables the network to dynamically adjust the weight of each convolution kernel during training to maximize the effect of feature extraction.

[0098] (3) Deformable convolution kernel:

[0099] Introduce a deformable convolution kernel in the convolution layer, and adjust the sampling position of the convolution kernel through the learned offset to adapt to the non-uniform, deformed or irregular features in the input data. The following formula allows the convolution kernel to be dynamically adjusted according to the characteristic morphology of the input image.

[0100]

[0101] Where n represents the input feature map x i The total number of processed, that is, the input feature map is divided into n parts, each part x i All pass through the corresponding convolution kernel w i and the trainable increment Δw i Processing. i is the weight of the convolution kernel, Δw i is the corresponding trainable increment.

[0102] (4) Context-aware mechanism:

[0103] ACC introduces a context-aware mechanism, integrates global context features and local context features, and improves the model's perception of scenes and objects, which can be expressed as:

[0104] y=Concat(GlobalContext(x),LocalContext(x))

[0105] Among them, GlobalContext(x) represents the global context feature, LocalContext(x) represents the local context feature, and Concat represents the concatenation operation.

[0106] In summary, the dynamic multi-scale convolutional network uses dynamic multi-scale convolution operations to replace the traditional static convolution layer, and dynamically adjusts the selection and weight of the convolution kernel through the attention mechanism to adapt to input data of different scales and complexities. It effectively improves the network's ability to capture multi-scale features, enhances its adaptability and generalization ability in complex scenarios, and thus improves the accuracy and stability of wind turbine defect detection.

[0107] Adaptive contextual convolution (ACC) introduces adaptive weight adjustment and deformable convolution kernel mechanism, which enables the network to automatically adjust the shape and position of the convolution kernel according to the irregular characteristics of the input data. Compared with the traditional convolution operation with a fixed receptive field, ACC improves the perception of defects of different scales and effectively improves the accuracy and coverage of small target defect detection. Through the context-aware mechanism (integration of GlobalContext and LocalContext), the global context feature and local context feature information are comprehensively utilized to improve the model's ability to understand complex scenes and targets.

[0108] Subsequently, the detection result output unit obtains the defect information of the fan blade according to the feature vector with more detailed information, and the defect information includes defect type, defect location, and defect probability. The detection result output unit of this embodiment includes a classification subunit and a regression subunit, wherein the classification subunit includes a fully connected layer and a Softmax layer connected in sequence, which are used to obtain the defect type and defect probability of the fan blade; the regression subunit includes a fully connected layer and a linear layer connected in sequence, which are used to obtain the defect location of the fan blade.

[0109] It should be noted that in actual use, the fan blade defect detection system needs to be trained. The specific training process is:

[0110] The equipment uses 4 NVIDIA 2080Ti graphics cards for training; the preprocessed data size is 608×608 pixels; the batch size is 32; a pre-trained model close to wind turbine defect detection is selected, and pre-trained models on the ImageNet and COCO datasets are used for preliminary feature extraction, extracting common features, freezing the weights of the first few layers of the pre-trained model, and only training subsequent layers to adapt to the wind turbine blade defect detection task.

[0111] In the multimodal data training stage, the input multimodal data includes pre-processed visible light images, infrared images, lidar data, and thermal imaging data. Each modal data is subjected to feature extraction through an independent convolutional neural network (CNN). A 1×1 convolution kernel is used to map and fuse the features of different modalities. An attention mechanism is introduced to calculate the weight of each pixel, dynamically adjust the weight of each modal feature in different areas, and adjust the number of channels to adapt to the subsequent network structure for feature fusion. In the defect detection training stage, a dynamic multi-scale convolutional network (DMSCNet) is used for deep feature extraction. The network structure includes multiple convolutional layers and residual blocks. The residual block contains dynamic multi-scale convolution (DMSC). Residual connections are introduced. Each residual submodule contains two convolutional layers and a jump connection to solve the gradient vanishing problem in deep networks and enhance the training effect of the network.

[0112] In the defect detection training phase, the dynamic multi-scale convolutional network (DMSCNet) is used for deep feature extraction. The network structure includes multiple convolutional layers and residual blocks. Each residual submodule contains two convolutional layers and a skip connection. The residual connection is introduced to solve the gradient vanishing problem in the deep network and enhance the network training effect.

[0113] In Adaptive Contextual Convolution (ACC), feature extraction is performed through multiple convolution kernels (such as 3x3, 5x5, 7x7), and the appropriate convolution kernel is selected through the attention mechanism. The feature map is reduced in dimension and adjusted in channels through a 5x5 average pooling layer and a 1x1 convolution layer. The weight of each convolution kernel is dynamically adjusted according to the input data. Due to the multimodal features, a deformable convolution kernel is introduced to learn the offset of the convolution kernel and adjust the sampling position to adapt to the input data. Combining global context features with local context features, the model's understanding of scenes and objects is enhanced, and deep feature extraction is completed.

[0114] In the classification and regression stage, the fully connected layer and the Softmax layer are combined for classification, and the regression module outputs the bounding box.

[0115] Hyperparameter Tuning:

[0116] Learning Rate: The initial learning rate is set to 0.001, and the cosine annealing learning rate decay strategy is adopted to gradually reduce the learning rate to improve training stability.

[0117] Optimizer: Use the Adam optimizer, combined with adaptive momentum estimation, to improve the model convergence speed and stability.

[0118] Loss Function: The classification loss uses the cross-entropy loss function (Cross-EntropyLoss), and the regression loss uses the mean squared error (MSE) loss function.

[0119] Weight Decay: Set to 0.0005 to prevent overfitting and improve model generalization ability.

[0120] IOU Threshold: In view of the characteristics of wind turbine blade defects (such as deformation, cracks, bulges, oil stains, sand holes and wind erosion), when calculating the bounding box regression loss, the IOU threshold is adjusted to 0.5 to make the model more sensitive to small defects and edge areas, ensuring that the model accurately locates the defective area.

[0121] Improved training efficiency:

[0122] Mixed Precision Training: Improves training speed and memory utilization; Data augmentation technology: Uses rotation, scaling, flipping and other techniques to enhance the diversity of training data and improve the robustness of the model.

[0123] Model performance evaluation:

[0124] Evaluation indicators: The mean average precision (mAP) is used as the main evaluation indicator to evaluate the detection performance of the model on the test set. Other indicators include classification accuracy and regression accuracy.

[0125] Performance improvement: Compared with traditional single-modal data, the classification accuracy is improved by about 24%, and the bounding box regression accuracy is improved by about 16%. On the test set, mAP is improved from 65% of the traditional model to 89% of the multimodal model. Through the feature fusion of multimodal data, the overall training loss is reduced by about 26%.

[0126] The wind turbine blade defect detection system significantly improves the detection performance of the model by combining multiple sensor data. Data alignment and preprocessing ensure the effective fusion of different modal data, and the dynamic multi-scale convolutional network (DMSCNet) and adaptive context convolution (ACC) achieve deep feature extraction and fine detection. By reasonably adjusting hyperparameters and optimization strategies, the model performs well in detecting wind turbine blade defects and has high practicality and accuracy.

[0127] The fan blade defect detection system based on multimodal perception has significant advantages in all-round performance improvement. The data fusion module obtains the deep features of each mode through data preprocessing and independent CNN feature extraction, splices them and further processes them through the feature fusion mechanism to obtain a fused feature map. The fused feature map output by the data fusion module is directly used as the input of the deep feature extraction and defect detection module to ensure seamless connection of the data flow.

[0128] The deep feature extraction and defect detection module uses the fused feature map as input to further design and train the deep learning network for defect recognition. In this process, loss calculation and back propagation are performed to avoid repeating these operations in the data fusion module, thereby improving the system's training efficiency and model optimization, so that the entire system can continuously improve recognition performance during the training process.

[0129] By introducing multimodal data and advanced detection methods, this embodiment solves the limitations of single-modal detection methods and improves the system's detection accuracy and adaptability to defects. Image fusion and dynamic multi-scale convolutional network technology make the system more flexible and stable, and the introduction of dynamic multi-scale convolution kernels and deformable convolution kernels effectively enhances the model's perception of defects of different scales. The overall innovative technical design will bring significant technological progress to the field of wind turbine blade defect detection.

[0130] Embodiment 2

[0131] Based on the first embodiment, this embodiment further provides a fan blade defect detection method based on multimodal perception, such as Figure 3 As shown, the defect detection method includes:

[0132] S1: Acquire data of multiple modes of the same scene on the surface of a wind turbine blade to be inspected, wherein the data of multiple modes include visible light images, infrared images, lidar data, and thermal imaging data;

[0133] S2: preprocessing the data of the multiple modalities to improve data quality;

[0134] S3: Extract and fuse the preprocessed multi-modal data to obtain a fused feature map, fuse the preprocessed multi-modal data to generate a feature map with richer information, and overcome the limitations of single-modal data;

[0135] S4: Based on a dynamic multi-scale convolutional network, the fused feature map is used to capture multi-scale feature information;

[0136] S5: Adaptive weight adjustment and deformable convolution kernel mechanism are introduced to automatically adjust the shape and position of the convolution kernel according to the irregular features of the input feature data. The global context and local context features are comprehensively utilized through the context-aware mechanism to obtain feature vectors with more detailed information. The extracted features are used to detect target defects on wind turbine blades, especially small target defects, through methods such as adaptive context convolution (ACC).

[0137] S6: Obtain defect information of the wind turbine blade according to the feature vector having more detailed information, wherein the defect information includes information such as defect type, defect location, defect probability and detection accuracy.

[0138] The present invention provides a fan blade defect detection system and method based on multimodal perception. Through multimodal data fusion technology, combined with visible light images, infrared images, laser radar data and thermal imaging data, the detailed information of the fan blade is fully captured, the limitations of the single modal detection method are overcome, and the accuracy of defect recognition is improved. Through the dynamic multi-scale convolutional network (DMSCNet) and adaptive context convolution (ACC), the system can flexibly adjust the convolution kernel size and sampling position, adapt to the features of different scales, and enhance the perception of complex backgrounds and diverse defects; global context connection enhances global information perception. The dynamic multi-scale convolutional network performs feature extraction on the basis of the residual network, significantly improves the detection capability of small target defects, ensures that more detailed information is retained during the reconstruction process, thereby improving the detection accuracy. By introducing dynamic multi-scale convolution and adaptive weight adjustment, the fan blade defect detection system of this embodiment can adaptively select and adjust the convolution kernel, effectively reduce information loss, maintain the integrity and accuracy of features, and thus achieve more reliable defect detection in complex scenes.

[0139] The present invention comprehensively utilizes technologies such as multimodal data fusion, dynamic multi-scale convolutional networks, and adaptive contextual convolutional networks, so that the system can maintain a high level of detection performance in various complex environments, significantly improving the all-round detection capability of wind turbine blade defects.

[0140] In the several embodiments provided by the present invention, it should be understood that the apparatus and method disclosed by the present invention can be implemented in other ways. For example, the apparatus embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0141] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0142] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A fan blade defect detection system based on multimodal perception, characterized in that: It includes data fusion module and deep feature extraction and defect detection module, among which, The data fusion module is used to obtain data of multiple modes in the same scene on the surface of the wind turbine blade to be detected, and preprocess, extract features and fuse the data of the multiple modes to obtain a fused feature map, wherein the data of the multiple modes include visible light images, infrared images, lidar data and thermal imaging data; The deep feature extraction and defect detection module is used to introduce a dynamic multi-scale convolutional network and an adaptive context convolution, perform multi-scale feature extraction on the fused feature map, obtain multi-scale feature information, and obtain a fan blade defect detection result based on the multi-scale feature information.

2. The fan blade defect detection system based on multimodal perception according to claim 1 is characterized in that: It also includes a data acquisition module, which is an unmanned aerial vehicle carrying a high-definition camera, an infrared camera, a lidar and a thermal imaging device, and is used to comprehensively scan the wind turbine blades to obtain visible light images, infrared images, lidar data and thermal imaging data on the surface of the wind turbine blades; the unmanned aerial vehicle also includes a flight control data acquisition module, which is used to perform data alignment, time and space synchronization and image matching on the collected visible light images, infrared images, lidar data and thermal imaging data.

3. The fan blade defect detection system based on multimodal perception according to claim 1 is characterized in that: The data fusion module includes a preprocessing unit, a feature extraction unit and a feature fusion unit, wherein: The preprocessing unit is used to perform uniform size, histogram equalization and normalization processing on the visible light image, the infrared image and the thermal imaging data, so as to adjust them to a uniform size and scale the image pixel values ​​to [0, 1]; The preprocessing unit is further used to convert the laser radar data in the form of point cloud into a two-dimensional image, and perform uniform size, histogram equalization and normalization processing on the two-dimensional image to adjust it to a uniform size and scale the image pixel values ​​to [0, 1]; The feature extraction unit is used to extract features from data of different modalities using a CNN feature extraction network to obtain feature maps of different modalities; The feature fusion unit is used to introduce an attention mechanism, perform weighted summation on feature maps of different modalities according to attention weights, and obtain a fused feature map.

4. The fan blade defect detection system based on multimodal perception according to claim 3 is characterized in that: The data of different modalities are respectively input into independent CNN feature extraction networks, which include thirteen convolutional layers, thirteen ReLU activation layers, four pooling layers and one fully connected layer, where: A ReLU activation layer is added after each convolutional layer, the first pooling layer is added after the second ReLU activation layer, the second pooling layer is added after the fourth ReLU activation layer, the third pooling layer is added after the seventh ReLU activation layer, the fourth pooling layer is added after the tenth ReLU activation layer, and the fully connected layer is added after the last ReLU activation layer.

5. The fan blade defect detection system based on multimodal perception according to claim 3 is characterized in that: The feature fusion unit introduces an attention mechanism in the fusion process, dynamically adjusts the attention weights of data of different modalities in different regions, and performs weighted summation on feature data of different modalities according to the attention weights, wherein the calculation formula of the attention weights is: in, Indicates that the data of the mth mode is at position (i, j ), Indicates that the data of the mth mode is at position (i, j ) The learning parameters of is the position (i, j ) feature data of different modes at each location, m∈{TIF, IRF, LiDAR, Thermal}; Then, the fused feature map is obtained based on the feature data of different modalities and the attention weights of different modalities: in, Represents the pixel value at position (i, j) of the fused feature map.

6. The fan blade defect detection system based on multimodal perception according to claim 1 is characterized in that: The deep feature extraction and defect detection module includes a deep feature extraction unit, an adaptive context convolution unit and a detection result output unit, wherein: The deep feature extraction unit is based on a dynamic multi-scale convolutional network and is used to capture multi-scale feature information using the fused feature map; The adaptive context convolution unit introduces an adaptive weight adjustment and deformable convolution kernel mechanism, which is used to automatically adjust the shape and position of the convolution kernel according to the irregular features of the input feature data, and through the context perception mechanism, comprehensively utilizes the global context and local context features to obtain a feature vector with more detailed information; The detection result output unit is used to obtain defect information of the wind turbine blade according to the feature vector with more detailed information, and the defect information includes defect type, defect location, and defect probability.

7. The fan blade defect detection system based on multimodal perception according to claim 6 is characterized in that: The dynamic multi-scale convolutional network includes a plurality of convolutional layers, residual blocks and dynamic multi-scale convolutional subunits connected in sequence, wherein: The specific expression of the residual block is: F residual =DMSC(ReLU(Conv(F input ))), Among them, F input represents the input features of the residual block, Conv represents the convolution operation, DMSC represents the dynamic multi-scale convolution operation, and F residual Represents the output features of the residual block; The expression of the dynamic multi-scale convolution operation performed by the dynamic multi-scale convolution subunit is: Among them, F′ input represents the input vector of the dynamic multi-scale convolution subunit, α k is the weight generated by the attention mechanism in the dynamic multi-scale convolution subunit, w k represents the original weight of the kth dynamic convolution kernel, Conv(F′ input , w k ) represents the convolution operation using the kth dynamic convolution kernel, K represents the number of dynamic convolution kernels, and F DMSC A feature vector representing the output of a dynamic multi-scale convolutional subunit.

8. The fan blade defect detection system based on multimodal perception according to claim 6 is characterized in that: The adaptive context convolutional network unit is specifically used for: A 5×5 average pooling layer and a 1×1 convolution layer are introduced to achieve adaptive selection of convolution kernels of different scales; Use the trainable convolution weight increment Δw to dynamically adjust the weight of each convolution kernel during training to maximize the effect of feature extraction; Introduce a deformable convolution kernel in the convolution layer, and adjust the sampling position of the convolution kernel through the learned offset to adapt to the non-uniform, deformed or irregular features in the input data; By introducing a context-aware mechanism and integrating global and local context features, the model's ability to perceive scenes and objects is improved.

9. A fan blade defect detection method based on multimodal perception, characterized in that: The method is performed using a wind turbine blade defect detection system according to any one of claims 1 to 8, the method comprising: S1: Acquire data of multiple modes of the same scene on the surface of a wind turbine blade to be inspected, wherein the data of multiple modes include visible light images, infrared images, lidar data, and thermal imaging data; S2: preprocessing the data of the multiple modalities to improve data quality; S3: Perform feature extraction and feature fusion on the preprocessed data of multiple modalities to obtain a fused feature map; S4: Based on a dynamic multi-scale convolutional network, the fused feature map is used to capture multi-scale feature information; S5: Adaptive weight adjustment and deformable convolution kernel mechanism are introduced to automatically adjust the shape and position of the convolution kernel according to the irregular features of the input feature data. In addition, the global context and local context features are comprehensively utilized through the context-aware mechanism to obtain feature vectors with more detailed information. S6: Obtain defect information of the wind turbine blade according to the feature vector having more detailed information, wherein the defect information includes defect type, defect location, and defect probability.

Images