Fan blade damage identification method and device based on audio and image
Through the comprehensive analysis of audio and image data, the octave algorithm and neural network technology are used to solve the problem of poor fan blade detection accuracy, and high accuracy and low cost fan blade damage detection is achieved.
Patent Information
- Application Number
- CN202410974059.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-11-15
AI Technical Summary
The existing fan blade detection technology has the limitations of contact and non-contact type, resulting in poor detection accuracy and the inability to achieve comprehensive fan blade damage detection.
The fan blade damage recognition method based on audio and image is adopted, and the sound pressure level characteristics of the audio signal are extracted through the octave algorithm, and the backpropagation neural network and the lightweight convolutional neural network are used for comprehensive judgment to improve the accuracy of the identification of damage state sex.
It improves the accuracy and efficiency of fan blade damage detection, reduces the detection cost and complexity, and is suitable for most practical application needs.
Smart Images

Figure CN118965154A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of wind turbine blade detection, and more specifically, to a method and device for identifying wind turbine blade damage based on audio and images. Background Art
[0002] In recent years, nondestructive testing technology for wind turbine blades has been widely developed and applied to the structural health monitoring of blades. Although various nondestructive testing technologies can effectively detect faults of wind turbine blades, they all have their own limitations, and a single testing method cannot detect all types of blade damage.
[0003] Currently commonly used detection methods usually include contact and non-contact non-destructive testing.
[0004] Contact-based nondestructive testing technologies include strain testing, acoustic emission testing, and ultrasonic testing. These testing technologies can detect various types of damage with high precision. However, the sensors need to be deployed on the surface or inside the blade, which increases the difficulty of installation and subsequent maintenance costs.
[0005] Strain sensors based on fiber Bragg gratings (FBGs) can be embedded in the blade structure, but they place higher demands on the manufacturing process of the blades. At the same time, as the size of the blades increases, the system needs to combine multiple sensors into a sensor array for all-round monitoring, which increases the complexity and cost of the system. In addition, contact-type non-destructive testing technology requires the wind turbine to be shut down during deployment, causing certain economic losses and further increasing the difficulty of deployment.
[0006] Non-contact non-destructive testing technologies include acoustic testing, machine vision testing, thermal imaging testing, etc. These testing technologies can be deployed when the wind turbine is running and are more suitable for monitoring the structural health of wind turbine blades. However, acoustic testing technology and machine vision testing also have the problem of difficulty in locating wind turbine blade damage and poor timeliness, resulting in poor detection accuracy.
[0007] Therefore, how to achieve all-round detection of wind turbine blades and improve detection accuracy is a technical problem that urgently needs to be solved. Summary of the invention
[0008] In view of the defects of the prior art, the purpose of this application is to provide a method and device for identifying wind blade damage based on audio and images, aiming to solve the problem of poor detection accuracy of wind blade damage due to limitations of contact and non-contact detection technologies in the prior art.
[0009] To achieve the above objectives, the present application provides a method for wind turbine blade damage identification based on audio and image, comprising:
[0010] Acquire audio signals and image data of wind turbine blades;
[0011] Dividing the frequency spectrum of the audio signal into a plurality of frequency bands by using an octave algorithm, and extracting sound pressure level features of the plurality of frequency bands;
[0012] Inputting the sound pressure level feature into a trained back propagation neural network model BPNN to obtain an audio recognition result of the damage condition of the fan blade;
[0013] Inputting the image data into a trained lightweight convolutional neural network model to obtain an image recognition result of the damage condition of the wind turbine blade;
[0014] Comprehensively judging the audio recognition result and the image recognition result to obtain a comprehensive recognition result of the damage condition of the wind turbine blade;
[0015] Among them, the back propagation neural network model is trained based on audio samples and audio labels, and the lightweight convolutional neural network model is established based on a lightweight convolutional structure and trained based on image samples and image labels.
[0016] The present application analyzes the audio signals of the fan blades through an octave algorithm to extract the sound pressure level characteristics of different frequency bands. The frequency band analysis helps to capture possible fault signals in different frequency ranges, and uses a lightweight convolutional neural network to process the image data of the fan blades to detect visual damage characteristics. The combined analysis results of the audio and image data can improve the accuracy of identifying the overall damage status of the fan blades.
[0017] Optionally, the dividing the frequency spectrum of the audio signal into a plurality of frequency ranges by using an octave algorithm, and extracting sound pressure level features of the plurality of frequency ranges, comprises:
[0018] The frequency spectrum of the audio signal is divided into a plurality of frequency ranges by using a reference frequency; the upper cut-off frequency, the lower cut-off frequency and the center frequency of the plurality of frequency ranges are in geometric proportion; the upper cut-off frequency is determined according to the reference frequency and a preset parameter, and the lower cut-off frequency and the center frequency are determined according to the upper cut-off frequency and a preset octave number;
[0019] Obtain a frequency range sound pressure magnitude based on the lower cut-off frequency, the center frequency, and the frequency component amplitude, and determine the sound pressure level characteristic based on the frequency range sound pressure magnitude and the reference sound pressure;
[0020] After extracting the sound pressure level features of the multiple frequency ranges, the method further includes:
[0021] The sound pressure level characteristics are normalized and dimensionless through minimum and maximum value normalization.
[0022] Optionally, the back propagation neural network model includes a multi-layer perceptron MLP and a linear layer;
[0023] The multilayer perceptron is used to divide the characteristic matrix of the sound pressure level characteristics into a plurality of octave characteristic vectors and obtain a plurality of preliminary recognition results;
[0024] The linear layer is used to perform vector concatenation on multiple result vectors of the preliminary recognition results to obtain the final audio recognition result.
[0025] Optionally, the training method of the back propagation neural network model includes:
[0026] Constructing an initial model of the back propagation neural network model, and randomly initializing the weight matrix and bias vector of the multilayer perceptron;
[0027] Inputting the sound pressure level characteristics corresponding to the audio sample into the initial model, and outputting the health status information of the fan blades;
[0028] The error between the health status information and the audio label is calculated, and the weight matrix and the bias vector are calculated and iteratively updated according to the error until a stopping condition is reached to obtain a trained back propagation neural network model.
[0029] Optionally, it also includes:
[0030] Performing performance evaluation on the trained back propagation neural network model to obtain a confusion matrix of the model, wherein the confusion matrix includes true positive examples, false negative examples, false positive examples, and true negative examples;
[0031] The accuracy, precision, recall and F1 value of the back propagation neural network model are determined based on the true positive examples, false negative examples, false positive examples and true negative examples.
[0032] Optionally, the lightweight convolutional neural network model includes a convolution layer, a lightweight convolutional structure layer, a global average pooling layer and a fully connected layer;
[0033] The convolution layer is used to extract features from the image data to obtain a feature image;
[0034] The lightweight convolution structure layer is used to process the visual information of the feature image at different scales to obtain multi-scale feature information;
[0035] The global average pooling layer is used to perform global average pooling on the multi-scale feature information, averaging the two-dimensional graphic information of each channel to obtain a feature value;
[0036] The fully connected layer is used to judge the eigenvector corresponding to the eigenvalue and predict the probability of each image recognition result.
[0037] Optionally, the convolution layer is a deformable convolution layer.
[0038] Optionally, the lightweight convolution structure layer includes branch A, branch B, branch C, branch D, a splicing sublayer and a residual structure sublayer;
[0039] The branch A is used to perform a 1×1 convolution operation on the feature image;
[0040] The branch B is used to perform 1×1 point-by-point convolution and 3×3 depth convolution on the feature image;
[0041] The branch C is used to perform 1×1 point-by-point convolution and two 3×3 depth convolutions on the feature image;
[0042] The branch D is used to perform a 3×3 maximum pooling operation on the feature image;
[0043] The stitching sublayer is used to stitch the output images of the branch A, branch B, branch C and branch D to obtain a stitched image;
[0044] The residual structure sublayer is used to add the feature image and the spliced image to obtain an output feature map.
[0045] In a second aspect, the present application further provides a wind turbine blade damage identification device based on audio and image, comprising:
[0046] A data acquisition module, used to acquire audio signals and image data of wind turbine blades;
[0047] An audio extraction module, used to divide the frequency spectrum of the audio signal into multiple frequency ranges using an octave algorithm, and extract sound pressure level features of the multiple frequency ranges;
[0048] An audio recognition module, used for inputting the sound pressure level feature into a trained back propagation neural network model BPNN to obtain an audio recognition result of the damage condition of the fan blade;
[0049] An image recognition module, used to input the image data into a trained lightweight convolutional neural network model to obtain an image recognition result of the damage condition of the wind turbine blade;
[0050] A comprehensive judgment module, used for comprehensively judging the audio recognition result and the image recognition result to obtain a comprehensive recognition result of the damage condition of the wind turbine blade;
[0051] Among them, the back propagation neural network model is trained based on audio samples and audio labels, and the lightweight convolutional neural network model is established based on a lightweight convolutional structure and trained based on image samples and image labels.
[0052] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.
[0053] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.
[0054] In a fifth aspect, the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.
[0055] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.
[0056] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:
[0057] (1) The present application analyzes the audio signal of the fan blade through an octave algorithm to extract the sound pressure level characteristics of different frequency bands. The frequency band analysis helps to capture possible fault signals in different frequency ranges, and uses a lightweight convolutional neural network to process the image data of the fan blade to detect visual damage characteristics. The combined analysis results of the audio and image data can improve the accuracy of identifying the overall damage status of the fan blade.
[0058] (2) This application uses a multi-scale lightweight convolutional neural network to extract damage information at multiple scales. The constructed lightweight convolutional neural network model has the advantages of small number of parameters, small amount of calculation and high recognition accuracy, and can well identify the health status of wind turbine blades. In view of the problem of diverse damage size and shape, deformable convolution is introduced into the model so that the model has a stronger ability to extract blade damage and can better identify the health status of blades.
[0059] (3) This application uses short-time octaves to obtain localized octave features in audio signals, and combines them with multi-input BPNN to overcome the curse of dimensionality without losing feature information. The combination of the two constitutes a high-accuracy, high-efficiency, low-cost and low-complexity wind blade damage detection method that can meet the requirements of most practical applications and promote the development of wind blade health monitoring systems. BRIEF DESCRIPTION OF THE DRAWINGS
[0060] Figure 1 It is a flow chart of a wind turbine blade damage identification method based on audio and image provided in an embodiment of the present application;
[0061] Figure 2 It is a schematic diagram of the frequency band division of the octave in the embodiment of the present application;
[0062] Figure 3 This is a schematic diagram of the octave-BPNN model of an embodiment of the present application;
[0063] Figure 4 This is a model training effect diagram of an embodiment of the present application;
[0064] Figure 5 is a schematic diagram of a confusion matrix of an embodiment of the present application;
[0065] Figure 6 is a schematic diagram of a convolution operation of an embodiment of the present application;
[0066] Figure 7 is a schematic structural diagram of a lightweight convolutional structure layer in an embodiment of the present application;
[0067] Figure 8 Schematic diagram of the confusion matrix before and after optimization of the embodiment of the present application;
[0068] Fig. 9 is a schematic structural diagram of a wind turbine blade damage identification device based on audio and image provided in an embodiment of the present application;
[0069] Fig.10 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0071] The term "and / or" in this application is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this application indicates that the associated objects are in an or relationship, for example, A / B means A or B.
[0072] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.
[0073] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.
[0074] First, the technical terms involved in the embodiments of the present application are introduced.
[0075] Next, the technical solutions provided in the embodiments of the present application are introduced.
[0076] Reference Figure 1 The present application provides a method for identifying wind turbine blade damage based on audio and image, comprising:
[0077] S101. Acquire the audio signal and image data of the fan blade;
[0078] S102. Dividing the frequency spectrum of the audio signal into a plurality of frequency bands using an octave algorithm, and extracting sound pressure level characteristics of the plurality of frequency bands;
[0079] S103. Inputting the sound pressure level feature into the trained back propagation neural network model BPNN to obtain the audio recognition result of the damage condition of the fan blade;
[0080] S104. Inputting the image data into a trained lightweight convolutional neural network model to obtain an image recognition result of the damage condition of the wind turbine blade;
[0081] S105. Comprehensively judge the audio recognition result and the image recognition result to obtain a comprehensive recognition result of the damage of the wind turbine blade;
[0082] Among them, the back propagation neural network model is trained based on audio samples and audio labels, and the lightweight convolutional neural network model is established based on a lightweight convolutional structure and trained based on image samples and image labels.
[0083] Specifically, in this embodiment, the audio signal and image data of the fan blades are first obtained from the fan system, wherein the audio signal may include sound information generated when the fan is running, and the image data reflects the visual state of the fan blades, including damage, cracks, etc.
[0084] Secondly, the acquired audio signal is processed by using the octave band algorithm to divide its spectrum into multiple frequency bands. The sound pressure level characteristics in each frequency band are extracted. These characteristics may include information such as sound intensity and frequency distribution within the frequency range.
[0085] Furthermore, on the one hand, the sound pressure level features are input into the trained back propagation neural network model BPNN to obtain the audio recognition results of the damage of the wind turbine blades. The sound pressure level features extracted from the audio signal are input as input into the pre-trained back propagation neural network (BPNN) model. The BPNN model has been trained with audio samples and corresponding damage labels. The BPNN will output an identification result of the damage of the wind turbine blades based on the input sound pressure level features, such as determining whether there are signs of abnormal sound or mechanical vibration.
[0086] On the other hand, the image data is input into the trained lightweight convolutional neural network model to obtain the image recognition results of the wind turbine blade damage. The image data is input into the trained lightweight convolutional neural network (CNN) model, which adopts a lightweight convolutional structure and is trained on image samples and corresponding damage labels. The CNN model will analyze the input wind turbine blade image to identify any visible visual features such as damage, cracks or deformation.
[0087] Finally, the audio recognition results and image recognition results are comprehensively judged to obtain the comprehensive recognition results of the wind turbine blade damage. By comprehensively considering the results obtained from the BPNN and CNN models and utilizing the information of different data modalities (audio and image), a more comprehensive understanding and judgment of the overall damage of the wind turbine blade can be obtained, thus improving the accuracy and reliability of wind turbine blade damage detection.
[0088] Back Propagation Neural Network Model (BPNN): Specialized for processing audio data, trained to identify associations between different sound patterns and types of damage.
[0089] Lightweight convolutional neural network model (CNN): used to process image data, with efficient computing power and real-time performance, suitable for application requirements in industrial scenarios.
[0090] By combining audio and image recognition technology, the accuracy and efficiency of wind turbine blade damage detection can be effectively improved, which helps to carry out preventive maintenance in advance and reduce the occurrence of equipment failures.
[0091] The present application analyzes the audio signals of the fan blades through an octave algorithm to extract the sound pressure level characteristics of different frequency bands. The frequency band analysis helps to capture possible fault signals in different frequency ranges, and uses a lightweight convolutional neural network to process the image data of the fan blades to detect visual damage characteristics. The combined analysis results of the audio and image data can improve the accuracy of identifying the overall damage status of the fan blades.
[0092] Optionally, the dividing the frequency spectrum of the audio signal into a plurality of frequency ranges by using an octave algorithm, and extracting sound pressure level features of the plurality of frequency ranges, comprises:
[0093] The frequency spectrum of the audio signal is divided into a plurality of frequency ranges by using a reference frequency; the upper cut-off frequency, the lower cut-off frequency and the center frequency of the plurality of frequency ranges are in geometric proportion; the upper cut-off frequency is determined according to the reference frequency and a preset parameter, and the lower cut-off frequency and the center frequency are determined according to the upper cut-off frequency and a preset octave number;
[0094] Obtain a frequency range sound pressure magnitude based on the lower cut-off frequency, the center frequency, and the frequency component amplitude, and determine the sound pressure level characteristic based on the frequency range sound pressure magnitude and the reference sound pressure;
[0095] After extracting the sound pressure level features of the multiple frequency ranges, the method further includes:
[0096] The sound pressure level characteristics are normalized and dimensionless through minimum and maximum value normalization.
[0097] Specifically, the octave algorithm in this application divides the spectrum of the audio signal into a series of frequency ranges with geometric lengths using 1000 Hz as the reference frequency. The upper cutoff frequency, lower cutoff frequency and center frequency of each frequency range are in geometric relationship, as shown below:
[0098]
[0099] Among them, f c Indicates the center frequency of the frequency range, f l Indicates the lower cutoff frequency of the frequency range, f u Indicates the upper cutoff frequency of the frequency range, n is an integer. N represents the octave number, N=3 is 1 / 3 octave, N=6 is 1 / 6 octave. Different octave numbers have different spectral resolutions, and the appropriate octave number needs to be selected according to the characteristics of the actual audio signal.
[0100] Reference Figure 2 , Figure 2 It is a schematic diagram of the octave frequency band division of the embodiment of the present application, showing the relationship between the full frequency band and the frequency range.
[0101] Furthermore, when there is damage to the fan blade, the sound pressure of the frequency range that overlaps with the frequency range of the damage information is greater than the sound pressure of the normal blade, which can reflect the damage of the blade and can be used to detect the health status of the blade. The sound pressure calculation method of the frequency range is shown in the following formula:
[0102]
[0103] Among them, P i Indicates the sound pressure of the frequency range, F j Indicates the amplitude of the frequency component. To facilitate subsequent algorithm analysis, the sound pressure characteristics need to be logarithmically processed to obtain the sound pressure level characteristics, as shown in the following formula:
[0104]
[0105] Among them, S i Indicates the sound pressure level of the frequency range, P ref represents the reference sound pressure. For air, P ref Usually 2×10 -5 Pa.
[0106] It should be noted that the audio signal collected by the microphone is affected by factors such as device sensitivity, blade direction, and external environmental noise. If the sound pressure level feature is used directly, the octave characteristics of different audio signals will be different in size. In the process of analyzing the actual audio signal, it is found that the curve changes of the octave characteristics can reflect different types of blade damage. Therefore, the minimum-maximum normalization (Min-Max Normalization, MMN) is used to normalize and dimensionlessly process the sound pressure level characteristics, as shown in the following formula:
[0107]
[0108] It needs to be further explained that in the process of octave feature extraction, how to choose the octave number is the key content. Different octave numbers have different frequency resolutions. When the octave number is small, the width of the frequency range is large and the frequency resolution is low; when the octave number is large, the width of the frequency range is small and the frequency resolution is high. Common octave numbers are 3, 6, 12, 24, etc. In the analysis of audio signals of different types of blade damage, it can be seen that the frequency resolution of 1 / 3 octave is low, and only some types of blade damage information can be extracted. The frequency resolution of 1 / 12 octave is high. Although it can extract all types of blade damage information, it is easily interfered by the external environment and the dimension of the feature is large, which is not conducive to subsequent blade health status detection. Therefore, the present application uses 1 / 6 octave to extract blade damage information in audio signals, which can not only effectively extract all types of blade damage information, but also has a small feature dimension to reduce the complexity of subsequent algorithms.
[0109] Optionally, the back propagation neural network model includes a multi-layer perceptron MLP and a linear layer;
[0110] The multilayer perceptron is used to divide the characteristic matrix of the sound pressure level characteristics into a plurality of octave characteristic vectors and obtain a plurality of preliminary recognition results;
[0111] The linear layer is used to perform vector concatenation on multiple result vectors of the preliminary recognition results to obtain the final audio recognition result.
[0112] Reference Figure 3 , Figure 3 This is a schematic diagram of the Octave-BPNN model of the embodiment of the present application. The back propagation neural network model in the present application is a multi-input BP neural network, including a multi-layer perceptron MLP and a linear layer.
[0113] Specifically, the multi-input BPNN consists of two stages, namely preliminary judgment and comprehensive judgment.
[0114] The preliminary judgment part is a multi-input multi-output network composed of MLP, that is, the multi-layer perceptron MLP mentioned above.
[0115] The audio signal of the fan blade is first extracted through short-time octave features, and the extracted feature matrix is input into the multi-input multi-output network. The preliminary judgment part divides the feature matrix extracted by the short-time octave into multiple octave feature vectors, which are input into the MLP at the same time and output multiple preliminary diagnosis results. The preliminary diagnosis part has two advantages. First, the shared multi-layer perceptron network can better represent the relationship between the octave features and the health status of the fan blades, and reduce the amount of calculation and parameters of the STO-MIBPNN. Second, the input data are all octave features, and using the same network can better learn the internal correlation information of the short-time octave features.
[0116] The comprehensive judgment part is composed of a linear layer, namely the above-mentioned linear layer. This part concatenates multiple result vectors of preliminary judgment and makes the final judgment and output.
[0117] Optionally, the training method of the back propagation neural network model includes:
[0118] Constructing an initial model of the back propagation neural network model, and randomly initializing the weight matrix and bias vector of the multilayer perceptron;
[0119] Inputting the sound pressure level characteristics corresponding to the audio sample into the initial model, and outputting the health status information of the fan blades;
[0120] The error between the health status information and the audio label is calculated, and the weight matrix and the bias vector are calculated and iteratively updated according to the error until a stopping condition is reached to obtain a trained back propagation neural network model.
[0121] Specifically, the training process of BPNN in this application is as follows:
[0122] Step 1: Randomly initialize the weight matrix and bias vector of the MLP between -0.5 and 0.5.
[0123] Step 2: Input the acoustic feature data of the wind turbine blades into the input layer, pass through the hidden layers of the multi-layer perceptron layer by layer, and output the health status information of the wind turbine blades through the output layer. The output of each neuron is:
[0124]
[0125] in, represents the output of the nth neuron in the kth layer, f represents the activation function of the neuron, and H k-1 represents the number of neurons in the k-1th layer, represents the weight from the mth neuron in the k-1th layer to the nth neuron in the kth layer, represents the bias of the nth neuron in the kth layer.
[0126] Step 3: Calculate the error between the actual output and the expected output of the MLP. The loss function is as follows:
[0127]
[0128] Among them, J represents the current loss cost, represents the loss function, represents the actual output of the MLP, and y represents the actual label of the sample.
[0129] Step 4: Calculate and update the weights and biases of the neural network. The back propagation algorithm calculates the partial derivatives of the weight matrix and bias vector through the loss cost, and modifies the weight matrix and bias of the previous round of training according to the learning rate to reduce the loss cost and improve the model recognition ability. The modification of the weight matrix and bias vector is shown below.
[0130]
[0131] Among them, W represents the weight matrix before correction, b represents the bias vector before correction, and W * represents the corrected weight matrix, b represents the corrected bias vector, and α represents the learning rate.
[0132] Step 5: Determine whether the stopping condition is met. If so, save the trained model and exit. If not, return to step 2.
[0133] According to the universal approximation theorem, a trained multilayer perceptron can approximate the potential relationship function between sample features and labels with arbitrary accuracy. However, the learning rate, the initial value of the network weights, and the number of hidden layers will affect the generalization ability of the model. Although the overfitting model has excellent performance on the training set, the effect on the test set is relatively poor. The underfitting model performs poorly on both the training set and the validation set. The model can be optimized by adaptive learning rate, increasing the number of samples, and terminating model training early at an appropriate number of iterations.
[0134] Optionally, it also includes:
[0135] Performing performance evaluation on the trained back propagation neural network model to obtain a confusion matrix of the model, wherein the confusion matrix includes true positive examples, false negative examples, false positive examples, and true negative examples;
[0136] The accuracy, precision, recall and F1 value of the back propagation neural network model are determined based on the true positive examples, false negative examples, false positive examples and true negative examples.
[0137] It should be noted that the training of BPNN in this application requires the use of a data set to adjust the internal parameters of the model, and the collected audio signals are divided into a training set, a validation set, and a test set. The training set is a data set used to train the internal parameters of the model, and adjust the model parameters to obtain better classification results. The validation set is a data set that verifies the generalization ability of the current model during the training process, and adjusts the training direction of the model weights and biases. The test set is used to evaluate the generalization ability of the model and can reflect the performance of the model in real situations. In order to avoid the bias introduced by the data set division that leads to a decrease in the generalization ability of the STO-MIBPNN model, the StratifiedShuffleSplit function provided by scikit-learn is used to implement stratified sampling of the fan blade audio data set. The training set, validation set, and test set account for 70%, 10%, and 20%, respectively. The specific distribution of each category of data is shown in Table 1.
[0138] Table 1 Distribution of audio datasets
[0139]
[0140] Furthermore, BPNN involves optimization in many cases. Due to the particularity of BPNN, its optimization problem seems simple, but in fact it becomes very complicated due to the combination of nonlinear functions. Without prior experience, problems such as local minima, gradient disappearance, and gradient explosion make it difficult for BPNN to obtain a reasonable accuracy. Therefore, it is necessary to set some parameters to adjust the model training process in order to obtain a good model.
[0141] (1) Model structure: The structure of MIBPNN is designed according to the dimension of octave features and the size of the data set. MIBPNN consists of two stages: preliminary judgment and comprehensive judgment. The preliminary judgment consists of three stacked linear layers with 128, 64 and 4 neurons respectively, and the activation function is the LeakyReLU function provided by Pytorch. The output of the preliminary judgment is flattened by the Flatten layer and input into the comprehensive judgment. The comprehensive judgment consists of a linear layer, and finally outputs the probability of each category.
[0142] (2) Loss function: The training process of the deep learning model is to modify the model's weight matrix and bias vector according to the size of the loss cost to continuously reduce the loss cost. The loss function determines the direction of model parameter optimization to find the global optimal point. The acoustic detection of wind turbine blades is a multi-classification problem. The most commonly used loss function is the cross entropy function, which can be written as:
[0143]
[0144] Among them, y c represents the true label of category C, p cRepresents the probability of model output category C, and M represents the number of categories in the multi-classification problem. Pytorch's Cross Entropy Loss loss function is used, and loss weights for each category are added to avoid the problem of sample imbalance.
[0145] (3) Optimizer: The optimizer calculates the weight matrix and bias vector gradient of the MLP through the chain rule during the model training process, and updates the parameters according to the learning rate and gradient to reduce the loss cost, so that the optimized model can better identify the health status of the blade. The current mainstream optimizers include Stochastic Gradient Descent (SGD), Adam, and Stochastic Gradient Descent with Momentum (SGDM). Adam is used as the optimizer for model training. Adam is an optimizer based on learning rate adaptation. It adjusts the correction size of different parameters by calculating the first-order moment estimate and second-order moment estimate of the weight gradient of the deep learning model.
[0146] (4) Learning rate strategy: The learning rate determines the size of the correction of the model weight each time the gradient descends. When the learning rate is set to a small value, the model can converge well during training, but the training process is slow or even unable to learn. If the learning rate is set to a large value, the performance of the model will fluctuate during training, and even cause gradient explosion. Therefore, the setting of the learning rate requires a trade-off between performance and speed. Use Pytorch's ReduceLROnPlateau learning rate scheduler. Choose a suitable initial learning rate. When the loss cost of the validation set does not change during the training process, the learning rate is reduced, and finally a higher performance model is obtained.
[0147] (5) Evaluation indicators: In the wind turbine blade damage detection task, the performance of the trained model needs to be evaluated to determine the generalization ability of the model. Among them, the confusion matrix is an effective evaluation criterion that can reflect the effect of the model from different aspects. The confusion matrix consists of true positive examples (True Positive, TP), false negative examples (False Negative, FN), false positive examples (False Positive, FP) and true negative examples (True Negative, TN), as shown in Table 2:
[0148] Table 2 Confusion matrix
[0149]
[0150] For the evaluation of the generalization ability of the model, accuracy is the primary evaluation indicator, but it is impossible to accurately evaluate the generalization ability of the model in the case of unbalanced samples. Therefore, precision, recall and F1 value are introduced to evaluate the generalization ability of the model from multiple aspects. The following formulas are the calculation formulas for accuracy, precision, recall and F1 respectively. In the F1 value calculation process, F1 is weighted averaged, and the weight is the proportion of samples of different types of leaf damage in the data set.
[0151]
[0152] Furthermore, in the embodiment of the present application, the training set is used to train the STO-MIBPNN model, with an initial learning rate of 0.01, a batch size of 64, and 1000 iterations. After each iteration, the validation set is used to analyze the effect of the current model, and the learning rate is adjusted according to the loss cost of the validation set. During the STO-MIBPNN training process, the accuracy and loss cost of the training set and validation set are as follows: Figure 4 As shown in the figure, (a) is the accuracy curve, and (b) is the iterative loss curve.
[0153] As can be seen from the figure, the accuracy of the training set and the validation set increases rapidly in the early stage and gradually stabilizes when the number of iterations is 200. When the training process converges, the difference between the accuracy of the training set and the validation set is about 2.27%, indicating that the model is not overfitted. In addition, during the model training process, the algorithm will save the optimal model based on the F1 value of the model in the validation set.
[0154] Furthermore, the model with the highest weighted F1 value in the validation set is taken to evaluate the generalization performance in the test set. The evaluation indicators in the test set are shown in Table 3 below. In the table, 28 and 57 are the dimensions of the feature vector after dimensionality reduction, where 28 indicates that the dimension of PCA retains 90% of the information, and 57 indicates that the dimension of PCA retains 95% of the information. From models 1 and 2 in Table 3, it can be seen that the accuracy of STO-BPNN is 2.32% lower than that of Octave-BPNN. This is due to the dimensionality disaster caused by the large characteristic dimension of the short-time octave, and the model is degraded. Among them, the characteristic dimension of the short-time octave is 310, while the characteristic dimension of the octave is 31. From models 2 to 8 in Table 2.4, it can be seen that the dimensionality reduction method can effectively improve the recognition effect of the model, and the recognition accuracy is greater than 90%. Among the three dimensionality reduction methods, KPCA has the best effect, indicating that the short-time octave feature is a nonlinear relationship. When the feature length after dimensionality reduction is 57, the accuracy is 94.73%, which is 5.81% and 3.49% higher than STO-BPNN and Octave-BPNN, respectively. STO-MIBPNN is the best performing model with the highest recognition accuracy of 95.04%, which is 6.12% and 3.8% higher than STO-BPNN and Octave-BPNN, respectively. In addition, the precision, recall and F1 of STO-MIBPNN are all greater than 95%.
[0155] Table 3 Evaluation indicators of different models on the test set
[0156]
[0157] Refer to Table 4 below, which describes the recognition accuracy of STO-MIBPNN and the baseline model in each category of the test set. Among them, STO-MIBPNN is the best performing model, with the best recognition accuracy for damage 1, damage 2 and damage 3, which are 91.19%, 98.00% and 95.81% respectively. By comparing with Octave-BPNN, STO-MIBPNN has improved its recognition ability for audio signals in the normal, damage 1, damage 2 and damage 3 categories, which are improved by 4.91%, 1.25%, 3.14% and 5.58% respectively, indicating that STO-MIBPNN can efficiently diagnose the health status of wind turbine blades. By comparing with STO-BPNN, STO-MIBPNN has improved its recognition accuracy for audio signals in the normal, damage 1, damage 2 and damage 3 categories by 3.12%, 8.8%, 5.14% and 8.83% respectively, verifying that STO-MIBPNN solves the dimensionality curse problem without losing feature information.
[0158] Table 4 Accuracy of different models in each category of the test set
[0159]
[0160]
[0161] Reference Figure 5 , Figure 5 The confusion matrix of Octave-BPNN and STO-MIBPNN in the test set is shown. The number of misjudged samples of STO-MIBPNN is 47, and the number of misjudged samples of Octave-BPNN is 83, which is 36 fewer misjudged samples. In addition, the recognition accuracy of STO-MIBPNN for the four categories of samples is higher than that of Octave-BPNN, indicating that the short-time octave-band based multi-input BPNN proposed in the embodiment of the present application can identify various types of blade damage with high precision, providing a new solution for wind turbine blade acoustic detection.
[0162] Optionally, the lightweight convolutional neural network model includes a convolution layer, a lightweight convolutional structure layer, a global average pooling layer and a fully connected layer;
[0163] The convolution layer is used to extract features from the image data to obtain a feature image;
[0164] The lightweight convolution structure layer is used to process the visual information of the feature image at different scales to obtain multi-scale feature information;
[0165] The global average pooling layer is used to perform global average pooling on the multi-scale feature information, averaging the two-dimensional graphic information of each channel to obtain a feature value;
[0166] The fully connected layer is used to judge the eigenvector corresponding to the eigenvalue and predict the probability of each image recognition result.
[0167] Specifically, in the embodiment of the present application, the convolutional neural network usually includes multiple convolutional layers. The convolutional layer close to the input has a smaller receptive field and is used to extract local feature information in the image. The convolutional layer close to the output has a larger receptive field and is used to extract global feature information of the image. The convolutional layer extracts feature information of the image by sliding the convolution kernel, and has the characteristics of local connection and parameter sharing. Taking the convolution kernel size of 3×3 as an example, the calculation formula of the convolution operation is as follows.
[0168]
[0169] Among them, g represents the feature map output by the convolution layer, (x, y) represents the coordinates of the central pixel, f represents the image input to the convolution layer, and h represents the convolution kernel of the convolution layer.
[0170] Reference Figure 6 , Figure 6 It is a schematic diagram of convolution operation;
[0171] When constructing a convolutional layer, you also need to define stride and padding. The stride is the distance the convolution kernel moves during the operation. You can control the size of the output feature map by setting different step sizes. Padding is to control the size of the input and output feature maps to be the same by extending the boundaries of the image. There are two main methods: zero padding and copying edge pixels.
[0172] Optionally, the convolution layer is a deformable convolution layer.
[0173] The lightweight convolution structure layer includes branch A, branch B, branch C, branch D, a splicing sublayer and a residual structure sublayer;
[0174] The branch A is used to perform a 1×1 convolution operation on the feature image;
[0175] The branch B is used to perform 1×1 point-by-point convolution and 3×3 depth convolution on the feature image;
[0176] The branch C is used to perform 1×1 point-by-point convolution and two 3×3 depth convolutions on the feature image;
[0177] The branch D is used to perform a 3×3 maximum pooling operation on the feature image;
[0178] The stitching sublayer is used to stitch the output images of the branch A, branch B, branch C and branch D to obtain a stitched image;
[0179] The residual structure sublayer is used to add the feature image and the spliced image to obtain an output feature map.
[0180] Reference Figure 7, the lightweight convolution structure layer in this application is based on Inception, and performs 1×1 convolution operation, 3×3 convolution operation, 5×5 convolution operation and 3×3 maximum pooling operation on the input feature map, which can realize multi-scale feature extraction and increase the depth and width of the model at a small computational cost. Splitting the 5×5 convolution operation of branch C into two 3×3 convolution operations not only reduces the number of parameters and the amount of calculation of the model, but also integrates multiple nonlinear activation function layers to increase the nonlinear processing capability of the model. After performing convolution operations of different scales, LWC splices and merges the processed feature maps, which will cause the dimension of the feature map to explode, and then cause the amount of calculation of the model to increase sharply. Therefore, a 1×1 convolution operation is added after the spliced and merged feature map to realize multi-scale feature information fusion and reduce the dimension of the output feature map, greatly reducing the amount of calculation of the model. In order to deploy the convolutional neural network to the edge computing device in the actual application scenario, it is necessary to further reduce the number of parameters and the amount of calculation of the model, so DSC is introduced to replace the traditional convolution layer. The 3×3 convolution operations of branches B and C are replaced by 3×3 depthwise convolution operations and 1×1 pointwise convolution operations. Finally, a residual structure is introduced to avoid problems such as degradation and gradient disappearance during the stacking of convolutional structures. The input feature map and the feature map output by the 1×1 convolution layer are added as the final output feature map.
[0181] In view of the characteristics of wind turbine blade data, the embodiment of the present application uses a lightweight convolutional structure to construct a lightweight convolutional neural network (WTBMoibleNet), and its structure is shown in Table 5. WTBMoibleNet has a total of 9 layers, namely traditional convolutional layer, 6 lightweight convolutional structure layers, global average pooling layer (Global Average Pooling, GAP) and fully connected layer (Fully Connected, FC). The lightweight convolutional structure can process visual information at different scales, which is suitable for higher-level visual information feature extraction, so the traditional convolutional layer is used as the first layer of the model. Then 6 LWC layers are used to process the visual information of the input image at different scales. Then the GAP layer is used to perform global average pooling on the extracted feature information, and the two-dimensional image information of each channel is averaged to generate a feature value. This operation can significantly reduce the number of parameters, enhance the accuracy and stability of the model, suppress the overfitting phenomenon in the model, and can map multiple features to a low-dimensional space, thereby significantly improving the generalization ability of the model. Finally, the FC layer is used to judge the feature vector of the global average pooling and predict the probability of different categories. In the whole structure, except for the first and last layers, Batchnorm layer and LeakReLu nonlinear activation function are added after other layers. Hardswish and Softmax nonlinear activation functions are added to the first and last layers respectively.
[0182] Table 5 Lightweight convolutional neural network structure
[0183] The following is a detailed description of WTBMobileNet and its optimization analysis:
[0184] In order to obtain clear images of wind turbine blades, this application chooses to collect images of wind turbine blade surface damage, and collects dozens of videos and hundreds of photos. Each video is processed by frame extraction, and finally 51,979 original wind turbine blade images are obtained, including damage types such as cracks, protective film damage, and lightning strikes. The collected original images are screened, and the damaged parts are cropped into sub-images of 586×371 size, with a total of 9235 images. Among them, there are 6431 normal samples and 2804 abnormal samples. The number of normal samples is about 2.5 times the number of abnormal samples. There is a serious sample imbalance, which increases the difficulty of model training.
[0185] In order to ensure the consistency of data distribution of each category in the data set, the stratified sampling method is used to divide the data set. The image data set is divided into training set, validation set and test set in a ratio of 7:1:2. The specific distribution of each category of data is shown in Table 6. There are a total of 6464 training set image samples, of which 4502 are normal samples and 1962 are abnormal samples. There are a total of 924 validation set image samples, of which 643 are normal samples and 281 are abnormal samples. There are a total of 1847 test set image samples, of which 1286 are normal samples and 561 are abnormal samples.
[0186] Table 6 Image dataset distribution
[0187]
[0188] The training parameters are fine-tuned according to the computing power of the high-performance computing public service platform. The specific parameter values are shown in Table 7. The loss function uses a weighted cross entropy function to reduce the loss weight of normal samples and increase the loss weight of abnormal samples to solve the sample imbalance problem of the image dataset. The initial value of the learning rate is 0.01 and the learning rate strategy is ReduceLROnPlateau provided by Pytorch, which can be adaptively scaled according to the performance of the model in the validation set. The optimizer is Adam, the number of iterations is 100, the batch size of each training is 64, and the size of the fan blade image input to the convolutional neural network is 512×512.
[0189] Table 7 Network model training parameters
[0190]
[0191] In order to verify the effectiveness of WTBMobileNet for machine vision inspection of wind turbine blades, common convolutional neural networks are compared with it. Since the lightweight convolutional structure of WTBMobileNet uses residual network, Inception structure and depth-separable convolutional structure, ResNet34, ResNet50, GoogLeNet and MobileNetV3 are selected as baseline models. In addition, in order to compare and analyze the effectiveness of convolutional network structure for wind turbine blade damage detection, VGG16 model and VGG19 model are added to analyze the shortcomings of traditional convolutional layer stacking.
[0192] In the embodiment of the present application, the input image size is 512×512 as an example, and the parameter quantity and computation amount of the model are calculated through the Python thop library. The specific values are shown in Table 8.
[0193] Table 8 Comparative analysis of parameters and computational complexity of WTBMobileNet and baseline models
[0194]
[0195] The number of model parameters directly affects the size of video memory occupied during model operation. The model with the smallest number of parameters in the baseline model is MobileNetV3, with a parameter size of 2.974MB. The baseline model with the largest number of model parameters is ResNet50, with a parameter size of 23.512MB. The parameter size of WTBMobileNet is smaller than all baseline models, only 0.313MB. The model sizes of MobileNetV3 and ResNet50 are approximately 9 times and 75 times that of lightweight convolutional neural networks, respectively. The model with the smallest number of floating point operations (FLOPs) in the baseline model is MobileNetV3, with a number of operations of 1.161GFLOPs. The model with the largest number of operations is VGG19, with a number of operations of 101.999GFLOPs. WTBMobileNet is the model with the smallest number of floating point operations, with a number of operations of 0.423GFLOPs, which is 36.4% and 4.15% of the operations of MobileNetV3 and VGG19, respectively. Therefore, compared with common baseline models, WTBMobileNet has significantly reduced the number of parameters and computation, making the model more suitable for edge computing devices and laying the foundation for the subsequent implementation of edge computing.
[0196] In order to verify the high performance of lightweight convolutional neural networks in machine vision inspection of wind turbine blades, the accuracy, precision, recall and F1 value comparison experiments were conducted on WTBMobileNet and common baseline models using the test set under the same training conditions. This application uses the classification_report function in the sklearn library to calculate the various evaluation indicators of the model, and the specific comparison results are shown in Table 9. As can be seen from Table 9, WTBMobileNet has a good performance in wind turbine blade image damage detection, with accuracy, precision, recall and F1 values of 90.58%, 90.59%, 90.58% and 0.9058 respectively. Among them, VGG16 and VGG19 performed poorly, with accuracy rates of 69.63% and 69.73% respectively. This shows that it is difficult for traditional convolutional layer stacking models to learn damage information in wind turbine blade images, and it is necessary to optimize the convolution structure to improve the learning ability of the model. Among the baseline models, GoogLeNet is the best performing model with a recognition accuracy of 90.63%, which is 0.05% higher than WTBMobileNet. However, the number of parameters and computational complexity of GoogLeNet are approximately 18 times and 19 times that of WTBMobileNet, respectively, making it difficult to deploy on edge computing devices to reduce the ring network pressure of wind farms.
[0197] Table 9 Comparative analysis of evaluation indicators of WTBMobileNet and baseline models
[0198]
[0199] In summary, the lightweight convolutional neural network designed in this application can achieve good recognition results in machine vision inspection of wind turbine blades. With the model parameters and computational complexity reduced exponentially, the recognition accuracy is only slightly reduced, and it can be deployed on edge computing devices with limited computing power and memory to achieve high-precision blade damage monitoring.
[0200] Furthermore, the lightweight convolutional neural network model is optimized:
[0201] By observing the damaged images of wind turbine blades, we can see that the damaged shapes of wind turbine blades are complex and changeable. However, the geometric shape of traditional convolution operations is fixed and cannot adapt to image information of different scales or deformations. Therefore, a deformable convolution kernel is introduced. The focus area of the convolution kernel will be adaptively adjusted according to the characteristics of the blade damage and is no longer limited to a rectangle. Therefore, it can better learn the shape and size of wind turbine blade damage and improve the performance of the model in machine vision inspection of wind turbine blades. The first traditional convolution layer is replaced with a deformable convolution layer. The specific model structure is shown in Table 10.
[0202] Table 10 Optimized WTBMobileNet network structure
[0203]
[0204]
[0205] In order to analyze the impact of the deformable convolution layer on the lightweight convolutional neural network, the test set is used to compare and analyze the WTBMobileNet before and after optimization, as shown in Tables 11 and 12. As shown in Table 11, after using the deformable convolution layer, the number of model parameters has not changed significantly, and the amount of calculation has increased by 0.004GFLOPs. Therefore, the optimized WTBMobileNet has no significant increase in the number of parameters and the amount of calculation, and can be deployed to edge computing devices to realize the health status monitoring of wind turbine blades based on edge computing.
[0206] Table 11 Comparative analysis of parameters and computational complexity before and after WTBMobileNet optimization
[0207]
[0208] Refer to Table 12 below, which shows the evaluation indicators and accuracy of each category of WTBMobileNet before and after optimization. After using the deformable convolution layer, the accuracy, precision and recall rates increased by 0.43%, 0.47% and 0.43% respectively, proving that deformable convolution has positive optimization for WTBMobileNet. In addition, from the accuracy of each category, it can be seen that the optimized WTBMobileNet has no improvement in the recognition effect of normal samples, but the recognition accuracy of abnormal samples has increased by 1.43%, proving that deformable convolution has a good recognition ability for wind turbine blade damage and can better learn surface damage of wind turbine blades of different proportions and shapes.
[0209] Table 12 Comparative analysis of evaluation indicators before and after WTBMobileNet optimization
[0210]
[0211] Reference Figure 8 The left and right figures in the figure respectively represent the confusion matrices of WTBMobileNet before and after optimization. Before optimization, there are 174 misjudged samples of WTBMobileNet, including 88 normal images and 86 abnormal images. After optimization, there are 166 misjudged samples of WTBMobileNet, including 88 normal images and 78 abnormal images. After the introduction of deformable convolution, the number of misjudgments of abnormal images decreased by 8, which verifies that deformable convolution can adaptively adjust the shape of the convolution kernel to improve the ability to detect wind turbine blade damage.
[0212] Reference Fig. 9The present application also provides a wind turbine blade damage identification device based on audio and image, comprising:
[0213] A data acquisition module 910 is used to acquire audio signals and image data of wind turbine blades;
[0214] An audio extraction module 920, configured to divide the frequency spectrum of the audio signal into a plurality of frequency bands using an octave algorithm, and extract sound pressure level features of the plurality of frequency bands;
[0215] An audio recognition module 930 is used to input the sound pressure level feature into a trained back propagation neural network model BPNN to obtain an audio recognition result of the damage condition of the fan blade;
[0216] An image recognition module 940 is used to input the image data into a trained lightweight convolutional neural network model to obtain an image recognition result of the damage condition of the wind turbine blade;
[0217] A comprehensive judgment module 950 is used to make a comprehensive judgment on the audio recognition result and the image recognition result to obtain a comprehensive recognition result of the damage condition of the wind turbine blade;
[0218] Among them, the back propagation neural network model is trained based on audio samples and audio labels, and the lightweight convolutional neural network model is established based on a lightweight convolutional structure and trained based on image samples and image labels.
[0219] Optionally, the dividing the frequency spectrum of the audio signal into a plurality of frequency ranges by using an octave algorithm, and extracting sound pressure level features of the plurality of frequency ranges, comprises:
[0220] The frequency spectrum of the audio signal is divided into a plurality of frequency ranges by using a reference frequency; the upper cut-off frequency, the lower cut-off frequency and the center frequency of the plurality of frequency ranges are in geometric proportion; the upper cut-off frequency is determined according to the reference frequency and a preset parameter, and the lower cut-off frequency and the center frequency are determined according to the upper cut-off frequency and a preset octave number;
[0221] Obtain a frequency range sound pressure magnitude based on the lower cut-off frequency, the center frequency, and the frequency component amplitude, and determine the sound pressure level characteristic based on the frequency range sound pressure magnitude and the reference sound pressure;
[0222] After extracting the sound pressure level features of the multiple frequency ranges, the method further includes:
[0223] The sound pressure level characteristics are normalized and dimensionless through minimum and maximum value normalization.
[0224] Optionally, the back propagation neural network model includes a multi-layer perceptron MLP and a linear layer;
[0225] The multilayer perceptron is used to divide the characteristic matrix of the sound pressure level characteristics into a plurality of octave characteristic vectors and obtain a plurality of preliminary recognition results;
[0226] The linear layer is used to perform vector concatenation on multiple result vectors of the preliminary recognition results to obtain the final audio recognition result.
[0227] Optionally, the training method of the back propagation neural network model includes:
[0228] Constructing an initial model of the back propagation neural network model, and randomly initializing the weight matrix and bias vector of the multilayer perceptron;
[0229] Inputting the sound pressure level characteristics corresponding to the audio sample into the initial model, and outputting the health status information of the fan blades;
[0230] The error between the health status information and the audio label is calculated, and the weight matrix and the bias vector are calculated and iteratively updated according to the error until a stopping condition is reached to obtain a trained back propagation neural network model.
[0231] Optionally, an evaluation module is further included, and the evaluation module is used to:
[0232] Performing performance evaluation on the trained back propagation neural network model to obtain a confusion matrix of the model, wherein the confusion matrix includes true positive examples, false negative examples, false positive examples, and true negative examples;
[0233] The accuracy, precision, recall and F1 value of the back propagation neural network model are determined based on the true positive examples, false negative examples, false positive examples and true negative examples.
[0234] Optionally, the lightweight convolutional neural network model includes a convolution layer, a lightweight convolutional structure layer, a global average pooling layer and a fully connected layer;
[0235] The convolution layer is used to extract features from the image data to obtain a feature image;
[0236] The lightweight convolution structure layer is used to process the visual information of the feature image at different scales to obtain multi-scale feature information;
[0237] The global average pooling layer is used to perform global average pooling on the multi-scale feature information, averaging the two-dimensional graphic information of each channel to obtain a feature value;
[0238] The fully connected layer is used to judge the eigenvector corresponding to the eigenvalue and predict the probability of each image recognition result.
[0239] Optionally, the convolution layer is a deformable convolution layer.
[0240] Optionally, the lightweight convolution structure layer includes branch A, branch B, branch C, branch D, a splicing sublayer and a residual structure sublayer;
[0241] The branch A is used to perform a 1×1 convolution operation on the feature image;
[0242] The branch B is used to perform 1×1 point-by-point convolution and 3×3 depth convolution on the feature image;
[0243] The branch C is used to perform 1×1 point-by-point convolution and two 3×3 depth convolutions on the feature image;
[0244] The branch D is used to perform a 3×3 maximum pooling operation on the feature image;
[0245] The stitching sublayer is used to stitch the output images of the branch A, branch B, branch C and branch D to obtain a stitched image;
[0246] The residual structure sublayer is used to add the feature image and the spliced image to obtain an output feature map.
[0247] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.
[0248] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.
[0249] Reference Fig.10 Based on the method in the above embodiment, the embodiment of the present application provides an electronic device, which may include: a processor (processor) 1010, a communication interface (Communications Interface) 1020, a memory (memory) 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020, and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 can call the logic instructions in the memory 1030 to execute the method in the above embodiment.
[0250] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.
[0251] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.
[0252] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.
[0253] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.
[0254] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.
[0255] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0256] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.
[0257] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A method for identifying wind turbine blade damage based on audio and image, characterized in that: include: Acquire audio signals and image data of wind turbine blades; Dividing the frequency spectrum of the audio signal into a plurality of frequency bands by using an octave algorithm, and extracting sound pressure level features of the plurality of frequency bands; Inputting the sound pressure level feature into a trained back propagation neural network model BPNN to obtain an audio recognition result of the damage condition of the fan blade; Inputting the image data into a trained lightweight convolutional neural network model to obtain an image recognition result of the damage condition of the wind turbine blade; Comprehensively judging the audio recognition result and the image recognition result to obtain a comprehensive recognition result of the damage condition of the wind turbine blade; Among them, the back propagation neural network model is trained based on audio samples and audio labels, and the lightweight convolutional neural network model is established based on a lightweight convolutional structure and trained based on image samples and image labels.
2. The wind turbine blade damage identification method based on audio and image according to claim 1 is characterized in that: The method of dividing the frequency spectrum of the audio signal into a plurality of frequency ranges by using an octave algorithm and extracting sound pressure level features of the plurality of frequency ranges includes: The frequency spectrum of the audio signal is divided into a plurality of frequency ranges by using a reference frequency; the upper cut-off frequency, the lower cut-off frequency and the center frequency of the plurality of frequency ranges are in geometric proportion; the upper cut-off frequency is determined according to the reference frequency and a preset parameter, and the lower cut-off frequency and the center frequency are determined according to the upper cut-off frequency and a preset octave number; Obtain a frequency range sound pressure magnitude based on the lower cut-off frequency, the center frequency, and the frequency component amplitude, and determine the sound pressure level characteristic based on the frequency range sound pressure magnitude and the reference sound pressure; After extracting the sound pressure level features of the multiple frequency ranges, the method further includes: The sound pressure level characteristics are normalized and dimensionless through minimum and maximum value normalization.
3. The wind turbine blade damage identification method based on audio and image according to claim 1 is characterized in that: The back propagation neural network model includes a multi-layer perceptron MLP and a linear layer; The multilayer perceptron is used to divide the characteristic matrix of the sound pressure level characteristics into a plurality of octave characteristic vectors and obtain a plurality of preliminary recognition results; The linear layer is used to perform vector concatenation on multiple result vectors of the preliminary recognition results to obtain the final audio recognition result.
4. The wind turbine blade damage identification method based on audio and image according to claim 3 is characterized in that: The training method of the back propagation neural network model includes: Constructing an initial model of the back propagation neural network model, and randomly initializing the weight matrix and bias vector of the multilayer perceptron; Inputting the sound pressure level characteristics corresponding to the audio sample into the initial model, and outputting the health status information of the fan blades; The error between the health status information and the audio label is calculated, and the weight matrix and the bias vector are calculated and iteratively updated according to the error until a stopping condition is reached to obtain a trained back propagation neural network model.
5. The wind turbine blade damage identification method based on audio and image according to claim 1 is characterized in that: Also includes: Performing performance evaluation on the trained back propagation neural network model to obtain a confusion matrix of the model, wherein the confusion matrix includes true positive examples, false negative examples, false positive examples, and true negative examples; The accuracy, precision, recall and F1 value of the back propagation neural network model are determined based on the true positive examples, false negative examples, false positive examples and true negative examples.
6. The wind turbine blade damage identification method based on audio and image according to claim 1 is characterized in that: The lightweight convolutional neural network model includes a convolution layer, a lightweight convolutional structure layer, a global average pooling layer and a fully connected layer; The convolution layer is used to extract features from the image data to obtain a feature image; The lightweight convolution structure layer is used to process the visual information of the feature image at different scales to obtain multi-scale feature information; The global average pooling layer is used to perform global average pooling on the multi-scale feature information, averaging the two-dimensional graphic information of each channel to obtain a feature value; The fully connected layer is used to judge the eigenvector corresponding to the eigenvalue and predict the probability of each image recognition result.
7. The wind turbine blade damage identification method based on audio and image according to claim 6 is characterized in that: The convolution layer is a deformable convolution layer.
8. The wind turbine blade damage identification method based on audio and image according to claim 6 is characterized in that: The lightweight convolution structure layer includes branch A, branch B, branch C, branch D, a splicing sublayer and a residual structure sublayer; The branch A is used to perform a 1×1 convolution operation on the feature image; The branch B is used to perform 1×1 point-by-point convolution and 3×3 depth convolution on the feature image; The branch C is used to perform 1×1 point-by-point convolution and two 3×3 depth convolutions on the feature image; The branch D is used to perform a 3×3 maximum pooling operation on the feature image; The stitching sublayer is used to stitch the output images of the branch A, branch B, branch C and branch D to obtain a stitched image; The residual structure sublayer is used to add the feature image and the spliced image to obtain an output feature map.
9. A wind turbine blade damage identification device based on audio and image, characterized in that: include: A data acquisition module, used to acquire audio signals and image data of wind turbine blades; An audio extraction module, used to divide the frequency spectrum of the audio signal into multiple frequency ranges using an octave algorithm, and extract sound pressure level features of the multiple frequency ranges; An audio recognition module, used for inputting the sound pressure level feature into a trained back propagation neural network model BPNN to obtain an audio recognition result of the damage condition of the fan blade; An image recognition module, used to input the image data into a trained lightweight convolutional neural network model to obtain an image recognition result of the damage condition of the wind turbine blade; A comprehensive judgment module, used for comprehensively judging the audio recognition result and the image recognition result to obtain a comprehensive recognition result of the damage condition of the wind turbine blade; Among them, the back propagation neural network model is trained based on audio samples and audio labels, and the lightweight convolutional neural network model is established based on a lightweight convolutional structure and trained based on image samples and image labels.
10. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 8.
