Wind turbine generator blade fault detection and diagnosis method, system and device based on YOLO and large language model and storage medium

Abstract

The invention discloses a wind turbine generator blade fault detection and diagnosis method, system and equipment based on YOLO and a large language model and a storage medium, and belongs to the field of wind turbine equipment fault detection.The wind turbine generator blade fault detection and diagnosis method comprises the steps that wind turbine generator blade images are collected and labeled, and an image data set containing fault labels is formed; performing data preprocessing through an image enhancement strategy; inputting the data set into a target detection model, and outputting an image containing fault identification information and a corresponding feature vector; based on the obtained feature vectors, converting the structured feature information into natural language text description by adopting a text mapping technology; and jointly inputting a fault recognition image output based on the target detection model and a natural language text obtained by conversion through a text mapping technology into a large language model for diagnosis analysis and outputting a result. According to the method, only the unmanned aerial vehicle is needed to collect the image, the YOLO technology, the key value image text mapping technology and the large language model technology are combined, detection and diagnosis of the blade fault are achieved, complex extra equipment is not needed, and the detection cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of wind power equipment fault detection technology, specifically to a method, system, device, and storage medium for wind turbine blade fault detection and diagnosis based on YOLO and a large language model, namely YOLO (You Only Look Once real-time object detection machine learning algorithm). Background Technology

[0002] As the core equipment of wind power generation, the health of wind turbine blades directly affects the stable operation of the entire wind power system. With the rapid development of the global wind power industry, the installed capacity of wind turbines is constantly increasing, and the economic losses and safety hazards caused by blade failures are becoming increasingly prominent. Traditional blade inspection methods have many shortcomings, such as the low efficiency and difficulty in detecting internal damage through early manual visual inspection; sensor-based monitoring suffers from complex wiring and signal delays; vibration-based detection methods have low accuracy due to the limited precision of sensors; and fault detection based on SCADA systems is limited by data processing capabilities.

[0003] With the development of computer vision technology, UAV-based detection systems have been widely used. These systems can acquire high-resolution, multi-angle images, providing data support for target detection algorithms. In the field of target detection, single-stage detection methods have become the mainstream for wind turbine blade fault detection due to their high efficiency. YOLO, as a representative of single-stage detection, has shown certain advantages in blade detection. However, due to the complex operating environment of wind turbines, the scarcity of fault samples, and the high cost of collecting high-quality fault samples, the available training data is limited, affecting the detection accuracy and ability to identify complex faults of the YOLO model. Furthermore, existing YOLO models in blade fault detection also suffer from problems such as simple diagnostic results, limited contextual interpretation capabilities, and a lack of interactive analysis support. While Large Language Models (LLMs) perform well in natural language understanding and multimodal interaction, they lack expertise in wind turbine blade fault diagnosis, making direct application difficult to meet practical needs. Therefore, this invention proposes a new method based on YOLO and Large Language Models to address these problems and improve the accuracy of wind turbine blade fault detection and the comprehensiveness of diagnostic analysis. Summary of the Invention

[0004] To address the aforementioned technical challenges, a method for wind turbine blade fault detection and diagnosis based on YOLO and a large language model is proposed. This method includes acquiring images of wind turbine blades and labeling them to form an image dataset containing fault markers.

[0005] Based on the image dataset, data preprocessing is performed using image enhancement strategies;

[0006] The dataset obtained from data preprocessing is input into the target detection model, which outputs an image containing fault identification information and the corresponding feature vector.

[0007] Based on the obtained feature vectors, text mapping technology is used to convert the structured feature information in the fault identification results into natural language text descriptions.

[0008] The fault identification image output by the target detection model and the natural language text converted by the text mapping technology are jointly input into the large language model for diagnostic analysis and output results.

[0009] As a preferred embodiment of the wind turbine blade fault detection and diagnosis method based on YOLO and large language model described in this invention, the data preprocessing based on the image dataset and the image enhancement strategy includes cropping the original image and adjusting the scaling factor to obtain image information at different resolutions, and then performing sliding cropping according to a fixed-size window and a set step size to obtain several sub-images to expand the dataset.

[0010] As a preferred embodiment of the wind turbine blade fault detection and diagnosis method based on YOLO and a large language model described in this invention, the step of inputting the dataset obtained from data preprocessing into the target detection model and outputting an image containing fault identification information and corresponding feature vectors includes,

[0011] Image features are extracted using a convolutional feature extraction network, and the fault category and fault region bounding box coordinates are determined based on the classification and regression modules, respectively, generating an image containing fault identification information and the corresponding feature vector.

[0012] As a preferred embodiment of the wind turbine blade fault detection and diagnosis method based on YOLO and a large language model described in this invention, the step of converting the structured feature information in the fault identification result into a natural language text description using text mapping technology based on the obtained feature vector includes:

[0013] The structured detection output information is mapped into specific natural language words. Based on the frequency of occurrence of each fault and the area of ​​the damaged area, adjectives describing the number and size of the faults are added to the mapped text.

[0014] As a preferred embodiment of the wind turbine blade fault detection and diagnosis method based on YOLO and a large language model described in this invention, the fault recognition image output by the target detection model and the natural language text converted by text mapping technology are jointly input into the large language model for diagnostic analysis, and the output results include...

[0015] The text description and the image with the recognition box obtained by the object detection model are input together into the first language model. The multimodal fusion capability is used to perform fault detection and diagnosis, and the text information of fault type and severity is output.

[0016] As a preferred embodiment of the wind turbine blade fault detection and diagnosis method based on YOLO and a large language model described in this invention, the method further includes: the fault recognition image output by the target detection model and the natural language text converted by text mapping technology are jointly input into the large language model for diagnostic analysis and output results.

[0017] The maintenance reports of wind turbine units and related content from professional knowledge about wind turbine units were compiled into a database of the second largest language model;

[0018] The textual information on fault type and severity output from the first language model is input into the second language model for induction. Through reasoning, fault explanations and maintenance suggestions are generated, and a structured fault detection and diagnosis report is output.

[0019] As a preferred embodiment of the wind turbine blade fault detection and diagnosis method based on YOLO and large language model described in this invention, the text mapping technology adopts a lightweight key-value image text mapping method, with fault information category as key and corresponding category text as value. The frequency of occurrence of each fault type is calculated based on the number of fault types detected by the target detection model, and a description with quantifiers is generated. The area of ​​the damaged region within the recognition box is estimated and compared to generate a description with area information.

[0020] The first large language model includes a modal encoder, a connector, and a generator. Key components in the connector include a multilayer perceptron, a query transformer, and a multi-head attention mechanism.

[0021] Before performing fault analysis and generating maintenance recommendations, the second language model needs to be fine-tuned using professional knowledge from the wind turbine maintenance process, actual expert maintenance reports, and domain-specific question-and-answer datasets as supporting data.

[0022] Another objective of this invention is to provide a wind turbine blade fault detection and diagnosis system based on YOLO and a large language model. This invention addresses the problems of low detection accuracy, lack of fault description capabilities, and insufficient diagnostic suggestions in existing wind turbine blade fault detection technologies. It enhances model training performance through image data augmentation, accurately extracts blade fault feature information using a YOLO object detection model, converts structured features into natural language descriptions using text mapping technology, and generates fault diagnosis and maintenance suggestions based on the multimodal fusion capabilities of a large language model. This achieves accurate identification, clear description, and scientific decision support for blade faults, improving fault handling efficiency and the level of intelligent operation and maintenance.

[0023] As a preferred embodiment of the wind turbine blade fault detection and diagnosis system based on YOLO and large language model described in this invention, it is characterized by including: an image data processing and enhancement module, a target detection and feature extraction module, an image feature vector to natural language text mapping module, and a large language model multimodal diagnosis and decision module.

[0024] The image data processing and enhancement module acquires raw image data of wind turbine blades, generates an labeled dataset through annotation, and enhances the image data.

[0025] The target detection and feature extraction module is based on the target detection model. It obtains the fault feature vector in the image through a convolutional feature extraction network, uses a classification module to identify the fault category, and uses a regression module to determine the fault location and region bounding box.

[0026] The image feature vector to natural language text mapping module converts the structured fault information output by the target detection module into a natural language description. It uses lightweight key-value mapping to convert fault category, frequency, and area information into descriptive text.

[0027] The large language model multimodal diagnosis and decision module uses the first large language model to fuse image information and text description, and outputs the fault type and severity through multimodal capabilities; it uses the second large language model to output specific fault cause explanations and maintenance suggestions through reasoning analysis, and outputs a structured diagnostic report.

[0028] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method for wind turbine blade fault detection and diagnosis based on YOLO and a large language model.

[0029] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for wind turbine blade fault detection and diagnosis based on YOLO and a large language model.

[0030] The beneficial effects of this invention are as follows: This invention only requires images collected by a drone, and combines technologies such as YOLO, key-value image-text mapping, and large language models to achieve efficient detection and diagnosis of blade faults, without the need for complex additional equipment, thus reducing detection costs. The improved YOLO model expands the dataset through multi-scale and sliding window pruning techniques, effectively enhancing fault feature extraction capabilities and improving fault detection accuracy. Lightweight key-value image-text mapping converts YOLO detection results into a text format suitable for large model processing, avoiding information loss caused by overlapping image labels and providing high-quality data support for fault analysis. After fine-tuning with domain knowledge, the large model can deeply integrate visual and textual information, accurately analyze fault characteristics, and generate professional fault detection reports, providing wind farm operation and maintenance personnel with comprehensive and reliable decision-making basis. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 The above is a flowchart of a wind turbine blade fault detection and diagnosis method based on YOLO and a large language model, which is provided as an embodiment of the present invention. Detailed Implementation

[0033] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0034] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a method for wind turbine blade fault detection and diagnosis based on YOLO and large language models, including:

[0035] S1. Collect images of wind turbine blades and label them to form an image dataset containing fault markers.

[0036] High-definition images of wind turbine blades are captured using drones equipped with high-definition cameras, and then fault areas are marked by wind turbine blade maintenance engineers.

[0037] S2. Based on the image dataset, perform data preprocessing using image enhancement strategies.

[0038] In a preferred embodiment of the present invention, the image is processed by multi-scale cropping and sliding window cropping techniques to expand the dataset and improve the feature extraction capability of the model.

[0039] (1) Multi-scale cropping:

[0040] First, the original image is cropped at multiple scales, and the scaling factor is adjusted to obtain image information at different resolutions, thereby expanding the dataset. The scaled image size is:

[0041]

[0042] Where H and W are the height and width of the original image, H i and W i These are the height and width of the scaled image, s i It is the scaling factor.

[0043] (2) Sliding window cropping:

[0044] The scaled image is cropped using a fixed-size window with a certain step size to obtain multiple sub-images to expand the dataset. Starting from the top left corner of the image, the top left corner coordinates (x, y) of the window satisfy:

[0045]

[0046] In formula (2), s is the step size, and w and h are the width and height of the window, respectively. Maintain a certain overlap during cropping.

[0047] In this example, the selected sliding window has a width w and a height h of 512 pixels. The sliding window moves 128 pixels at a time in both the horizontal (x-direction) and vertical (y-direction) directions; that is, the step size s is set to 128 pixels.

[0048] After multi-scale cropping and sliding window processing, the number of images in the dataset increased from 510 to 3204. The total number of labels increased from 1480 to 7414, with the number of labels for crack fault types increasing from 124 to 522.

[0049] In an optional embodiment of the present invention, the image enhancement strategy may be based on a complex environment enhancement strategy involving affine transformation and noise perturbation.

[0050] To address the challenges posed by intense light variations, complex background interference, and image noise in the actual operating environment of wind farms, an image enhancement strategy based on affine transformation and noise perturbation is proposed to improve the robustness of the model in fault identification under complex scenarios. The original image undergoes random rotation (rotation angle range -30° to +30°), translation (random horizontal and vertical offsets not exceeding 10% of the image width and height), scaling (scale range 0.8-1.2 times), and shearing transformations.

[0051] By simulating blade images from different shooting angles and tilted perspectives, the model's adaptability to complex blade installation angles and varying shooting perspectives is improved. Image brightness (range ±20%), contrast (range ±30%), and saturation (range ±15%) are randomly adjusted to simulate image acquisition scenarios under complex meteorological conditions such as cloudy days, strong sunlight, and high reflectivity. Gaussian noise and salt-and-pepper noise are superimposed on the images to simulate noise interference from the image sensor and distortion during image transmission, improving the model's robustness to low-quality images. The coordinates and size of fault label boxes are simultaneously adjusted in the enhanced images to ensure consistency between the enhanced image data and the original label data. This approach is suitable for model optimization and generalization improvement stages, enhancing the model's recognition capabilities in complex environments and improving its reliability in field applications.

[0052] However, introducing affine transformations and noise perturbations may distort image features, affecting the model's effective extraction of real fault features; it requires higher computational resources and longer model training time; if the enhancement method's parameters are not set reasonably, it may introduce too many invalid features, leading to a decrease in the model's recognition accuracy.

[0053] It should be further noted that the present invention fully covers fault feature areas of different sizes through multi-scale cutting, and can maintain high recognition accuracy in the detection of fine cracks and large-area scratches.

[0054] While the control group based on complex environment enhancement strategies using affine transformation and noise perturbation can improve environmental adaptability through affine transformation, it cannot perform refined enhancement for small target fault characteristics and easily overlooks subtle defect features.

[0055] This invention effectively preserves the complete feature information of local fault areas by performing multi-scale cropping followed by sliding window cropping in a layered processing method, thus avoiding feature distortion caused by image distortion or noise superposition.

[0056] The control group based on the complex environment enhancement strategy of affine transformation and noise perturbation is prone to introducing unrealistic image changes and noise interference during the affine transformation and noise perturbation process, which leads to the model learning invalid or even wrong features.

[0057] This invention employs a layered, progressive pruning and enhancement strategy to improve the ability to identify small target faults while ensuring the integrity and effectiveness of fault features. This effectively solves the problem of low model recognition accuracy caused by insufficient samples and feature ambiguity in existing technologies. Compared to the control group, this invention not only improves the model's recognition accuracy in wind turbine blade fault detection but also reduces the model's dependence on complex environmental factors.

[0058] S3. Input the dataset obtained from data preprocessing into the target detection model, and output the image containing fault identification information and the corresponding feature vector.

[0059] In a preferred embodiment of the present invention, the expanded dataset after S2 processing is input into the YOLOv12 detection model to obtain an image with a recognition box and the corresponding feature vector.

[0060] The augmented dataset after S2 processing was divided into training and test sets in a 9:1 ratio. The YOLOv12 algorithm detection model was trained using the training set, with stochastic gradient descent (SGD) as the optimizer. The number of samples per batch was 32, and the training epochs were 500, ultimately yielding images with bounding boxes and corresponding feature vectors.

[0061] In a preferred embodiment of the present invention, the target detection model may use YOLOv5, YOLOv8, YOLOv10, or YOLOv11.

[0062] Meanwhile, performance comparisons were conducted among YOLO models in the same series. Evaluation parameters included mAP50 (mean precision at an IoU threshold of 0.5, measuring overall model detection accuracy), recall (reflecting the model's ability to detect positive samples), precision (reflecting the model's accuracy in predicting positive samples), F1 (the harmonic mean of precision and recall, combining both performance metrics), and mean precision for four categories (cracks, epidermal degumming, fiber failure, and surface depressions) (evaluating the model's detection performance for the corresponding categories). Performance evaluations were conducted on YOLOv5, YOLOv8, YOLOv10, YOLOv11, YOLOv12, and models based on YOLOv12 after image preprocessing, from different dimensions. Selected evaluation results are shown in Table 1.

[0063] Table 1. Data Comparison Results

[0064]

[0065] Experimental results show that the image preprocessing model based on YOLOv12 outperforms other models in terms of mAP50, recall, and other parameters. Specifically, the image preprocessing model based on YOLOv12 achieves a high mAP50 of 0.96, a recall of 0.87, a precision of 0.95, and an F1 score of 0.91. Overall, the proposed image preprocessing model based on YOLOv12 demonstrates the best comprehensive performance.

[0066] In an optional embodiment of the present invention, the object detection model may be the Faster R-CNN object detection model.

[0067] The image-enhanced blade dataset was input into the Faster R-CNN object detection model; ResNet-50 was used as the backbone feature extraction network to maintain moderate model complexity; the dataset was divided into training and test sets in a 9:1 ratio; the SGD optimizer was used with a learning rate of 0.001, a batch size of 16, and 300 training epochs; the model was comprehensively evaluated using metrics such as mAP50, recall, precision, and F1 score. This model is mainly applicable to scenarios with high accuracy requirements but low real-time requirements, such as offline large-scale historical image analysis or batch fault data processing on centralized servers in wind farms.

[0068] In this invention, it can be used in large model analysis modules to perform detailed analysis and result verification of complex faults, but it is not suitable as a real-time online detection model.

[0069] Table 2. Data Comparison Results

[0070]

[0071]

[0072] The preferred image preprocessing model based on YOLOv12 exhibits optimal overall performance in small target fault detection and adaptability to complex environments, making it widely applicable to real-time fault monitoring of wind turbine blades and edge device deployment. While the Faster R-CNN model offers some accuracy advantages, its complex two-stage detection architecture results in slow inference speeds, making it unsuitable for real-time monitoring. Furthermore, it lacks sufficient detection capability for small target faults such as microcracks, leading to a high false negative rate. In summary, Faster R-CNN is more suitable for offline batch fault data analysis and is ill-suited for online fault detection tasks in actual wind turbines. Therefore, the preferred solution of this invention has greater engineering value and practicality in real-world applications.

[0073] S4. Based on the obtained feature vectors, text mapping technology is used to convert the structured feature information in the fault identification results into natural language text descriptions.

[0074] A lightweight key-value image-to-text mapping method is employed to convert the feature vectors obtained in S3 into concise text descriptions. Key-value mapping maps structured detection output information (keys), such as blade fault categories and bounding box information, into specific natural language words / phrases (values). Furthermore, this method comprehensively considers the frequency of fault types and the area of ​​damaged regions. Based on the frequency f of each fault and the area of ​​the damaged region, adjectives describing the number and size of faults are added to the key-value mapped text to obtain a clearer text description. The frequency f is calculated as follows:

[0075]

[0076] In formula (3), f is the frequency of occurrence of a certain fault, and G i G represents the number of faults detected for a specific fault, and G represents the total number of faults.

[0077] A lightweight key-value image-to-text mapping method is employed to convert the feature vectors obtained in S3 into concise text descriptions. Furthermore, this method comprehensively considers the frequency of fault types and the area of ​​damaged regions. Based on the frequency f of each fault and the area of ​​the damaged region, adjectives describing the number and size of faults are added to the key-value mapped text to obtain a clearer text description.

[0078] In formula (3), f is the frequency of occurrence of a certain fault, and G i Let f be the number of faults detected, and G be the total number of faults. During key-value conversion, set 0.2, 0.4, 0.5, and 0.8 as thresholds. When f is greater than 0.2, insert the quantifier "very few" into the text; when f is greater than 0.4, insert the quantifier "a small amount"; when f is greater than 0.5, insert the quantifier "more than half"; and when f is greater than 0.8, insert the quantifier "almost all".

[0079] S5. The fault identification image output by the target detection model and the natural language text converted by the text mapping technology are jointly input into the large language model for diagnostic analysis and output results.

[0080] The first language large model is the Qwen2-VL large model, and the second language large model is the Llama large model.

[0081] The text description obtained in S4 and the image with bounding boxes obtained in S3 after being recognized by the YOLO algorithm are input into the Qwen2-VL large model. Its multimodal fusion capability is used for further fault detection and diagnosis, and the output result is detailed text information on the fault type and severity.

[0082] The text description obtained in S4 and the image with recognition boxes obtained in S3 are input into the Qwen2-VL module. This multimodal large model module, with its excellent multimodal fusion capabilities, can deeply integrate and analyze data from two different modalities: text and images. In this process, it can not only understand key information in the text description but also combine visual cues from the image to conduct a comprehensive and detailed assessment of the equipment's fault condition. Utilizing its multimodal fusion capabilities, it assists in fault detection and diagnosis, outputting text information covering the fault type and severity.

[0083] The maintenance reports of wind turbine units and relevant content from professional knowledge about wind turbine units are compiled into a database to serve as a knowledge base for fine-tuning the Llama large model. By fine-tuning the Llama large model, it can better detect and diagnose wind turbine blades.

[0084] A knowledge base was compiled from wind turbine maintenance reports and relevant professional knowledge about wind turbines, serving as a fine-tuning platform for the large model. The knowledge base is structured in a question-and-answer pair format, with 61 pairs clearly presenting wind turbine fault-related questions and their detailed solutions, largely aligning with the dialogue-based questioning style used in subsequent human-computer interaction. Furthermore, fine-tuning allows the large model to output more professional and comprehensive text content related to wind turbines.

[0085] The output of the Qwen2-VL large model is input into the finely tuned Llama large model, enabling it to perform inductive reasoning to generate professional fault explanations and maintenance suggestions, and integrate and output structured fault detection and diagnostic reports.

[0086] Example 2 is the second embodiment of the present invention. This embodiment provides a wind turbine blade fault detection and diagnosis system based on YOLO and a large language model, including an image data processing and enhancement module, a target detection and feature extraction module, an image feature vector to natural language text mapping module, and a large language model multimodal diagnosis and decision module.

[0087] The image data processing and enhancement module acquires raw image data of wind turbine blades, generates an labeled dataset through annotation, and then enhances the image data.

[0088] The object detection and feature extraction module is based on the object detection model. It obtains the fault feature vector in the image through a convolutional feature extraction network, uses the classification module to identify the fault category, and uses the regression module to determine the fault location and region bounding box.

[0089] The image feature vector to natural language text mapping module converts the structured fault information output by the target detection module into a natural language description. It uses lightweight key-value mapping to convert fault category, frequency, and area information into descriptive text.

[0090] The large language model multimodal diagnosis and decision module uses the first large language model to fuse image information and text descriptions, and outputs the fault type and severity through multimodal capabilities; it uses the second large language model to output specific fault cause explanations and maintenance suggestions through reasoning analysis, and outputs a structured diagnostic report.

[0091] Example 3 is the third embodiment of the present invention, which differs from the previous two embodiments in that:

[0092] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0093] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0094] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0095] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0096] Example 4, the fourth embodiment of the present invention, provides a method for wind turbine blade fault detection and diagnosis based on YOLO and large language models. To verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0097] High-definition images of wind turbine blades are captured by drones equipped with high-definition cameras, and the fault areas are marked by wind turbine blade maintenance engineers.

[0098] High-resolution images of wind turbine blades were captured using a drone equipped with a high-definition camera, totaling 510 images. Next, domain experts labeled the fault areas, including fault type classification and fault location, resulting in 1480 valid labels.

[0099] Image processing, employing multi-scale cropping and sliding window cropping techniques, expands the dataset and enhances the model's feature extraction capabilities. Further expansion involves using a fixed-size window with a defined step size to obtain multiple sub-images. In this example, the sliding window's width w and height h are both 512 pixels. The window moves 128 pixels horizontally (x-direction) and vertically (y-direction) at each step, i.e., the step size s is set to 128 pixels.

[0100] After multi-scale cropping and sliding window processing, the number of images in the dataset was expanded from 510 to 3204. The total number of labels increased from 1480 to 7414, with the number of labels for crack fault types increasing from 124 to 522. The expanded dataset processed by S2 was then input into the YOLOv12 detection model to obtain images with bounding boxes and corresponding feature vectors.

[0101] The augmented dataset after S2 processing was divided into training and test sets in a 9:1 ratio. The YOLOv12 algorithm detection model was trained using the training set, with stochastic gradient descent (SGD) as the optimizer. The number of samples per batch was 32, and the training epochs were 500. The final result was an image with bounding boxes and the corresponding feature vectors.

[0102] Meanwhile, performance comparisons were conducted among YOLO models in the same series. Evaluation parameters included mAP50 (mean precision at an IoU threshold of 0.5, measuring overall model detection accuracy), recall (reflecting the model's ability to detect positive samples), precision (reflecting the model's accuracy in predicting positive samples), F1 (the harmonic mean of precision and recall, combining both performance metrics), and mean precision for four categories (cracks, epidermal degumming, fiber failure, and surface depressions) (evaluating the model's detection performance for the corresponding categories). Performance evaluations were conducted on YOLOv5, YOLOv8, YOLOv10, YOLOv11, YOLOv12, and models based on YOLOv12 after image preprocessing, from different dimensions. Selected evaluation results are shown in Table 1.

[0103] Table 1. Data Comparison Results

[0104]

[0105] Experimental results show that the image preprocessing model based on YOLOv12 outperforms other models in terms of mAP50, recall, and other parameters. Specifically, the image preprocessing model based on YOLOv12 achieves a high mAP50 of 0.96, a recall of 0.87, a precision of 0.95, and an F1 score of 0.91. Overall, the proposed image preprocessing model based on YOLOv12 demonstrates the best comprehensive performance.

[0106] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for fault detection and diagnosis of wind turbine blades based on YOLO and large language models, characterized in that: include, Collect images of wind turbine blades and label them to create an image dataset containing fault markers; Based on the image dataset, data preprocessing is performed using image enhancement strategies; The dataset obtained from data preprocessing is input into the target detection model, which outputs an image containing fault identification information and the corresponding feature vector. Based on the obtained feature vectors, text mapping technology is used to convert the structured feature information in the fault identification results into natural language text descriptions. The fault identification image output by the target detection model and the natural language text converted by the text mapping technology are jointly input into the large language model for diagnostic analysis and output results.

2. The method for wind turbine blade fault detection and diagnosis based on YOLO and large language models as described in claim 1, characterized in that: The image dataset-based data preprocessing using image enhancement strategies includes cropping the original image and adjusting the scaling factor to obtain image information at different resolutions, then performing sliding cropping according to a fixed-size window and a set step size to obtain several sub-images to augment the dataset.

3. The method for wind turbine blade fault detection and diagnosis based on YOLO and large language models as described in claim 2, characterized in that: The step of inputting the dataset obtained from data preprocessing into the target detection model and outputting an image containing fault identification information and corresponding feature vectors includes, Image features are extracted using a convolutional feature extraction network, and the fault category and fault region bounding box coordinates are determined based on the classification and regression modules, respectively, generating an image containing fault identification information and the corresponding feature vector.

4. The method for wind turbine blade fault detection and diagnosis based on YOLO and large language models as described in claim 3, characterized in that: The process of converting the structured feature information in the fault identification results into natural language text descriptions using text mapping techniques based on the obtained feature vectors includes: The structured detection output information is mapped into specific natural language words. Based on the frequency of occurrence of each fault and the area of ​​the damaged area, adjectives describing the number and size of the faults are added to the mapped text.

5. The method for wind turbine blade fault detection and diagnosis based on YOLO and large language models as described in claim 4, characterized in that: The fault identification image output by the target detection model and the natural language text converted by the text mapping technology are jointly input into the large language model for diagnostic analysis. The output results include: The text description and the image with the recognition box obtained by the object detection model are input together into the first language model. The multimodal fusion capability is used to perform fault detection and diagnosis, and the text information of fault type and severity is output.

6. The method for wind turbine blade fault detection and diagnosis based on YOLO and large language models as described in claim 5, characterized in that: The natural language text obtained by mapping the fault identification image output by the target detection model with the text is jointly input into the large language model for diagnostic analysis. The output results also include... The maintenance reports of wind turbine units and related content from professional knowledge about wind turbine units were compiled into a database of the second largest language model; The textual information on fault type and severity output from the first language model is input into the second language model for induction. Through reasoning, fault explanations and maintenance suggestions are generated, and a structured fault detection and diagnosis report is output.

7. The method for wind turbine blade fault detection and diagnosis based on YOLO and large language models as described in claim 6, characterized in that: The text mapping technology employs a lightweight key-value image text mapping method, where fault information categories are used as keys and corresponding category texts are used as values. The frequency of occurrence of each fault type is calculated based on the number of fault types detected by the target detection model, generating a description with quantifiers. The area of ​​the damaged region within the recognition box is estimated and compared to generate a description with area information. The first large language model includes a modal encoder, a connector, and a generator. Key components in the connector include a multilayer perceptron, a query transformer, and a multi-head attention mechanism. Before performing fault analysis and generating maintenance recommendations, the second language model needs to be fine-tuned using professional knowledge from the wind turbine maintenance process, actual expert maintenance reports, and domain-specific question-and-answer datasets as supporting data.

8. A wind turbine blade fault detection and diagnosis system based on YOLO and a large language model, employing the wind turbine blade fault detection and diagnosis method based on YOLO and a large language model as described in any one of claims 1 to 7, characterized in that, It includes: an image data processing and enhancement module, an object detection and feature extraction module, an image feature vector to natural language text mapping module, and a large language model multimodal diagnosis and decision-making module; The image data processing and enhancement module acquires raw image data of wind turbine blades, generates an labeled dataset through annotation, and enhances the image data. The target detection and feature extraction module is based on the target detection model. It obtains the fault feature vector in the image through a convolutional feature extraction network, uses a classification module to identify the fault category, and uses a regression module to determine the fault location and region bounding box. The image feature vector to natural language text mapping module converts the structured fault information output by the target detection module into a natural language description. It uses lightweight key-value mapping to convert fault category, frequency, and area information into descriptive text. The large language model multimodal diagnosis and decision module uses the first large language model to fuse image information and text description, and outputs the fault type and severity through multimodal capabilities; it uses the second large language model to output specific fault cause explanations and maintenance suggestions through reasoning analysis, and outputs a structured diagnostic report.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for wind turbine blade fault detection and diagnosis based on YOLO and large language model as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for wind turbine blade fault detection and diagnosis based on YOLO and large language model as described in any one of claims 1 to 7.

Images