Fan blade defect detection method based on large segmentation model and related equipment

Abstract

The embodiment of the invention provides a fan blade defect detection method based on a large segmentation model and related equipment, and belongs to the technical field of defect detection. The method comprises the steps of inputting multiple groups of acquired detection images and reference images into an initial segmentation large model, performing feature extraction processing on the detection images to obtain detection image coding features, generating defect description texts of the detection images, and performing semantic feature extraction processing on the defect description texts to obtain text prompt coding features; mask coding features are obtained based on mask prompt information determined by the defect mask features and the reference image features; performing multi-modal feature joint decoding processing on the detection image coding features, the text prompt coding features and the mask coding features to obtain target detection information, and obtaining a trained target segmentation large model based on the target detection information; and inputting the target detection image into the target segmentation large model to obtain a defect detection result. According to the invention, the precision of fan blade defect detection can be improved.

Description

Technical Field

[0001] This application relates to the field of defect detection technology, and in particular to a method and related equipment for detecting defects in wind turbine blades based on a segmented large model. Background Technology

[0002] Wind turbine blade defect detection refers to the process of periodically or in real-time inspecting the blades of wind turbine generators to identify various defects or damage that may affect their performance and safety. These defects may include cracks, erosion, delamination, lightning strike damage, manufacturing flaws, and wear caused by long-term exposure to harsh environmental conditions. Related technologies utilize deep learning detection models to locate blade defects and determine their category.

[0003] However, the defect detection methods used in related technologies are relatively crude. With the rapid development of onshore and offshore wind turbines, the types of wind turbine blade defects are becoming increasingly complex. Conventional detection models are prone to feature loss during feature processing due to limited feature extraction capabilities and insufficient generalization, which in turn leads to low accuracy in wind turbine blade defect detection. Summary of the Invention

[0004] The main objective of this application is to propose a method and related equipment for detecting wind turbine blade defects based on a segmented large model, aiming to improve the accuracy of wind turbine blade defect detection.

[0005] To achieve the above objectives, a first aspect of this application proposes a method for detecting defects in wind turbine blades based on a segmented large model, the method comprising:

[0006] Multiple sets of detection images and reference images are acquired and input into the initial segmentation large model. The initial segmentation large model includes a large model, a first image encoder, a second image encoder, a mask cue attention module, a text cue encoder, a mask cue encoder, and a classification mask decoder. Each reference image includes defect reference annotations, and different sets of defect reference annotations belong to different defect categories.

[0007] For the same set of detection images and reference images, the first image encoder is used to perform image feature extraction processing on the detection images to obtain the detection image encoding features. The large model is used to generate the defect description text of the detection images, and the text prompt encoder is used to perform semantic feature extraction processing on the defect description text to obtain the text prompt encoding features.

[0008] Guided by defect reference annotations, the second image encoder determines the defect mask features and reference image features of the reference image. The mask cueing attention module determines mask cueing information based on the defect mask features and reference image features. The mask cueing encoder encodes the mask cueing information to obtain mask coding features.

[0009] The target detection information is obtained by multimodal joint decoding of the detection image coding features, text prompt coding features and mask coding features using a classification mask decoder. Based on the target detection information, the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder are trained to obtain a large target segmentation model.

[0010] The target detection image is acquired and input into the target segmentation large model to obtain the defect detection result.

[0011] In some embodiments, target detection information is obtained by performing multimodal feature joint decoding processing on the detection image coding features, text prompt coding features, and mask coding features using a classification mask decoder, including:

[0012] Based on the self-attention mechanism, the classification mask decoder performs feature enhancement processing on the text prompt encoding features to obtain enhanced text prompt encoding features;

[0013] Based on the cross-attention mechanism, feature fusion processing is performed on the enhanced text prompt coding features and the detection image coding features to obtain the first fused coding features, and the fused coding features are then subjected to nonlinear transformation processing to obtain nonlinear coding features;

[0014] The mask coding features are mapped to obtain the mapped mask coding features. Based on the cross attention mechanism, the mapped mask coding features and the nonlinear coding features are fused to obtain the second fused coding features.

[0015] The target detection information is obtained by concatenating the enhanced text prompt coding features, the mapped mask coding features, and the second fused coding features.

[0016] In some embodiments, generating defect description text for the detected image using a large model includes:

[0017] The large model is used to perform image feature analysis on the detected image to obtain initial image features, and the defect area of ​​the corresponding wind turbine blade is determined based on the initial image features.

[0018] If the defect area is greater than the preset area threshold, the large model obtains prior knowledge from the pre-built prior knowledge base. The prior knowledge includes at least the corresponding defect morphology information and defect location information of the wind turbine blade under various defect categories.

[0019] Based on the initial image features, the current defect category of the corresponding wind turbine blade is determined. From the multiple defect categories indicated by prior knowledge, at least one that matches the current defect category is selected as the target defect category. Based on the defect morphology information and defect location information under the target defect category, a defect description text of the detected image is generated.

[0020] If the defect area is not greater than a preset area threshold, feature enhancement processing is performed on the detection image to obtain an enhanced detection image, and defect description text is generated for the enhanced detection image.

[0021] In some embodiments, feature enhancement processing is performed on the detection image to obtain an enhanced detection image, including:

[0022] The detected image is divided into multiple initial image blocks, and histogram equalization is performed on each initial image block to obtain equalized image blocks.

[0023] Using a preset bilinear interpolation function, grayscale value mapping is performed on the equalized image blocks to obtain grayscale value image blocks. An initial enhanced detection image is obtained based on all grayscale value image blocks.

[0024] Using a preset wavelet basis function, the initial enhanced detection image is decomposed into multiple sub-bands using wavelet decomposition. Based on a set gain value, the sub-bands are then subjected to feature enhancement processing to obtain the enhanced detection image.

[0025] In some embodiments, guided by defect reference annotations, a second image encoder is used to determine defect mask features and reference image features of the reference image, including:

[0026] The second image encoder performs feature extraction on the reference image to obtain initial reference image features, and then performs convolution and pooling on the initial reference image features to obtain the reference image features.

[0027] The second image encoder, guided by defect reference annotations, performs average pooling on the features of the initial reference image to obtain defect mask features.

[0028] In some embodiments, the mask cueing attention module determines mask cueing information based on defect mask features and reference image features, including:

[0029] The features of the initial reference image are subjected to linear transformation to obtain sub-reference image features at different scales;

[0030] The reference image features are used as query information, and the sub-reference image features at different scales are used as key information, and the defect mask features are used as value information.

[0031] Based on query information, key-value information, and value information, determine the feature dimension information;

[0032] The mask cueing attention module determines mask cueing information based on query information, key-value information, value information, and feature dimension information.

[0033] In some embodiments, the initial segmentation large model further includes a temporal feature encoder;

[0034] After the mask cue encoder encodes the mask cue information to obtain the mask-coded features, the process also includes:

[0035] Obtain the historical detection images corresponding to the detected images at historical moments;

[0036] The temporal feature encoder performs feature encoding processing on the detected image and historical detected images to obtain temporal encoded features;

[0037] Enhanced detection information is obtained by using a classification mask decoder to perform multimodal feature joint decoding of detection image coding features, text prompt coding features, mask coding features, and temporal coding features;

[0038] Based on the enhanced detection information, the module parameters of the mask cue attention module and the decoder parameters of the classification mask decoder are adjusted respectively, and the trained target segmentation large model is obtained after the parameter adjustment is completed.

[0039] To achieve the above objectives, a second aspect of this application proposes a wind turbine blade defect detection device based on a segmented large model, the device comprising:

[0040] The acquisition module is used to acquire multiple sets of detection images and reference images, and input the multiple sets of detection images and reference images into the initial segmentation large model. The initial segmentation large model includes a large model, a first image encoder, a second image encoder, a mask cue attention module, a text cue encoder, a mask cue encoder, and a classification mask decoder. In addition, each reference image includes defect reference annotations, and different sets of defect reference annotations belong to different defect categories.

[0041] The first encoding module is used to extract image features from the detection images using a first image encoder to obtain detection image encoding features for the same set of detection images and reference images, generate defect description text for the detection images using a large model, and extract semantic features from the defect description text using a text prompt encoder to obtain text prompt encoding features.

[0042] The second encoding module is used to determine the defect mask features and reference image features of the reference image by using the second image encoder, guided by the defect reference annotation. The mask cue attention module determines the mask cue information based on the defect mask features and reference image features. The mask cue encoder encodes the mask cue information to obtain the mask encoded features.

[0043] The training module is used to perform multimodal feature joint decoding processing on the detection image coding features, text prompt coding features and mask coding features using the classification mask decoder to obtain target detection information. Based on the target detection information, the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder are trained to obtain the trained target segmentation large model.

[0044] The target processing module is used to acquire target detection images and input them into the target segmentation large model to obtain defect detection results.

[0045] To achieve the above objectives, a third aspect of the present application proposes an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the wind turbine blade defect detection method based on a segmented large model described in the first aspect.

[0046] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the wind turbine blade defect detection method based on a segmented large model described in the first aspect.

[0047] This application proposes a wind turbine blade defect detection method and related equipment based on a large segmentation model. The method acquires multiple sets of detection images and reference images, inputting them into an initial large segmentation model. This initial large segmentation model includes a large model, a first image encoder, a second image encoder, a mask cue attention module, a text cue encoder, a mask cue encoder, and a classification mask decoder. Each reference image includes defect reference annotations, with different sets of reference annotations belonging to different defect categories. For the same set of detection images and reference images, the first image encoder performs image feature extraction on the detection images to obtain encoded features. The large model generates defect description text for the detection images, and the text cue encoder extracts semantic features from the defect description text. The text prompt encoding features are obtained through processing. Guided by defect reference annotations, a second image encoder is used to determine the defect mask features and reference image features of the reference image. The mask prompt attention module determines the mask prompt information based on the defect mask features and reference image features. The mask prompt encoder encodes the mask prompt information to obtain mask encoding features. The classification mask decoder performs multimodal feature joint decoding processing on the detection image encoding features, text prompt encoding features, and mask encoding features to obtain target detection information. Based on the target detection information, the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder are trained to obtain a trained target segmentation large model. The target detection image is acquired and input into the target segmentation large model to obtain the defect detection result. Thus, this embodiment utilizes a large model to generate defect description text as text prompts, injecting high-order semantic prior knowledge into the detection model and effectively compensating for the shortcomings of single visual features being easily lost or confused in complex scenes. Secondly, the detection image encoding features of the detection image are generated through a first image encoder. Next, labeled reference images and a specially designed mask prompt attention module are introduced to provide the model with intuitive visual examples of the defects to be tested, greatly enhancing the model's ability to identify and generalize defects with complex shapes and diverse types. Finally, by jointly decoding multimodal features and efficiently training only the key prompt module and decoder parameters, the model can accurately locate and determine the defect category based on the fusion of visual, semantic, and mask complementary information, thereby significantly improving the overall accuracy and robustness of wind turbine blade defect detection without sacrificing deployment efficiency. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of an optional implementation environment for the wind turbine blade defect detection device based on a segmented large model provided in this application embodiment;

[0049] Figure 2This is an optional flowchart of the wind turbine blade defect detection method based on a segmented large model provided in the embodiments of this application;

[0050] Figure 3 This is a schematic diagram of the structure and data processing of an optional initial segmented large model for the wind turbine blade defect detection method based on a segmented large model provided in this application embodiment;

[0051] Figure 4 yes Figure 2 Step 102 in the flowchart is an optional implementation.

[0052] Figure 5 yes Figure 4 Step 102.1.4 is an optional implementation flowchart;

[0053] Figure 6 yes Figure 2 Step 103 is an optional implementation flowchart;

[0054] Figure 7 yes Figure 2 Another optional implementation flowchart for step 103 in the diagram;

[0055] Figure 8 yes Figure 2 Step 103 in the flowchart is another optional implementation.

[0056] Figure 9 yes Figure 2 Step 104 in the flowchart is an optional implementation.

[0057] Figure 10 This is a schematic diagram of the internal data processing of an optional classification mask decoder in the wind turbine blade defect detection method based on a segmentation large model provided in the embodiments of this application;

[0058] Figure 11 This is a schematic diagram of an optional device module for a wind turbine blade defect detection device based on a segmented large model provided in this application embodiment;

[0059] Figure 12 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit the scope of this application.

[0063] Wind power, as a clean and renewable energy technology, is closely related to global energy transition and environmental protection. With the energy crisis and rising environmental awareness, wind power has gradually gained attention. Wind turbines use wind energy to drive blades to rotate, converting this energy into electricity through a generator. This technology can significantly reduce dependence on traditional fossil fuels and effectively reduce greenhouse gas emissions. As the core component for capturing wind energy, the blades' performance directly determines power generation efficiency and turbine safety. However, during blade operation, defects such as corrosion, cracks, and aging can occur due to improper production and maintenance, as well as the influence of the natural environment. If these defects are not detected and repaired in time, they can easily lead to blade failure or even turbine accidents. Therefore, blade defect detection technology with real-time monitoring capabilities is crucial.

[0064] Wind turbine blade defect detection refers to the process of periodically or in real-time inspecting the blades of wind turbine generators to identify various defects or damage that may affect their performance and safety. These defects may include cracks, erosion, delamination, lightning strike damage, manufacturing flaws, and wear caused by long-term exposure to harsh environmental conditions. Related technologies utilize deep learning detection models to locate blade defects and determine their category.

[0065] Traditional methods for wind turbine blade defect detection based on large segmentation models include ultrasonic testing, infrared thermography, and deep learning. While ultrasonic testing can accurately locate internal defects, different types of defects require different probes. Infrared thermography is suitable for long-distance, large-area detection, but it has high requirements for on-site conditions. Deep learning technology uses convolutional neural networks, Transformers, and other deep learning algorithms to extract and fuse features from blade images to identify and diagnose blade defects. However, with the rapid development of onshore and offshore wind turbines, wind turbine blade images are becoming increasingly complex, and the types of wind turbine blade defects are also becoming more complex. Traditional wind turbine blade defect detection methods based on large segmentation models are relatively crude, exhibiting limited feature extraction capabilities and insufficient generalization, thus easily causing feature loss during feature processing, leading to low accuracy in wind turbine blade defect detection.

[0066] Based on this, this application provides a method and related equipment for wind turbine blade defect detection based on a segmentation large model. Specifically, this application innovatively constructs a multimodal prompting and guidance framework. First, the large model generates defect description text as text prompts, injecting high-order semantic prior knowledge into the detection model, effectively compensating for the shortcomings of single visual features being easily lost or confused in complex scenes. Second, the detection image encoding features of the detection image are generated through a first image encoder. Next, labeled reference images and a specially designed mask prompting attention module are introduced to provide the model with intuitive visual examples of the defects to be tested, greatly enhancing the model's ability to identify and generalize defects with complex shapes and diverse types. Finally, by jointly decoding the multimodal features and efficiently training only the key prompting module and decoder, the model can accurately locate and determine the defect category based on the integration of visual, semantic, and example complementary information, thereby significantly improving the overall accuracy and robustness of wind turbine blade defect detection without sacrificing deployment efficiency.

[0067] For example, such as Figure 1 As shown, Figure 1This is a schematic diagram of an optional implementation environment for the wind turbine blade defect detection device based on a segmentation large model provided in this application embodiment. The implementation environment includes a client 11 and a server 12, wherein the wind turbine blade defect detection device based on the segmentation large model (which can also be simply referred to as the "detection device" for ease of description) is deployed on the server 12. The server 12 receives multiple sets of detection images and reference images sent from at least one client 11. Then, the multiple sets of detection images and reference images are input into an initial segmentation large model, which includes a large model, a first image encoder, a second image encoder, a mask cue attention module, a text cue encoder, a mask cue encoder, and a classification mask decoder. Each reference image includes defect reference annotations, and different sets of defect reference annotations belong to different defect categories. Next, for the same set of detection images and reference images, the first image encoder performs image feature extraction processing on the detection images to obtain detection image encoding features, the large model generates defect description text for the detection images, and the text cue encoder performs semantic feature extraction processing on the defect description text to obtain text. The process involves several steps: First, guided by defect reference annotations, a second image encoder determines the defect mask features and reference image features. A mask cueing attention module then determines mask cueing information based on these features. The mask cueing encoder encodes this information to obtain mask-encoded features. Next, a classification mask decoder performs multimodal feature joint decoding on the detection image encoding features, text cueing features, and mask-encoded features to obtain target detection information. Based on this target detection information, the module parameters of the mask cueing attention module and the decoder parameters of the classification mask decoder are trained to obtain a large-scale target segmentation model. Finally, the target detection image is acquired and input into the large-scale target segmentation model to obtain the defect detection result.

[0068] The server 12 can send the trained target segmentation model to the client 11, and the client 11 can perform subsequent processing on the acquired target detection image; or the server 12 can directly use the trained target segmentation model to perform subsequent data processing. This application embodiment does not limit this.

[0069] The server 12 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Additionally, the server 12 can also be a node server in a blockchain network. The client 11 can be a mobile phone, computer, smart voice interaction device, smart wearable device, smart home appliance, in-vehicle terminal, etc., but is not limited to these. The client 11 and server 12 can be connected directly or indirectly through wired or wireless communication, and this embodiment of the application does not impose any limitations.

[0070] It should be noted that in this application embodiment, when information related to user characteristics, such as basic user information or user identity, is required, the user's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when this application embodiment needs to obtain sensitive personal information of a user, the user's individual permission or consent will be obtained first. Only after obtaining the user's individual permission or consent will the necessary data for the normal operation of this application embodiment be obtained. For example, before obtaining the detection image and reference image, the authorization or consent of relevant personnel will be obtained first. Furthermore, other relevant data obtained by the detection device of this application are all authorized data, and will not be elaborated upon here.

[0071] In this application embodiment, the description will focus on the dimension of a wind turbine blade defect detection device based on a segmented large model, such as... Figure 2 As shown, Figure 2 This is an optional flowchart of the wind turbine blade defect detection method based on a segmented large model provided in the embodiments of this application. Figure 2 The method may include, but is not limited to, the following steps 101 to 105. When the wind turbine blade defect detection device based on the segmented large model executes the wind turbine blade defect detection method based on the segmented large model, the specific process is as follows. It should be noted first that this embodiment... Figure 2 The order of steps 101 to 105 is not specifically limited. The order of steps can be adjusted or some steps can be reduced or added according to actual needs.

[0072] Step 101: Obtain multiple sets of detection images and reference images, and input the multiple sets of detection images and reference images into the initial segmentation large model. The initial segmentation large model includes a large model, a first image encoder, a second image encoder, a mask cue attention module, a text cue encoder, a mask cue encoder, and a classification mask decoder. Furthermore, each reference image includes defect reference annotations, and different sets of defect reference annotations belong to different defect categories.

[0073] Step 101 will be described in detail below.

[0074] In some embodiments, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the structure and data processing of an optional initial segmentation large model for the wind turbine blade defect detection method based on a segmentation large model provided in this application embodiment. A structure-specific initial segmentation large model is constructed and initialized. This model is a composite architecture integrating multiple dedicated functional modules. Its core components include a large model for generating text descriptions, a first image encoder and a second image encoder for extracting visual features from different images, a mask cue attention module for generating visual guidance information, a text cue encoder for processing text information, a mask cue encoder for processing visual guidance information, and a classification mask decoder for final decision-making.

[0075] Here, "large model" refers to a pre-trained deep learning foundational model with massive parameters. Because large models are pre-trained on large-scale general data (such as natural images and text), they possess powerful feature extraction and semantic understanding capabilities. In this embodiment, the large model serves as the semantic and visual foundation, providing cross-modal prior knowledge to help the model understand the detected image and generate corresponding defect description text. For example, the large model could be a series of Generative Pre-trained Transformer (GPT) models or a Large Language Model Meta AI (LLaMA) series; this embodiment does not impose any limitations on this.

[0076] The first image encoder is used to encode the detection image to extract its deep visual features. Specifically, the first image encoder converts the input detection image into a high-dimensional feature map or feature vector for use by the subsequent decoder or attention module. In this embodiment, the first image encoder is a Segment Anything Model (SAM), which is responsible for efficiently extracting deep visual feature representations rich in semantic and spatial information from the input unlabeled detection image, thereby capturing the relationship between the contour, texture, and context of the wind turbine blades in the detection image. The second image encoder is structurally similar to the first image encoder, except that it is used to encode a reference image with defect reference annotations, so as to extract the visual features of the defect region from the reference image by combining the defect annotation information. In this embodiment, the second image encoder is a Vision Transformer (ViT), which segments the reference image into multiple patches and converts them into a sequence input, and uses the global self-attention mechanism of the Transformer to process and obtain visual features, wherein the visual features include the defect mask features corresponding to the reference image and the reference image features.

[0077] The mask cue attention module receives the defect shape (defect mask features) and defect appearance (reference image features) provided by the second image encoder. Then, by calculating the importance of each feature, it generates a weighted and more informative mask cue. The mask cue not only tells the downstream model what the defect "shapes" but also emphasizes "which visual feature points are most critical" within this shape, thus upgrading a static template into a dynamic and focused guiding signal.

[0078] The text prompt encoder receives human-readable defect description text generated by a large model and converts it into a numerical vector that the machine can understand and process—the text prompt encoding feature. In this way, the text prompt encoder encodes the text, transforming the high-order semantic concepts such as defect category, morphology, and location into a mathematical representation that can be calculated and compared in a multi-dimensional space.

[0079] The mask cue encoder receives mask cue information rich in context generated by the mask cue attention module and performs final encoding processing to obtain a compact and uniform mask-encoded feature. This step aims to further refine and normalize the features of the attention-weighted complex visual cues, making them a standard-format feature vector so that they can be aligned in dimension and format with features from other modalities (such as text and original images), preparing them for multimodal feature fusion in the decoder.

[0080] The classification mask decoder is used to fuse all input information and make the final defect judgment. Specifically, as the hub of information fusion, the classification mask decoder receives and integrates features from three (or four, a fourth type of feature will be introduced later) different channels: detection image encoding features representing the original vision, text cue encoding features representing semantic knowledge, and mask encoding features representing visual paradigms (and possibly temporal encoding features). Through its complex internal self-attention and cross-attention mechanisms, the classification mask decoder performs deep joint decoding processing on these multimodal features, allowing different information sources to verify and enhance each other, ultimately outputting accurate target detection information, that is, determining the location, category, and contour of the defect. Based on the target detection information, the module parameters of the mask cue attention module and the decoder parameters of the classification mask decoder can be adjusted to obtain the trained target segmentation large model.

[0081] Furthermore, after constructing the initial segmentation model, the original dataset is obtained, which includes multiple sets of detection images and reference images. These images are then input into the initial segmentation model, allowing its various modules to extract, align, and fuse features, achieving accurate localization and classification of defect regions. It should also be noted that each reference image includes defect reference annotations, and different sets of these annotations belong to different defect categories.

[0082] Defect reference annotation refers to the precise marking of defect regions and related information in a reference image, either manually or semi-automatically. One method of defect reference annotation involves delineating a region in the reference image and labeling that region with the corresponding defect category. Defect reference annotation provides a "standard answer" for the initial training of the large segmentation model, guiding the model to learn how to identify and locate specific defect regions. During training, it also serves as a supervisory signal, helping the model establish a mapping relationship between the input image and defect locations.

[0083] Defect categories are a labeling system used to classify different types of defects, distinguishing their nature or cause. For example, in wind turbine blade defect detection, common defect categories include, but are not limited to, "cracks," "delamination," "edge peeling," "surface scratches and wear," and "dents and impact damage." Each defect category represents a defect pattern with specific visual characteristics. Defect reference labels in different sets of reference images belong to different defect categories, meaning that the entire original dataset covers multiple types of defects, thus enabling the trained target segmentation model to have generalized detection capabilities.

[0084] Furthermore, the content of the detection image and the reference image can be the same or different. When the content is the same, the initial segmentation model uses known reference annotations to guide the precise localization of defects in the same image, enhancing the model's sensitivity to details. When the content is different, this is commonly seen in cross-image contrastive learning or template matching tasks. For example, a defect-free "standard sample" is used as a reference to detect abnormal regions in another image to be inspected, thereby achieving defect identification and segmentation. Therefore, whether the image content is the same depends on the task requirements: the same content is conducive to fine-grained optimization and iterative enhancement, while different content is more in line with the "standard vs. inspected" application logic in actual industrial inspection, thus improving the model's generalization ability.

[0085] Step 102: For the same set of detection images and reference images, the first image encoder is used to perform image feature extraction processing on the detection images to obtain detection image encoding features. The large model is used to generate defect description text for the detection images, and the text prompt encoder is used to perform semantic feature extraction processing on the defect description text to obtain text prompt encoding features.

[0086] Step 102 is described in detail below.

[0087] In some embodiments, for the same set of detection images and reference images, the initial segmentation large model processes the detection images in parallel to extract their inherent visual information and higher-order semantic information, respectively. Specifically, using the first image encoder, deep feature extraction is performed on the detection images to obtain detection image encoding features containing rich texture, shape, and spatial information; simultaneously, leveraging the powerful image understanding and text generation capabilities of the large model, a descriptive defect description text is automatically generated for the detection image. Subsequently, this defect description text is fed into a text prompt encoder for processing, and through semantic analysis and encoding, it is transformed into machine-understandable text prompt encoding features containing defect concept knowledge.

[0088] In some embodiments, such as Figure 4 As shown, Figure 4 yes Figure 2 Step 102, an optional implementation flowchart, utilizes a large model to generate defect description text for the detected image, including the following steps:

[0089] 102.1.1 The large model is used to perform image feature analysis on the detected image to obtain initial image features, and the defect area of ​​the corresponding wind turbine blade is determined based on the initial image features;

[0090] 102.1.2 If the defect area is greater than the preset area threshold, the large model obtains prior knowledge from the pre-built prior knowledge base. The prior knowledge includes at least the corresponding defect morphology information and defect location information of the wind turbine blade under various defect categories.

[0091] 102.1.3 Based on the initial image features, determine the current defect category of the corresponding wind turbine blade. From the multiple defect categories indicated by prior knowledge, select at least one that matches the current defect category as the target defect category. Based on the defect morphology information and defect location information under the target defect category, generate the defect description text of the detection image.

[0092] 102.1.4 If the defect area is not greater than the preset area threshold, the detection image is subjected to feature enhancement processing to obtain an enhanced detection image, and a defect description text of the enhanced detection image is generated.

[0093] Steps 102.1.1 to 102.1.4 are described in detail below.

[0094] In this embodiment, to generate high-quality defect description text, a large model is first used to perform preliminary image feature analysis on the input detection image to extract initial image features. Based on these initial image features, the initial segmentation large model can perform preliminary defect region evaluation, thereby determining the defect area occupied by potential defects in the detection image.

[0095] Furthermore, when the defect area exceeds a preset area threshold, it indicates that the defect is a significant large target, making it suitable to utilize structured expert knowledge for guidance. In this case, the large model will call upon and access a pre-built prior knowledge base, which is a structured database or knowledge graph that stores a large amount of expert experience and historical data on wind turbine blade defects. The prior knowledge is carefully organized and includes at least the typical defect morphology information (such as shape, texture, and color) and common defect location information (such as blade tip, blade root, and leading edge) corresponding to various defect categories (such as cracks, corrosion, and delamination), so as to generate specific and accurate defect description text later.

[0096] Furthermore, after acquiring prior knowledge, the large model first uses general knowledge to analyze and process the initial image features to perform preliminary classification of the detected image, determining the most likely current defect category to which the wind turbine blade indicated by the detected image belongs. Then, the large model matches this current defect category with multiple defect categories from prior knowledge, selecting at least one that best matches the current situation as the target defect category. Finally, the large model comprehensively utilizes the specific defect morphology and defect location information under this target defect category to generate a rich and specific defect description text for the current detected image. For example, the defect description text might describe it as "A suspected linear crack was detected in the middle of the leading edge of the blade, distributed longitudinally."

[0097] It should be noted that in actual wind turbine blade defect detection, multiple concurrent or related defect types may exist in the same damaged area or adjacent areas. Therefore, this application embodiment will use at least one defect category as the target defect category. For example, a lightning strike may simultaneously cause blade surface ablation (coating peeling) and structural carbonization (cracks or delamination), or leading edge wear may further trigger edge spalling and surface cracks. From another perspective, when faced with fuzzy or atypical defect features, the initial segmentation model usually cannot uniquely determine a defect category, but instead provides multiple candidate categories with higher probability (such as the categories with the highest confidence) to support subsequent manual review or multimodal verification. Therefore, allowing the selection of one or more matching defect categories as the target defect category can more comprehensively and accurately reflect the complex damage situation in real-world scenarios, thereby improving the robustness of the model's defect detection.

[0098] In some embodiments, if the defect area is not greater than a preset area threshold, it means that the wind turbine blade defect may be a small, hard-to-detect target or early damage. In this case, directly analyzing the original detection image may be ineffective due to the low signal-to-noise ratio. Therefore, the initial detection model first performs feature enhancement processing on the original detection image, such as through contrast stretching, sharpening, or super-resolution techniques, to highlight the features of the small defect, thereby obtaining an enhanced detection image. Subsequently, the larger model generates corresponding defect description text based on this enhanced detection image with more obvious defect features. This ensures that even when dealing with small target defects, the model can obtain high-quality text prompts, avoiding information loss due to the small size of the target. It should be noted that the preset area threshold can be set according to actual conditions, and this application embodiment does not limit it.

[0099] Furthermore, after obtaining the defect description text, it can be preprocessed to improve its quality. Specifically, preprocessing includes, but is not limited to, text cleaning, word segmentation and stop word removal, and formatting.

[0100] It is understood that the embodiments of this application can differentiate between large and small area defects: for large defects, a precise and detailed defect description is generated using structured prior knowledge. For small defects, features are first amplified using image enhancement technology to ensure the effectiveness of the prompt information; then, prior knowledge is used to generate defect description text. This differentiated processing strategy results in higher quality and more information-rich defect description text, thus providing a very strong semantic constraint for subsequent multimodal fusion decoding, greatly improving the model's accuracy and robustness in identifying defects of various types and sizes of wind turbine blades.

[0101] The following describes how the embodiments of this application specifically perform feature enhancement processing on detection images containing small defect areas:

[0102] In some embodiments, such as Figure 5 As shown, Figure 5 yes Figure 4 Step 102.1.4, an optional implementation flowchart, describes the process of performing feature enhancement on the detection image to obtain an enhanced detection image, including the following steps:

[0103] A.1 Divide the detection image into multiple initial image blocks, and perform histogram equalization on each initial image block to obtain equalized image blocks;

[0104] A.2 Using a preset bilinear interpolation function, grayscale value mapping is performed on the equalized image blocks to obtain grayscale value image blocks, and an initial enhanced detection image is obtained based on all grayscale value image blocks;

[0105] A.3 Using a preset wavelet basis function, the initial enhanced detection image is decomposed into multiple sub-bands using wavelet decomposition, and the sub-bands are then subjected to feature enhancement processing based on a set gain value to obtain the enhanced detection image.

[0106] Steps A.1 to A.3 are described in detail below.

[0107] In some embodiments, a given inspection image may include both large and small defects. If a suspected small defect is detected, it indicates a complex background. To avoid overlooking any defects and misidentifying a complex background as a wind turbine blade defect, feature enhancement processing is required. Specifically, the entire inspection image is first divided into multiple initial image blocks of uniform size. Then, histogram equalization is performed on each individual initial image block. Histogram equalization redistributes the pixel gray levels within the image block, making the pixel values ​​of each block more evenly distributed across the entire gray range, thus resulting in an equalized image block with improved contrast. It is understood that by using block-based processing rather than global processing, image effects dominated by a single lighting condition can be effectively avoided, thus better adapting to complex lighting variations on the blade surface.

[0108] Furthermore, to smooth out potential boundary effects caused by independent processing of each equalized image block and to integrate the results of block processing, after obtaining the equalized image blocks, the large model will use a pre-defined bilinear interpolation function to perform grayscale value mapping on all equalized image blocks. The bilinear interpolation function is a method of interpolation in two-dimensional space that estimates the value of a point by weighted averaging of four known neighboring points (usually pixels). Specifically, this method first performs linear interpolation in the horizontal direction, then linear interpolation in the vertical direction, or vice versa, ultimately obtaining the interpolated result for the target point.

[0109] In this embodiment, the large model determines the grayscale values ​​of the four nearest neighbor pixels around each pixel using a bilinear interpolation function, and calculates a new grayscale value by weighting their distances to the target location, thus obtaining a grayscale image block. In this way, grayscale mapping processing effectively eliminates the "blocking effect" between equalized image blocks, making the grayscale changes in the transitions between image blocks more natural and smooth. Furthermore, after processing all image blocks, they are stitched together to obtain an initial enhanced detection image with improved overall contrast and a coherent visual effect.

[0110] Furthermore, after obtaining the initial enhanced detection image, it can be directly used as the final enhanced detection image, or the initial enhanced detection image can be further enhanced, and the further enhanced image can be used as the final enhanced detection image. In the latter case, the large model will use preset wavelet basis functions to perform feature enhancement processing on the initial enhanced detection image. Wavelet basis functions are functions generated by scaling (scaling) and translating (displacement) a mother wavelet function, used to analyze the local features of signals at different scales and locations, thus providing good localization capabilities in both the time and frequency domains.

[0111] To further sharpen and highlight the edge and texture details of defects, in this embodiment, the large model first uses a preset wavelet basis function (such as Haar wavelet or Daubechies wavelet) to perform multi-level wavelet decomposition on the initial enhanced detection image obtained in the previous step. Wavelet decomposition can decompose the image into multiple sub-bands of different frequencies and directions. The high-frequency sub-band usually contains abrupt information such as the edges and details of the image—these are the most prominent features of defects (such as cracks, scratches, and burrs). Therefore, enhancing the high-frequency sub-band directly strengthens these key visual details, making the wind turbine blade defects clearer. The low-frequency sub-band, on the other hand, represents the outline and background of the detection image. For example, the low-frequency sub-band can be a large area of ​​gray background, such as the wall, sky, or other parts with gentle color changes. That is, the low-frequency sub-band represents the smooth area and overall outline of the detection image. It does not contain the key details of the defects. Therefore, enhancing the low-frequency sub-band not only fails to highlight the defects but may also cause the overall detection image to be too bright, the contrast to be unbalanced, or even to cover up the real defects.

[0112] Therefore, the large model will selectively perform feature enhancement processing on the high-frequency decomposition subband containing defect details based on a pre-set gain value (usually a coefficient greater than 1); and then reconstruct the enhanced high-frequency subband and the unprocessed low-frequency subband through wavelet inverse transform to finally obtain an enhanced detection image with significantly enhanced detail features.

[0113] Understandably, the embodiments of this application address the problem of uneven local illumination through block segmentation and histogram equalization; then, grayscale mapping processing ensures the integrity and smoothness of the detected image; finally, wavelet transform is used to accurately amplify the edge and detail features of wind turbine blade defects at multiple scales without excessively enhancing background noise. This gradual and progressive enhancement strategy provides a detected image with a higher signal-to-noise ratio and more prominent defect features for subsequent large-scale model analysis, thereby significantly reducing the risk of missed detections due to small defect size or poor imaging quality, and providing a solid foundation for defect detection of the entire wind turbine blade.

[0114] The above describes how to process the detection image using the first image encoder, the large model, and the text encoder. Next, we will describe in detail how the mask cue attention module and the mask cue encoder in the initial segmentation large model process the reference image:

[0115] Step 103: Guided by the defect reference annotation, the second image encoder determines the defect mask features and reference image features of the reference image. The mask cue attention module determines the mask cue information based on the defect mask features and reference image features. The mask cue encoder encodes the mask cue information to obtain the mask encoded features.

[0116] Step 103 will be described in detail below.

[0117] In some embodiments, the reference image is analyzed using a second image encoder, guided by the defect reference annotations corresponding to the reference image, to determine two key pieces of information: (1) defect mask features representing the precise shape of the defect; and (2) reference image features representing the overall visual content of the reference image. Then, through the attention mechanism within the mask cue attention module, the aforementioned defect mask features and reference image features are fused to generate a mask cue information highly focused on the defect paradigm. This information is then encoded by the mask cue encoder to form a compact mask-encoded feature, which acts as an intuitive "visual sample," thus prompting the model to identify the specific "look" of the defect during the initial segmentation model training process, better assisting in the initial segmentation model training.

[0118] In some embodiments, such as Figure 6 As shown, Figure 6 yes Figure 2 Step 103, an optional implementation flowchart, guided by defect reference annotations, utilizes a second image encoder to determine the defect mask features and reference image features of the reference image, including the following steps:

[0119] 103.1.1 The second image encoder performs feature extraction processing on the reference image to obtain initial reference image features, and performs convolution and pooling processing on the initial reference image features to obtain reference image features;

[0120] 103.1.2 The second image encoder, guided by the defect reference annotation, performs average pooling on the features of the initial reference image to obtain the defect mask features.

[0121] Steps 103.1.1 to 103.1.2 are described in detail below.

[0122] In some embodiments, to accurately separate visual information representing defect regions from a reference image, a second image encoder is first used to perform preliminary feature extraction processing on the reference image to obtain initial reference image features. In this embodiment, the second image encoder is a Vision Transformer (ViT), which, under the supervision of reference defect annotations, divides the reference image into small blocks and uses the self-attention mechanism of the Transformer to model the global contextual relationships in the reference image and capture the dependencies between distant regions in the reference image, thereby learning the differential features between defect regions and normal regions. Then, through upsampling or mask prediction, defect mask features aligned with the input reference image are generated.

[0123] For example, the reference image is I r The initial reference image features f(I) are extracted by the ViT image encoder. r Furthermore, ViT performs two more convolutional processes on the initial reference image features and inputs the convolutional results into a pooling layer to obtain the final reference image features f′(I). r This process can be achieved through the following formula: <1> express:

[0124] f′(I r =POOL(Conv(Conv(f(I)) r )))) <1>

[0125] Where Conv represents convolution; POOL represents pooling.

[0126] Furthermore, ViT will generate a binary defect mask I for the reference image based on the defect range and type indicated by the defect reference annotation. rm Furthermore, for the defect mask I rm The initial reference image features are then subjected to average pooling to obtain the defect mask features f(I). rm This process can be achieved through the following formula: <2> express:

[0127] f(I rm ) = avgpoolmask(f(I r ),I rm ) <2>

[0128] Understandably, steps 103.1.1 to 103.1.2 can be guided by the defect reference annotations of the reference image, and during the model training process, the interference of background information can be eliminated to the greatest extent, so as to focus entirely on the key attributes of the defect itself, such as shape, texture and color. In this way, the initial segmentation model can obtain a clear and unambiguous reference during the model learning process, which greatly improves the model's ability to identify and generalize defects of specific categories, and solves the problem of difficulty in distinguishing similar texture backgrounds from defects in traditional defect detection methods.

[0129] In some embodiments, the mask cueing attention module can directly utilize defect mask features and reference image features f′(I r Determine the mask hint information; or, based on steps 103.2.1 to 103.2.5, first determine the sub-reference image features at different scales, and then jointly determine the mask hint information based on the sub-reference image features, reference image features, and defect mask features:

[0130] In some embodiments, such as Figure 7 As shown, Figure 7 yes Figure 2 In step 103, another optional implementation flowchart shows that the mask cueing attention module determines mask cueing information based on defect mask features and reference image features, including:

[0131] 103.2.1 Perform linear transformation on the initial reference image features to obtain sub-reference image features at different scales;

[0132] 103.2.2 Use reference image features as query information, sub-reference image features at different scales as key information, and defect mask features as value information;

[0133] 103.2.3 Based on query information, key-value information, and value information, determine feature dimension information;

[0134] 103.2.4 The mask cue attention module determines mask cue information based on query information, key information, value information, and feature dimension information.

[0135] Steps 103.2.1 to 103.2.4 are described in detail below.

[0136] In this embodiment, to enable the initial detection model to understand the characteristics of wind turbine blade defects from multiple perspectives and levels, the second image encoder also performs linear transformation processing on the initial reference image features, for example, by using different fully connected layers or convolutional kernels to generate a set of sub-reference image features f(I) at different scales. rsSub-reference image features can be understood as different "interpretations" or "abstractions" of the original reference features; some may focus more on the global contour, while others may focus more on local texture. This multi-scale representation enriches the feature information and provides a more diverse perspective for subsequent attention calculations, thus helping the initial segmentation model to capture a more comprehensive picture of wind turbine blade defects.

[0137] Furthermore, to determine the mask cues, the mask cue attention module designates reference image features as query information (Q), which represents the core question of "what am I looking for"; it designates sub-reference image features at different scales as key information (K), which constitutes an "information index" for querying; and it designates defect mask features representing the precise shape of the defect as value information (V), which is the "content itself" that ultimately needs to be extracted and weighted. By defining Q, K, and V, the foundation for subsequent attention calculations is laid.

[0138] Furthermore, in order to appropriately scale the attention score during subsequent calculations to stabilize the training process, this method needs to predetermine a key normalization factor (feature dimension information). In this embodiment, the feature dimension information is Q, K, and V, the aligned dimensions of the three features. This feature dimension information can scale the dot product result during subsequent calculation of dot product attention, thereby preventing the dot product result from being too large due to excessive dimensionality, which would affect the learning efficiency of the initial segmentation model.

[0139] Furthermore, the mask cueing attention module determines the mask cue information Q1 based on query information, key-value information, value information, and feature dimension information. This process can be specifically described by the following formula. <3> express:

[0140]

[0141] Where T represents transpose processing; d represents feature dimension information.

[0142] Specifically, the mask cue attention module first converts the query information Q into the transpose K of the key-value information K. T Matrix multiplication is performed to obtain a transpose result, where each element represents the similarity or relevance between a part of the query information and a part of the key-value information. Subsequently, to prevent gradient vanishing or exploding, the system divides this transpose result element-wise with the feature dimension information (usually its square root) determined in the previous step, thus obtaining a scaled ratio result.

[0143] Furthermore, the mask cue attention module normalizes this ratio result, typically using the Softmax function, to obtain a normalized result. This normalized result is a weight matrix whose sum of all elements is 1, and each element represents the importance of the corresponding part of the value information (V). Finally, this normalized result (i.e., the attention weights) is multiplied by the defect mask features that constitute the value information (V). This weighted summation process essentially uses the correlation calculated by Q and K to selectively extract information from V according to its importance, and the final output is the mask cue information containing rich contextual information.

[0144] The Softmax function is a commonly used activation function for multi-class classification problems. It transforms a vector containing arbitrary real numbers into a probability distribution vector, where each element takes a value between 0 and 1, and the sum of all elements is 1. Specifically, Softmax highlights larger values ​​and suppresses smaller values ​​by performing an exponentiation operation on each element of the vector and then dividing by the sum of all element exponents, thus clearly expressing the relative probability of each class. It is often used in the output layer of neural networks to represent the predicted probability of each class.

[0145] Understandably, the mask cue attention module intelligently assigns weights to different parts of the defect mask (V) by calculating the correlation between the reference image features (Q) and their multi-scale representation (K). This elevates the simple binary shape information (defect mask) into a dynamic, weighted feature representation that includes "which part of the defect should be emphasized." This mechanism enables the initial segmentation model to not only know the "shape" of the defect based on the mask cue information but also understand the "primary and secondary" aspects of its internal features. This provides the downstream decoder with a higher-quality, more guiding visual paradigm, thereby helping to improve the model's ability to accurately identify specific defects.

[0146] In some embodiments, such as Figure 8 As shown, Figure 8 yes Figure 2 Step 103 in the flowchart is another optional implementation. After the mask cue encoder encodes the mask cue information to obtain the mask encoded features, it also includes the following steps:

[0147] 103.3.1 Obtain the historical detection images corresponding to the detected images at historical time points;

[0148] 103.3.2 The temporal feature encoder performs feature encoding processing on the detected image and historical detected images to obtain temporal encoded features;

[0149] 103.3.3 Enhanced detection information is obtained by using a classification mask decoder to perform multimodal feature joint decoding of the detection image coding features, text prompt coding features, mask coding features, and temporal coding features;

[0150] 103.3.4 Based on the enhanced detection information, the module parameters of the mask cue attention module and the decoder parameters of the classification mask decoder are adjusted respectively, and the trained target segmentation large model is obtained after the parameter adjustment is completed.

[0151] Steps 103.3.1 to 103.3.4 are described in detail below.

[0152] In some embodiments, the initial segmentation of the large model can, in addition to, be as follows: Figure 3 As shown, in addition to the large model, the first image encoder, the second image encoder, the mask cue attention module, the text cue encoder, the mask cue encoder, and the classification mask decoder, the initial segmentation large model can also include a temporal feature encoder based on these modules.

[0153] In other words, to incorporate time-dimensional information to aid in determining the evolution trend of defects, in some embodiments, for the detection image to be analyzed, at least one historical detection image corresponding to one or more historical moments in the past can also be obtained. This at least one historical detection image constitutes a temporal sequence prior sample of the detection image in its current state, thereby enabling the initial segmentation model to expand from static single-frame image analysis to dynamic change process analysis.

[0154] The temporal feature encoder can be a network module adept at processing sequential data, such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or a video processing network based on an attention mechanism. The temporal feature encoder takes the currently detected image and the previously acquired historical images as input, performs temporal feature encoding on these two (or more) images, thereby capturing the differences between images at different times, predicting possible growth, expansion, or morphological changes, and ultimately generating a temporal encoded feature that can characterize this temporal dynamic.

[0155] Furthermore, similar to steps 104 and 105, the classification mask decoder will serve as the central hub for information fusion. The input it receives not only includes the original three features—namely, the detection image coding feature representing the current vision, the text prompt coding feature representing semantic knowledge, and the mask coding feature representing the visual paradigm—but also the temporal coding feature generated in the previous step, representing dynamic changes. Thus, the classification mask decoder can utilize its powerful multimodal feature joint decoding processing capability to deeply fuse these four features from different sources and with different properties, and provide a more comprehensive and accurate prediction result by integrating all information, i.e., enhancing the detection information.

[0156] Furthermore, after obtaining the enhanced detection information with fused temporal information, the module parameters of the mask cue attention module and the decoder parameters of the classification mask decoder are adjusted and optimized only in a targeted manner, so as to complete the parameter adjustment of the initial segmentation model with as little resource consumption as possible, and obtain the target segmentation model.

[0157] It is understood that the embodiments of this application greatly enhance the model's defect diagnosis and prediction capabilities by introducing temporal dimension information. Thus, not only can substantial features be determined based on the currently input detection image and reference image, but the "state" of the defect—whether static and stable, or dynamic and evolving—can also be predicted by comparing historical images. Furthermore, by jointly decoding temporal encoded features with the other three modal features, the model's decision-making basis becomes more comprehensive and multi-dimensional. While maintaining the advantages of efficient parameter training, this further improves the accuracy and reliability of defect detection and endows the model with preliminary defect evolution trend analysis capabilities.

[0158] Steps 104 and 105 will be explained in detail below. Readers can also gradually appreciate the beneficial effects of steps 103.3.1 to 103.3.4 as they read on.

[0159] Step 104: Use the classification mask decoder to perform multimodal feature joint decoding processing on the detection image coding features, text prompt coding features and mask coding features to obtain target detection information. Based on the target detection information, train the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder to obtain the trained target segmentation large model.

[0160] Step 104 is described in detail below.

[0161] In some embodiments, the classification mask decoder, acting as the center of information fusion and decision-making, receives and processes three heterogeneous features generated in the preceding steps: detection image encoding features representing the original vision, text prompt encoding features representing semantic knowledge, and mask encoding features representing visual paradigms. The classification mask decoder, through its complex internal network structure, performs joint decoding processing on these three modalities to synthesize all information sources and provide a preliminary prediction result, i.e., target detection information.

[0162] Furthermore, based on the difference between the detected object information and the ground truth annotations, the module parameters of the mask cue attention module and the decoder parameters of the classification mask decoder are specifically optimized and trained iteratively. This parameter-efficient training strategy greatly reduces computational resource consumption while ensuring model performance, ultimately resulting in a well-trained and high-performance large-scale object segmentation model.

[0163] The module parameters of the mask cueing attention module mainly include the projection weight matrices of the query, key, and value in the multi-head attention mechanism, the scaling factor for attention score normalization, and the layer normalization parameters for cross-modal interactions. The decoder parameters of the classification mask decoder cover the weights of the multi-modal feature fusion layer, the fully connected layer parameters of the classification head, the weights and biases of the mask prediction convolutional kernel, and the learnable parameters of any dynamic convolutional or cross-attention layers. Both are jointly trained and optimized to form the core functional components of the large-scale target segmentation model. Of course, the specific module and decoder parameters can be adjusted according to actual conditions, and this embodiment does not impose any limitations on this.

[0164] In some embodiments, such as Figure 9 As shown, Figure 9 yes Figure 2 Step 104, an optional implementation flowchart, utilizes a classification mask decoder to perform multimodal feature joint decoding processing on the detection image encoding features, text prompt encoding features, and mask encoding features to obtain target detection information, including the following steps:

[0165] 104.1.1 Based on the self-attention mechanism, the classification mask decoder performs feature enhancement processing on the text prompt encoding features to obtain enhanced text prompt encoding features;

[0166] 104.1.2 Based on the cross-attention mechanism, feature fusion processing is performed on the enhanced text prompt coding features and the detection image coding features to obtain the first fused coding features, and nonlinear transformation processing is performed on the fused coding features to obtain nonlinear coding features;

[0167] 104.1.3 The mask coding features are mapped to obtain the mapped mask coding features. Based on the cross-attention mechanism, the mapped mask coding features and the nonlinear coding features are fused to obtain the second fused coding features.

[0168] 104.1.4 By concatenating the enhanced text prompt coding features, the mapped mask coding features, and the second fused coding features, target detection information is obtained.

[0169] Steps 104.1.1 to 104.1.4 are described in detail below.

[0170] In some embodiments, such as Figure 10 As shown, Figure 10 This is a schematic diagram illustrating the internal data processing of an optional classification mask decoder in the wind turbine blade defect detection method based on a segmentation large model provided in this application embodiment. To fully explore the semantic relationships within the text prompts, the classification mask decoder first enhances the encoded features of the input text prompts internally based on a self-attention mechanism. The self-attention mechanism can calculate the interdependencies between elements within a sequence. For text, this means that the initial segmentation large model can understand the grammatical structure and semantic relationships between different words in the text (e.g., the relationship between "crack" and "edge"), thereby better capturing the deeper meaning of the text prompts and obtaining an enhanced text prompt encoding feature with richer contextual information and clearer internal logic, laying a solid foundation for subsequent fusion with visual features.

[0171] Furthermore, to guide visual information through textual information, the text and image features are fused across modally for the first time. Specifically, the enhanced text cue encoding features obtained in the previous step and the original detection image encoding features are fused using a cross-attention mechanism. In cross-attention, features from one side (e.g., text) serve as the query, and features from the other side (e.g., image) serve as the key and value, thereby calculating the "focus" of the textual information on the visual feature map. This process effectively maps semantic concepts to image features, resulting in a first fused encoding feature that is an intermediate representation combining text and image. Subsequently, the classification mask decoder performs a nonlinear transformation on this first fused encoding feature, for example, using a multilayer perceptron (MLP) to enhance the model's expressive power, resulting in the final nonlinear encoding feature.

[0172] Furthermore, to incorporate the third modality feature (i.e., mask-encoded feature) into the decoding process, the classification mask decoder performs a second cross-modal fusion. First, the input mask-encoded feature is mapped, for example, through a linear layer to match its dimension with the non-linear coding feature, resulting in a mapped mask-encoded feature. Then, based on a cross-attention mechanism, this mapped mask-encoded feature and the non-linear coding feature generated in the previous step are fused. In this fusion, the mapped mask-encoded feature serves as a query to find the most similar part among the fused non-linear coding features, ultimately yielding a second fused coding feature that integrates information from all three modalities.

[0173] Furthermore, after completing all independent fusion steps, the classification mask decoder integrates three key feature information—enhanced text cue coding features, mapped mask coding features, and second fusion coding features—through a concatenation operation to form the final output. For example, the detection image coding features extracted by the first image encoder are f(I) d The text coding features extracted by the text prompt encoder are f(K). p The mask-encoded features Q1 extracted by the mask cue encoder are concatenated and fused, and used as input to the classification mask decoder, and then processed using the following formula. <4> The target detection information is obtained by fusion:

[0174] f=CrossAttention(Q1,MLP(CrossAttention(f(K p ),f(I d )))) <4>

[0175] In the formula, CrossAttention represents cross-attention.

[0176] Furthermore, during model training, the parameters of the main model, the first image encoder, the second image encoder, the text cue encoder, and the mask cue encoder are frozen. Only the module parameters of the mask cue attention module and the decoder parameters of the classification mask decoder are trained. During training, the training and validation sets from the original dataset are used to obtain segmentation image results and confidence scores. Evaluation metrics such as accuracy, recall, and F1 score are used to assess model performance and analyze the model's performance across different defect categories. The test set is used for model inference, and the inference results and causes of errors are calculated. After multiple iterations and model updates, the model parameters and structure are optimized. Additionally, the trained model parameters and decoder parameter structure can be saved to a specified storage medium.

[0177] Understandably, the classification mask decoder first strengthens the internal logic of the text prompts through self-attention, then gradually combines text guidance with visual features through two hierarchical cross-attention processes, further corroborates the visual paradigms with the image-text fusion results, and finally retains all key information through a concatenation operation. This structured fusion strategy ensures that information from different modalities can fully interact, complement, and verify each other, rather than simply being mixed together. This results in extremely high-quality target detection information, providing a solid and reliable basis for the model to make accurate defect judgments, and significantly improving the accuracy and robustness of detection.

[0178] Furthermore, if temporal coding features are introduced as in steps 103.3.1 to 103.3.4, they can be fused as a fourth modality of information. In the first cross-attention operation, the temporal coding features are used as a query and participate in the cross-attention calculation together with the enhanced text prompt coding features and the detection image coding features to capture the correlation in the temporal dimension. The resulting first fused coding feature, after MLP processing, continues to undergo a second cross-attention fusion with the mask coding features. Finally, the enhanced text prompt coding features, the mapped mask coding features, and the second fused coding features are integrated through a concatenation operation to form an output that combines four modalities of information. This multi-level, multi-modal fusion strategy can more comprehensively utilize temporal information and further improve the model's defect detection accuracy and robustness in dynamic scenes.

[0179] Step 105: Obtain the target detection image and input the target detection image into the target segmentation large model to obtain the defect detection result.

[0180] Step 105 is described in detail below.

[0181] In some embodiments, after completing the model training to obtain the target segmentation large model, the transition from the training stage to the practical application or inference stage is: acquiring new wind turbine blade images that the model has never seen before, performing defect detection, and using them as input, directly feeding them into the trained target segmentation large model. Then, the model will automatically perform the complete feature extraction, multimodal information fusion and decoding process, and finally output accurate defect detection results.

[0182] Defect detection results are typically presented in the form of structured data and visual output, and may include the following (example):

[0183] (1) Pixel-level segmentation mask: Precise pixel-level segmentation of the defect region (binary mask), which can be used to calculate the area or shape of the defect. Example output: A matrix of the same size as the input image, with pixel values ​​of 1 for the defect region and 0 for the background.

[0184] (2) Defect Category and Confidence: The defect category can be represented as "crack", "corrosion", or "coating peeling". Each defect category corresponds to a confidence score (e.g., "crack: 92%"). Example output: {"defect_type":"crack","confidence":0.92,"location":[x1,y1,x2,y2]}.

[0185] (3) Defect location and bounding box: The rectangular coordinates of the defect in the target detection image (e.g., pixel-level bounding box [x_min, y_min, x_max, y_max]). Example visualization: Mark the defect area with a red box on the input image and label the category name.

[0186] (4) Multimodal association results: Based on text prompts (such as “detect blade leading edge cracks”), output enhanced results of specific defects, such as only displaying defects that match the text description.

[0187] It should be noted that the specific content included in the defect detection results can be adjusted according to the actual situation, and the embodiments of this application do not impose any restrictions on this.

[0188] It is understood that the embodiments of this application construct a multimodal defect detection framework guided by both text prompts and mask prompts. This framework efficiently integrates the visual features of the detected image itself, the semantic description features generated by the large model, and the exemplary visual features provided by reference images. This enables the model to collaboratively analyze from three dimensions: "seeing the object clearly," "understanding the concept," and "referencing the sample." This greatly overcomes the problems of feature loss and insufficient generalization ability caused by the single feature information in conventional models. Through this complementarity and enhancement of multi-source information, combined with a parameter-efficient training strategy, the trained target segmentation model can significantly improve the detection accuracy and robustness of wind turbine blade defects in complex backgrounds and of various types while ensuring the model's lightweight nature and ease of deployment.

[0189] like Figure 11 As shown, Figure 11 This is a schematic diagram of an optional device module for a wind turbine blade defect detection device based on a segmented large model, provided in an embodiment of this application. The wind turbine blade defect detection device based on a segmented large model may include the following modules 201 to 205:

[0190] The acquisition module 201 is used to acquire multiple sets of detection images and reference images, and input the multiple sets of detection images and reference images into the initial segmentation large model. The defect detection model includes a large model, a first image encoder, a second image encoder, a mask cue attention module, a text cue encoder, a mask cue encoder, and a classification mask decoder. Furthermore, each reference image includes defect reference annotations, and different sets of defect reference annotations belong to different defect categories.

[0191] The first encoding module 202 is used to extract image features from the detection images using a first image encoder to obtain detection image encoding features for the same set of detection images and reference images, generate defect description text of the detection images using a large model, and extract semantic features from the defect description text using a text prompt encoder to obtain text prompt encoding features.

[0192] The second encoding module 203 is used to determine the defect mask features and reference image features of the reference image by using the second image encoder, guided by the defect reference annotation. The mask cueing attention module determines the mask cue information based on the defect mask features and reference image features. The mask cueing encoder encodes the mask cue information to obtain the mask encoding features.

[0193] Training module 204 is used to perform multimodal feature joint decoding processing on the detection image coding features, text prompt coding features and mask coding features using the classification mask decoder to obtain target detection information. Based on the target detection information, the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder are trained to obtain the trained target segmentation large model.

[0194] The target processing module 205 is used to acquire the target detection image and input the target detection image into the target segmentation large model to obtain the defect detection result.

[0195] The specific implementation of the wind turbine blade defect detection device based on the segmented large model is basically the same as the specific implementation of the wind turbine blade defect detection method based on the segmented large model described above, and will not be repeated here.

[0196] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method for detecting wind turbine blade defects based on a segmented large model. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0197] like Figure 12 As shown, Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0198] The processor 301 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0199] The memory 302 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 302 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and called and executed by the processor 301.

[0200] Input / output interface 303 is used to implement information input and output;

[0201] The communication interface 304 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0202] Bus 305 transmits information between various components of the device (e.g., processor 301, memory 302, input / output interface 303, and communication interface 304);

[0203] The processor 301, memory 302, input / output interface 303, and communication interface 304 are connected to each other within the device via bus 305.

[0204] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for detecting wind turbine blade defects based on a segmented large model.

[0205] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0206] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0207] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0208] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0210] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0211] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0213] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0215] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0216] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A fan blade defect detection method based on a segmentation large model, characterized in that, The method comprises the steps of: obtaining a plurality of sets of detection images and reference images, and inputting the plurality of sets of detection images and reference images into an initial segmentation large model, wherein the initial segmentation large model comprises a large model, a first image encoder, a second image encoder, a mask prompt attention module, a text prompt encoder, a mask prompt encoder and a classification mask decoder, and each reference image comprises a defect reference label, and the defect categories of different sets of defect reference labels are different; for the same set of detection images and reference images, the first image encoder is used for image feature extraction processing of the detection images to obtain detection image coding features, the large model is used for generating defect description text of the detection images, and the text prompt encoder is used for semantic feature extraction processing of the defect description text to obtain text prompt coding features; using the defect reference label as a guide, the second image encoder is used to determine defect mask features and reference image features of the reference image, the mask prompt attention module is used to determine mask prompt information based on the defect mask features and the reference image features, and the mask prompt encoder is used for encoding processing of the mask prompt information to obtain mask coding features; the classification mask decoder is used for multi-modal feature joint decoding processing of the detection image coding features, the text prompt coding features and the mask coding features to obtain target detection information, and based on the target detection information, the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder are trained respectively to obtain a trained target segmentation large model; obtaining a target detection image and inputting the target detection image into the target segmentation large model to obtain a defect detection result.

2. The fan blade defect detection method based on the segmentation large model according to claim 1, characterized in that, The multi-modal feature joint decoding processing of the detection image coding features, the text prompt coding features and the mask coding features by the classification mask decoder to obtain the target detection information comprises: based on a self-attention mechanism, the classification mask decoder is used for feature enhancement processing of the text prompt coding features to obtain enhanced text prompt coding features; based on a cross-attention mechanism, the enhanced text prompt coding features and the detection image coding features are subjected to feature fusion processing to obtain first fusion coding features, and the fusion coding features are subjected to nonlinear transformation processing to obtain nonlinear coding features; the mask coding features are subjected to mapping processing to obtain mapped mask coding features, and based on a cross-attention mechanism, the mapped mask coding features and the nonlinear coding features are subjected to feature fusion processing to obtain second fusion coding features; the enhanced text prompt coding features, the mapped mask coding features and the second fusion coding features are spliced to obtain the target detection information.

3. The fan blade defect detection method based on the segmentation large model according to claim 1, characterized in that, The generation of the defect description text of the detection image by the large model comprises: the large model is used for image feature analysis processing of the detection image to obtain initial image features, and based on the initial image features, a defect area of a corresponding fan blade is determined. If the defect area is greater than a preset area threshold, the large model obtains prior knowledge from a pre-constructed prior knowledge base, wherein the prior knowledge at least includes corresponding defect morphology information and defect position information of the fan blade under various defect categories; Based on the initial image features, a current defect category to which the fan blade belongs is determined, at least one defect category matching the current defect category is selected from the multiple defect categories indicated by the prior knowledge as a target defect category, and based on the defect morphology information and the defect position information under the target defect category, the defect description text of the detection image is generated; If the defect area is not greater than a preset area threshold, the detection image is subjected to feature enhancement processing to obtain an enhanced detection image, and the defect description text of the enhanced detection image is generated.

4. The fan blade defect detection method based on the segmentation large model according to claim 3, characterized in that, The feature enhancement processing of the detection image to obtain an enhanced detection image comprises: dividing the detection image into multiple initial image blocks, performing histogram equalization processing on each initial image block to obtain an equalized image block; using a preset bilinear interpolation function to perform gray value mapping processing on the equalized image block to obtain a gray value image block, and based on all the gray value image blocks, an initial enhanced detection image is obtained; using a preset wavelet basis function to perform wavelet decomposition on the initial enhanced detection image to obtain multiple decomposition subbands, and based on a set gain value, feature enhancement processing is performed on the decomposition subbands to obtain the enhanced detection image.

5. The fan blade defect detection method based on the segmentation large model according to claim 1, characterized in that, The second image encoder determines the defect mask feature and the reference image feature of the reference image using the defect reference label as a guide, comprising: The second image encoder performs feature extraction processing on the reference image to obtain initial reference image features, and performs convolution processing and pooling processing on the initial reference image features to obtain the reference image features; The second image encoder performs average pooling processing on the initial reference image features using the defect reference label as a guide to obtain the defect mask feature.

6. The fan blade defect detection method based on the segmentation large model according to claim 5, characterized in that, The mask prompt attention module determines mask prompt information based on the defect mask feature and the reference image feature, comprising: performing linear transformation processing on the initial reference image features to obtain sub-reference image features under different scales; taking the reference image features as query information, taking the sub-reference image features under different scales as key-value information, and taking the defect mask feature as value information; determining feature dimension information based on the query information, the key-value information, and the value information; The mask prompt attention module determines the mask prompt information based on the query information, the key-value information, the value information, and the feature dimension information.

7. The fan blade defect detection method based on the segmentation large model according to claim 1, characterized in that, The initial segmentation large model further comprises a time sequence feature encoder; After the mask prompt encoder encodes the mask prompt information to obtain a mask coding feature, it further comprises: obtaining a historical detection image corresponding to the detection image at a historical time; The time sequence feature encoder performs feature encoding processing on the detection image and the historical detection image to obtain time sequence encoding features; The classification mask decoder is used for performing multi-modal feature joint decoding processing on the detection image encoding features, the text prompt encoding features, the mask encoding features and the time sequence encoding features to obtain enhanced detection information; Based on the enhanced detection information, the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder are adjusted respectively, and the target segmentation large model after training is obtained after the parameter adjustment is completed. 8.A fan blade defect detection device based on a segmentation large model, characterized in that, Comprise: The acquisition module is used for acquiring multiple groups of detection images and reference images, and inputting the multiple groups of detection images and reference images into an initial segmentation large model, wherein the initial segmentation large model comprises a large model, a first image encoder, a second image encoder, a mask prompt attention module, a text prompt encoder, a mask prompt encoder and a classification mask decoder, and each reference image comprises a defect reference label, and the defect categories of different groups of defect reference labels are different; The first encoding module is used for performing image feature extraction processing on the detection image by using the first image encoder to obtain detection image encoding features, generating a defect description text of the detection image by using the large model, and performing semantic feature extraction processing on the defect description text by using the text prompt encoder to obtain text prompt encoding features for the same group of detection images and reference images; The second encoding module is used for determining defect mask features and reference image features of the reference image by using the second image encoder with the defect reference label as a guide, determining mask prompt information based on the defect mask features and the reference image features by using the mask prompt attention module, and performing encoding processing on the mask prompt information by using the mask prompt encoder to obtain mask encoding features; The training module is used for performing multi-modal feature joint decoding processing on the detection image encoding features, the text prompt encoding features and the mask encoding features by using the classification mask decoder to obtain target detection information, and training the module parameters of the mask prompt attention module and the decoder parameters of the classification mask decoder based on the target detection information to obtain a target segmentation large model after training; The target processing module is used for acquiring a target detection image, inputting the target detection image into the target segmentation large model to obtain a defect detection result.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the fan blade defect detection method based on the segmentation large model in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the fan blade defect detection method based on the segmentation large model in any one of claims 1 to 7.

Images