Blade damage detection method, apparatus, device, medium, and product

Abstract

This disclosure relates to the field of aero-engine technology, and in particular to a method, apparatus, device, medium, and product for blade damage detection. The method includes: acquiring an image of the blade to be detected and descriptive text; processing the blade image using a pre-trained visual large model to obtain image features; converting the descriptive text into word embedding vectors using the visual large model, and fusing the image features and word embedding vectors to obtain multimodal features; identifying the damage type in the blade image and confirming the damage location based on the visual large model and multimodal features; and acquiring damage contour information in the blade image using the visual large model based on the damage type and location. By employing a visual large model as the feature extraction backbone and fusing image features and word embedding vectors to obtain multimodal features, cross-modal feature alignment and interaction are achieved, leveraging the guiding and enhancing role of textual information in the visual detection task.

Description

Technical Field

[0001] This disclosure relates to the field of aero-engine technology, and in particular to a method, apparatus, equipment, medium, and product for detecting blade damage. Background Technology

[0002] The aircraft engine is the core power unit of an aircraft. Its internal blades operate continuously under extreme conditions of high temperature, high pressure, and high speed, making them one of the most critical and vulnerable components of the engine. The health of the blades directly affects the engine's thrust output, fuel efficiency, and overall safety. Any minor damage, such as cracks or ablation, if not detected in time, can lead to serious consequences and threaten flight safety.

[0003] Borehole inspection is a commonly used method for detecting damage to aero-engine blades. It involves inserting an endoscope probe into the engine to obtain high-resolution images of the blade surface without disassembling the engine, thus determining the presence and extent of damage. However, this method has a high false negative rate; minute cracks and early ablation are easily overlooked by the naked eye. Furthermore, it is inefficient, often requiring several hours to inspect a single engine and demanding highly skilled technicians. Summary of the Invention

[0004] In view of the above problems, this disclosure is made to provide a method, apparatus, equipment, medium and product for detecting blade damage.

[0005] According to one aspect of this disclosure, a method for detecting leaf damage is provided, comprising:

[0006] Acquire images of the leaves to be detected and descriptive text; the descriptive text describes the damage to the leaves. The leaf images are processed using a pre-trained large visual model to obtain image features; The descriptive text is converted into word embedding vectors by using a large visual model. The image features and word embedding vectors are then fused to obtain multimodal features. Based on large visual models and multimodal features, the damage types in leaf images are identified, and the damage locations in leaf images are confirmed. Based on the damage type and location, damage contour information in the leaf image is obtained using a large visual model.

[0007] The beneficial effects of this scheme are as follows: Using a large visual model as the backbone of feature extraction, image features and word embedding vectors are fused to obtain multimodal features, achieving cross-modal feature alignment and interaction, and leveraging the guiding and enhancing role of textual information in visual detection tasks; using image features and word embedding vectors as bimodal inputs to the large visual model facilitates accurate localization and pixel-level segmentation of leaf damage, fully integrating visual features and domain prior knowledge, which helps improve the accuracy and generalization ability of leaf damage detection; through a cascaded strategy of detecting damage first and then segmenting the contour, pixel-level accurate segmentation of damage is achieved while ensuring detection efficiency.

[0008] Furthermore, according to one aspect of the leaf damage detection method disclosed herein, a pre-trained large visual model is used to process leaf images to obtain image features, including: The leaf image is input into a pre-trained large-scale visual model, which then processes the leaf image as follows: The leaf image is divided into multiple image blocks, and the built-in Transformer encoder analyzes the relationship between adjacent image blocks to generate a global feature map. Multi-scale feature maps are extracted from different levels of the Transformer encoder, and unified image features are obtained based on the multi-scale feature maps.

[0009] In one or more embodiments, the beneficial effects of this solution are as follows: by extracting multi-scale feature maps through the Transformer encoder, and fusing dual-scale features of large and small damage (such as large ablation and fine cracks), the detection capability of damage targets of different sizes can be improved, and a unified image feature representation can be obtained.

[0010] Furthermore, according to one aspect of the leaf damage detection method of this disclosure, based on a large visual model and multimodal features, the damage type in a leaf image is identified, including: Based on the large visual model and multimodal features, the similarity between each preset damage type and the word embedding vector is calculated according to the preset damage type. Based on the comparison of similarity, the damage type in the leaf image was identified.

[0011] In one or more embodiments, the beneficial effects of this solution are: by comparing similarity, damage type identification of open vocabulary is achieved, new types of damage can be identified without retraining, and the model iteration cost is significantly reduced.

[0012] Furthermore, according to one aspect of the leaf damage detection method of this disclosure, the location of damage in a leaf image is identified, including: Based on the visual large model and the damage type in the leaf image, a bounding box surrounding the damage is generated. Based on the size and coordinate information of the bounding box, the location of damage in the blade image is identified.

[0013] In one or more embodiments, the beneficial effect of this solution is that spatial localization of damage can be achieved through relevant information of the bounding box.

[0014] Furthermore, according to one aspect of the leaf damage detection method disclosed herein, based on the damage type and location, damage contour information in the leaf image is obtained using a large visual model, including: Based on the type and location of the damage, the local area of ​​damage is identified; By using a large visual model, pixel-level masking segmentation is performed in the local damaged area to generate damage contour information in the blade image.

[0015] In one or more embodiments, the beneficial effects of this solution are as follows: based on the relevant information of the bounding box, a large visual model (such as an instance segmentation task head based on the DINOv3 backbone) is called in the local area of ​​the damage to perform pixel-level mask segmentation of the damage and obtain accurate damage contour information, such as the specific length, area and direction of the crack, as a reference for whether the blade needs to be replaced.

[0016] Furthermore, according to one aspect of the blade damage detection method disclosed herein, the large visual model is trained end-to-end using a total loss function, which includes: Weighted cross-entropy category loss function is used to balance the distribution of damage types; Bounding box regression loss function is used to improve the localization accuracy of the bounding box; The No-object loss function is used to suppress false detections.

[0017] In one or more embodiments, the beneficial effects of this solution are as follows: a labeled, smoothed weighted cross-entropy category loss function is used to handle the classification problems of uneven damage category distribution and ambiguous damage; a bounding box regression loss function combining L1 loss and GIoU loss is used to optimize the localization accuracy of the bounding box; and a No-object loss function is used to suppress false detections, ensuring that the large visual model outputs only true damage results in complex backgrounds. This effectively solves the problems of difficult identification of small target damage, uneven category distribution, and false detection, improving the accuracy and robustness of detection, and providing effective technical support for intelligent detection of damage to aero-engine blades.

[0018] According to another aspect of this disclosure, a blade damage detection device is provided, comprising: The acquisition module is used to acquire images and descriptive text of the leaf to be detected; the descriptive text is used to describe the damage condition of the leaf. The image processing module is used to process leaf images using a pre-trained large visual model to obtain image features; The fusion module is used to convert descriptive text into word embedding vectors through a large visual model, and to fuse image features and word embedding vectors to obtain multimodal features; The damage detection module is used to identify the type of damage in leaf images and to confirm the location of damage in leaf images based on large visual models and multimodal features. The damage segmentation module is used to obtain damage contour information in the blade image based on the damage type and location using a large visual model.

[0019] According to another aspect of this disclosure, a computer device is provided, including a memory, a processor, and a computer program stored in the memory, the processor executing the computer program to implement the method of one aspect above.

[0020] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method of one aspect above.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method of the above-described aspect.

[0022] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0023] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0024] Figure 1 This is a system architecture diagram illustrating a blade damage detection method according to an embodiment of the present disclosure.

[0025] Figure 2 This diagram illustrates the framework for constructing a large visual model of DINOv3 according to an embodiment of this disclosure.

[0026] Figure 3 This is a flowchart illustrating a blade damage detection method according to an embodiment of the present disclosure.

[0027] Figure 4This is a schematic diagram of the structure of a blade damage detection device according to an embodiment of the present disclosure.

[0028] Figure 5 This is a schematic diagram illustrating the structure of a computer device according to an embodiment of the present disclosure.

[0029] Figure 6 This is a schematic diagram illustrating a computer program product according to an embodiment of the present disclosure. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0031] The aircraft engine is the core power unit of an aircraft. Its internal blades operate continuously under extreme conditions of high temperature, high pressure, and high speed, making them one of the most critical and vulnerable components of the engine. The health of the blades directly affects the engine's thrust output, fuel efficiency, and overall safety. Any minor damage, such as cracks or ablation, if not detected in time, can lead to serious consequences and threaten flight safety.

[0032] Borehole inspection is a commonly used method for detecting damage to aero-engine blades. It involves inserting an endoscope probe into the engine to obtain high-resolution images of the blade surface without disassembling the engine, thereby determining whether damage exists and its extent. However, this method has the following problems: (1) High rate of missed detection, damage such as fine cracks and early ablation is easily overlooked by the human eye; (2) The testing efficiency is low. Testing a single engine often takes several hours and requires highly skilled technicians to operate. (3) It is highly subjective, and different testing personnel may have different judgments on the same damage; fourth, the difficulty of testing is significantly increased in complex field environments.

[0033] These issues have limited the widespread application of borescope inspection technology in the "digital maintenance system" of next-generation fighter jets. However, with the rapid development of artificial intelligence and computer vision technologies, deep learning-based image recognition methods have provided a new technological path for automatic blade damage detection. Automatic detection methods are mainly divided into two categories: I. Detection methods based on traditional convolutional neural networks identify damaged areas in blade images by training classification or object detection models such as YOLO and Faster R-CNN. These methods have achieved good results on specific datasets, but their generalization ability is limited. When facing new types of damage or different engine models, a large amount of data needs to be re-labeled and the model needs to be retrained. Second, semantic segmentation-based methods can locate damaged areas at the pixel level, but they still have shortcomings in accurately measuring damage size and comprehensively determining damage type.

[0034] Furthermore, both of the above methods rely solely on visual information and fail to fully utilize the prior knowledge of the inspectors and textual information such as descriptions of the working conditions.

[0035] In recent years, breakthroughs have been made in visual large models and multimodal large models. Pre-trained large models such as DINOv3, SAM, and CLIP have demonstrated powerful visual feature extraction and cross-modal understanding capabilities, providing new ideas for solving the above problems.

[0036] The above description, with reference to the accompanying drawings, illustrates a blade damage detection method, apparatus, device, medium, and product according to embodiments of the present disclosure. Based on the DINOv3 visual large model as the feature extraction backbone, a visual-language multimodal fusion module is constructed. Using blade images and operational condition description text as bimodal inputs, the module achieves precise localization and pixel-level segmentation of blade damage through detection and segmentation task heads. It fully integrates visual features and domain prior knowledge, enabling accurate detection and segmentation of typical blade damage such as cracks, ablation, and notches. This effectively improves the accuracy, generalization ability, and engineering practicality of blade damage detection.

[0037] To facilitate understanding of this embodiment, a detailed description of the blade damage detection method disclosed in this disclosure is provided first. The execution entity of the blade damage detection method provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, vehicle-mounted device, wearable device, etc. In some possible implementations, the blade damage detection method can be implemented by a processor calling computer-readable instructions stored in memory.

[0038] like Figure 1 The diagram shown is a system architecture diagram of the blade damage detection method provided in this embodiment of the present disclosure, including: an image feature extraction module 1, a multimodal feature fusion module 2, and a damage detection and segmentation module 3. The following is a detailed description of each module: Image Feature Extraction Module 1: Based on a large visual model (such as DINOv3), the ViT self-supervised architecture is used to extract deep semantic features from borehole images. For example... Figure 2The diagram shown illustrates the framework for building the DINOv3 visual large model. The specific implementation process is as follows: S1: Image block embedding.

[0039] A borehole image (blade image) with dimensions W×H×C is divided into P×P image patches. Each patch is mapped to an N×C dimensional feature vector through a linear embedding layer, where N = WH / (P×P) is the number of patches. In this embodiment, the input image size is 224×224×3, the patch size is 14×14, and the number of patches N = 256. The embedding process can be represented as follows:

[0040] in, Represents a borehole image. This indicates an image segmentation operation. It is represented as a linear mapping layer.

[0041] S2: Transformer encoding.

[0042] The patch sequence is input into a Transformer encoder composed of multiple blocks. The Transformer encoder analyzes the appearance of each patch and its relationship with surrounding patches; for example, if a dark patch is next to a light-colored patch, it might be a crack edge. Each block contains a multi-head attention layer, a multilayer perceptron (MLP), and a layer normalization layer, and its structure can be represented as follows:

[0043]

[0044] in, The input features (i.e., patches) The representation uses layer regularization, MSA (Multi-head Self-Attention) mechanism, and MLP (Multi-layer Perceptron). The Transformer encoder achieves deep feature extraction through L× iterative stacking, and finally outputs an N×D global feature map after layer regularization.

[0045] S3: Extract multi-scale feature maps.

[0046] P3 and P4 multi-scale feature maps are extracted from different levels of the Transformer encoder. A dual-scale fusion strategy is used to enhance the visual large model's ability to detect damage targets of different sizes, resulting in unified image features.

[0047] Multi-scale feature maps can simultaneously preserve features that are viewed coarsely (such as large ablation) and features that are viewed finely (such as hair-thin cracks).

[0048] Multimodal Feature Fusion Module 2: This module enables deep interaction between image and text features through the text encoder and visual-language multimodal fusion module within the large visual model. Specifically, it includes: S1: Text encoding.

[0049] The user-input descriptive text (e.g., "There is a notch on the engine blade") is converted into word embedding vectors by a text encoder. The text encoder extracts semantic representations of key damage types (e.g., notch, crack, burn, pit, etc.) and generates word embedding vectors T∈R^(K×D), where K is the number of words and D is the embedding dimension.

[0050] S2: Cross-modal feature alignment.

[0051] The image features V output by the large visual model and the word embedding vector T are input into the visual language multimodal fusion module. Through a cross-modal attention mechanism, deep interaction between visual features and textual semantics is achieved to generate an image-aware embedded representation (multimodal features). This enables the large visual model to understand text queries and locate the corresponding damaged areas.

[0052] This embodiment also supports user-defined damage type vocabulary, expanding the detection category range of the large-scale visual model through a dynamic vocabulary injection mechanism. Users can add new damage type descriptions according to actual detection needs, and the large-scale visual model can recognize the newly added damage types without retraining, thus realizing open vocabulary detection capabilities.

[0053] Damage detection and segmentation module 3: Based on the fused multimodal features, a cascaded strategy of first detecting damage and then segmenting the contour is adopted. Damage category recognition and spatial localization are achieved through Text Contrastive Head and Box Head, and pixel-level masks of damage are obtained by segmenting in local areas.

[0054] 1. Damage Detection Task. The detection task is implemented using a two-branch structure: Text Contrastive Head: Calculates the contrastive similarity between image features and word embedding vectors to achieve damage category identification for open vocabulary. For each target query q and word embedding vector t, a contrastive score is calculated. :

[0055] Here, the target query q represents a preset damage type, such as crack, ablation, etc. This represents the L2 norm (vector magnitude).

[0056] Box Head: Based on the regression of the bounding box coordinates (x, y, w, h) of the damage region using fused features, spatial localization of the damage is achieved.

[0057] Target embedding: Fuse text comparison results and bounding box features (such as size information and coordinate information) to generate target embedding vectors, and output damage category, confidence level and bounding box location.

[0058] 2. Damage segmentation task.

[0059] Based on the bounding box results output by the damage detection task, the instance segmentation task head based on the DINOv3 backbone is called within the local damage area to perform pixel-level mask segmentation on the damage area and obtain the precise contour information of the damage.

[0060] The large-scale visual model described in this embodiment can employ pre-trained large-scale visual models such as DINOv3 and SAM image encoders. The multimodal fusion module can be implemented using methods such as cross-attention and feature concatenation. The detection framework can employ open-vocabulary detection models such as Grounding DINO and YOLO-World. Taking DINOv3 as an example, the following is the model construction and training process: (1) Dataset construction.

[0061] Data acquisition was conducted in a real-world scenario using a borescope, focusing on four typical types of blade damage: notches (material loss at the blade edge), ablation (surface material detachment due to high temperatures), pits (dents formed by foreign object impact), and cracks (linear surface cracking). A total of 1315 valid images were obtained, including 332 images of notches, 328 of ablation, 325 of pits, and 330 of cracks, with the ratio of images for the four damage types approximately 1:1:1:1. The LabelImg annotation tool was used to annotate the damage regions according to the YOLO object detection format, ensuring that the bounding boxes covered ≥95% of the damage. The images were divided into training and test sets at a 4:1 ratio.

[0062] (2) Data augmentation.

[0063] To improve the model's robustness and generalization ability in complex real-world environments, several data augmentation strategies are employed: flipping the image horizontally with a 50% probability; performing random rotation (-15° to +15°) with a 30% probability; perturbing brightness and contrast with a 30% probability; applying affine transformation with a 20% probability; performing cutout random occlusion (32×32 pixels) with a 20% probability; applying Gaussian blur with a 20% probability; adding Gaussian noise with a 20% probability; and performing HSV color perturbation with a 20% probability.

[0064] (3) Training strategies.

[0065] Model training is divided into two stages: In the first stage, the backbone is frozen, and only the detection head is trained. The official self-supervised pre-trained weights DINOv2 ViT-B / 14 are loaded as teacher weights, the backbone network parameters are fixed, and the target query capability of the detection head is quickly converged, enabling it to predict the damage type and location with preliminary accuracy.

[0066] In the second stage, the last three layers of the backbone network are unfrozen and trained together with the detection head. This allows the high-level semantic features of the backbone network to interact more deeply with the detection head, further refining the high-level semantic features and improving the ability to identify complex damage.

[0067] (4) Optimization strategy.

[0068] Two dynamic learning rate adjustment strategies, ReduceLROnPlateau and StepLR, are employed. When the mAP on the validation set fails to improve for five consecutive epochs, the learning rate is reduced to 0.1 times its original value; every 10 epochs, the learning rate is reduced to 0.9 times its original value. An early stopping mechanism is introduced: if the loss value on the validation set does not decrease for 10 consecutive epochs, training is automatically stopped to avoid overfitting.

[0069] (5) Loss function design.

[0070] Considering the unique characteristics of damage detection in borehole images of aero-engine blades, it is necessary to simultaneously address four core issues: difficulty in identifying small target damage, uneven category distribution, accuracy of bounding box localization, and suppression of false detections. Therefore, this embodiment designs a multi-branch weighted total loss function, specifically including: ① Employ a weighted cross-entropy class loss function with label smoothing (smoothing=0.1). :

[0071] in, Indicates category weight, This represents the label smoothing cross-entropy. Label smoothing transforms "hard labels" (e.g., [1,0,0,0] for category 1) into "soft labels" (e.g., [0.9,0.033,0.033,0.033]), preserving judgment tolerance space for the model, improving the classification robustness to fuzzy damage, and balancing the distribution of damage types. For example, a weight of 2.0 is assigned to difficult-to-detect damage such as pits and cracks, while the weight of categories such as notches and ablation is set to 1.0.

[0072] ② The bounding box regression loss function, composed of L1 loss and GIoU loss:

[0073]

[0074] in, This indicates the L1 loss for bounding box coordinate regression. This represents the GIoU loss (Generalized Cross-Union Loss). To match the target number of queries, To predict the coordinates of the bounding box, The coordinates of the actual bounding box. The loss applies a linear penalty to the difference in bounding box coordinates. The GIoU loss evaluates the overlap between the predicted and ground truth bounding boxes by calculating the ratio of the minimum bounding box area of ​​the predicted bounding box to the area of ​​their union. The two work together to optimize the localization accuracy of the bounding boxes.

[0075] ③ No-object loss function: This function uses cross-entropy loss to optimize target queries that do not match real damage. For example, in this embodiment, the model generates 8 target queries, but the actual number of damages in a single leaf image is usually ≤4. The No-object loss function suppresses false detections by applying a 0.5 times class loss weight to unmatched queries, ensuring that the model only outputs real damage results in complex backgrounds.

[0076] Therefore, the expression for the total loss function is:

[0077] in, The weights represent the weights of the weighted cross-entropy class loss function. The weights representing the L1 loss of the bounding box coordinates, The weights represent the GIoU loss of bounding box regression. This represents the weights of the No-object loss function.

[0078] (6) Hungarian match.

[0079] During model training, the Hungarian matching algorithm is used to establish the optimal correspondence between predicted queries and real damage annotations. The matching cost is calculated by weighting the cross-entropy class loss function, L1 loss, and GIoU loss according to their weights. By finding a one-to-one matching scheme with the minimum total loss, it is ensured that each real damage corresponds to a unique predicted query, generating a unique optimal bounding box for each damage, avoiding duplicate detection and missed detection problems, and realizing end-to-end training of the model.

[0080] In this preferred embodiment, to verify the effectiveness of the proposed solution, damage detection and segmentation experiments were conducted based on a real dataset of borehole images of aero-engine blades, as detailed below: (1) Loss convergence.

[0081] In the first phase (the first 30 rounds), the training loss rapidly decreased from nearly 4.0 to approximately 2.0, while the validation loss simultaneously decreased from approximately 2.8 to around 2.0, with both showing consistent downward trends and minimal differences. In the second phase (rounds 30-200), the training loss continued to decrease slowly from 2.0 and stabilized in the 1.5-1.8 range, while the validation loss also gradually decreased from 2.0 to around 1.5. The segmentation loss converged to approximately 1.0-1.2, with no significant overfitting, indicating a good match between the model structure and the data size.

[0082] (2) Evaluation of detection performance.

[0083] The detection results for various types of damage are shown in Table 1: Table 1 Test Results

[0084] As shown in Table 1, the mAP for notch damage is as high as 0.9912, and both Precision and Recall are close to 1.0, indicating that the model has almost no error in identifying notch damage with regular edges and high contrast. The mAP for ablation damage is 0.9805. Due to the large area of ​​the ablation region and obvious surface texture features, the model can efficiently capture its "regional damage" characteristics. The mAPs for pits and cracks, as typical small-target damage, reach 0.8528 and 0.8755, respectively, and the Recall for cracks is as high as 0.9215, indicating that the model can effectively detect more than 92% of crack areas. The overall mAP reaches 0.9250, and the F1 value reaches 0.9422, which meets the requirements of engineering inspection.

[0085] This embodiment uses the DINOv3 visual large-scale model as the feature extraction backbone, which has powerful visual semantic understanding capabilities and cross-domain transfer capabilities. It constructs a visual-language multimodal fusion module, which fully utilizes the domain prior knowledge in the text condition description to guide damage detection, effectively improving the model's ability to identify typical damages such as notches, ablation, pits, and cracks. It adopts Text Contrastive Head to realize open vocabulary detection, which can identify new damage categories without retraining, significantly reducing the model iteration cost. Through the cascaded strategy of detection before segmentation, pixel-level accurate segmentation of damage is achieved while ensuring detection efficiency. The P3 / P4 dual-scale feature fusion strategy enhances the model's ability to detect damage of different sizes. The multi-branch weighted loss function effectively solves the problems of difficult identification of small target damage, uneven category distribution, and false detection, improving the accuracy and robustness of detection, and providing effective technical support for intelligent detection of damage in aero-engine blades.

[0086] Based on the above embodiments, this embodiment also provides a method for detecting leaf damage, such as... Figure 3 The flowchart shown is for a leaf damage detection method, including steps S301-S305: S301: Obtain the image and descriptive text of the leaf to be detected.

[0087] The descriptive text is used to describe the damage to the leaf, such as the engineer's words / writing: "There is a crack at the base of the left leaf."

[0088] S302: Use a pre-trained large visual model to process the leaf image and obtain image features.

[0089] S303: The descriptive text is converted into word embedding vectors through a large visual model, and the image features and word embedding vectors are fused to obtain multimodal features.

[0090] S304: Based on large visual models and multimodal features, identify the damage type in leaf images and confirm the location of damage in leaf images.

[0091] S305: Based on the damage type and location, damage contour information in the blade image is obtained using a large visual model.

[0092] In one or more embodiments, a pre-trained large visual model is used to process the leaf image to obtain image features, including: The leaf image is input into a pre-trained large visual model, which processes the leaf image as follows: the leaf image is divided into multiple image blocks, and the relationship between adjacent image blocks is analyzed by the built-in Transformer encoder to generate a global feature map; multi-scale feature maps are extracted from different levels of the Transformer encoder, and a unified image feature is obtained based on the multi-scale feature maps.

[0093] In one or more embodiments, identifying damage types in leaf images based on a large visual model and multimodal features includes: calculating the contrast similarity between each preset damage type and a word embedding vector according to preset damage types, based on the large visual model and multimodal features; and confirming the damage type in the leaf image based on the contrast similarity. Specifically: Text Contrastive Head: Calculates the contrastive similarity between image features and word embedding vectors to achieve damage category identification for open vocabulary. For each target query q and word embedding vector t, a contrastive score is calculated. :

[0094] Here, the target query q represents the preset damage type, such as crack, ablation, etc.

[0095] In one or more embodiments, identifying the damage location in a blade image includes: generating a bounding box surrounding the damage based on a large visual model and the damage type in the blade image; and identifying the damage location in the blade image based on the size and coordinate information of the bounding box. Specifically: Box Head: Based on the regression of the bounding box coordinates (x, y, w, h) of the damage region using fused features, spatial localization of the damage is achieved.

[0096] Target embedding: Fuse text comparison results and bounding box features (such as size information and coordinate information) to generate target embedding vectors, and output damage category, confidence level and bounding box location.

[0097] In one or more embodiments, damage contour information in a leaf image is obtained using a large visual model based on the damage type and damage location, including: identifying the local damage region based on the damage type and damage location; and performing pixel-level mask segmentation in the local damage region using a large visual model to generate damage contour information in the leaf image.

[0098] In one or more embodiments, the large visual model is trained end-to-end using a total loss function, which includes: a weighted cross-entropy category loss function to balance the distribution of damage types; a bounding box regression loss function to improve the localization accuracy of bounding boxes; and a No-object loss function to suppress false detections.

[0099] According to another aspect of the embodiments of this disclosure, a blade damage detection device is provided, such as... Figure 4 As shown, the device includes: The acquisition module 401 is used to acquire images of the leaf to be detected and descriptive text; wherein, the descriptive text is used to describe the damage condition of the leaf. Image processing module 402 is used to process leaf images using a pre-trained large visual model to obtain image features; The fusion module 403 is used to convert the descriptive text into word embedding vectors through a large visual model, and to fuse image features and word embedding vectors to obtain multimodal features; The damage detection module 404 is used to identify the damage type in the leaf image and confirm the location of the damage in the leaf image based on the visual large model and multimodal features. The damage segmentation module 405 is used to obtain damage contour information in the blade image based on the damage type and damage location using a large visual model.

[0100] The blade damage detection device and the blade damage detection method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0101] This disclosure also provides a computer device for performing the above-described blade damage detection method. Please refer to... Figure 5 It illustrates a schematic diagram of a computer device provided by some embodiments of this disclosure. For example... Figure 5 As shown, the computer device 5 includes: a processor 500, a memory 501, a bus 502, and a communication interface 503. The processor 500, the communication interface 503, and the memory 501 are connected via the bus 502. The memory 501 stores a computer program that can run on the processor 500. When the processor 500 runs the computer program, it executes the blade damage detection method provided in any of the foregoing embodiments of this disclosure.

[0102] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this device network element and at least one other network element is achieved through at least one communication interface 503 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc.

[0103] Bus 502 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The memory 501 is used to store programs. After receiving an execution instruction, the processor 500 executes the program. The blade damage detection method disclosed in any of the foregoing embodiments of this disclosure can be applied to the processor 500, or implemented by the processor 500.

[0104] The processor 500 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 500 or by instructions in software form. The processor 500 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPTA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 501. The processor 500 reads the information in memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0105] The computer equipment provided in this disclosure and the blade damage detection method provided in this disclosure are based on the same inventive concept and have the same beneficial effects as the methods they employ, operate, or implement.

[0106] This disclosure also provides a computer-readable storage medium corresponding to the blade damage detection method provided in the foregoing embodiments. The computer-readable storage medium is an optical disc, on which a computer program (i.e., a computer program product) is stored. When the computer program is run by a processor, it executes the blade damage detection method provided in any of the foregoing embodiments.

[0107] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here.

[0108] The computer-readable storage medium provided in the above embodiments of this disclosure and the blade damage detection method provided in the embodiments of this disclosure are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0109] This disclosure also provides a computer program product; please refer to [reference needed]. Figure 6 The computer program product 600 carries program code, namely computer program 601. The instructions included in the computer program 601 can be used to execute the steps of the blade damage detection method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0110] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0111] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0112] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0113] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0114] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0115] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0116] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0117] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for detecting leaf damage, characterized in that, include: Acquire an image of the leaf to be detected and a descriptive text; wherein the descriptive text is used to describe the damage condition of the leaf; The leaf images are processed using a pre-trained large visual model to obtain image features; The visual big model is used to convert the descriptive text into word embedding vectors, and the image features and word embedding vectors are fused to obtain multimodal features; Based on the aforementioned large visual model and the aforementioned multimodal features, the damage type in the leaf image is identified, and the location of the damage in the leaf image is confirmed. Based on the damage type and location, the damage contour information in the leaf image is obtained using the large visual model.

2. The leaf damage detection method as described in claim 1, characterized in that, The leaf image is processed using a pre-trained large visual model to obtain image features, including: The leaf image is input into a pre-trained large-scale visual model, which then performs the following processing on the leaf image: The leaf image is divided into multiple image blocks, and the built-in Transformer encoder analyzes the relationship between adjacent image blocks to generate a global feature map. Multi-scale feature maps are extracted from different levels of the Transformer encoder, and unified image features are obtained based on the multi-scale feature maps.

3. The leaf damage detection method as described in claim 1, characterized in that, Based on the aforementioned large visual model and the aforementioned multimodal features, the damage type in the leaf image is identified, including: Based on the visual large model and the multimodal features, the similarity between each preset damage type and the word embedding vector is calculated according to the preset damage type. Based on the comparison similarity, the damage type in the leaf image is confirmed.

4. The leaf damage detection method as described in claim 1, characterized in that, Identifying the location of damage in the blade image includes: Based on the large visual model and the damage type in the leaf image, a bounding box surrounding the damage is generated. Based on the size and coordinate information of the bounding box, the location of damage in the blade image is confirmed.

5. The leaf damage detection method as described in claim 1, characterized in that, Based on the damage type and location, the damage contour information in the leaf image is obtained using the large visual model, including: Based on the damage type and location, the local area of ​​damage was identified; The large visual model is used to perform pixel-level masking segmentation within the damaged local area to generate damage contour information in the blade image.

6. The blade damage detection method as described in claim 1, characterized in that, The large visual model is trained end-to-end using a total loss function, which includes: Weighted cross-entropy category loss function is used to balance the distribution of damage types; Bounding box regression loss function is used to improve the localization accuracy of the bounding box; The No-object loss function is used to suppress false detections.

7. A blade damage detection device, characterized in that, include: The acquisition module is used to acquire images of the leaves to be detected and descriptive text; wherein, the descriptive text is used to describe the damage condition of the leaves; The image processing module is used to process the leaf image using a pre-trained large visual model to obtain image features; The fusion module is used to convert the descriptive text into word embedding vectors through the visual large model, and to fuse the image features and the word embedding vectors to obtain multimodal features; The damage detection module is used to identify the damage type in the leaf image and confirm the damage location in the leaf image based on the visual large model and the multimodal features. The damage segmentation module is used to obtain damage contour information in the leaf image based on the damage type and damage location using the large visual model.

8. A computer embedded device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 6.

Images