Power generation equipment defect identification method and system based on open set and multi-modal model

Abstract

The application provides a power generation equipment defect identification method and system based on an open set and a multi-modal model, and the method comprises the following steps: step 1: an optimized open set target detection model is used to realize open set detection and positioning of equipment, and the equipment category label and the corresponding boundary box coordinates are obtained; step 2: the equipment boundary box output in step 1 is subjected to coordinate correction and image cropping processing, and is sorted to generate an image of an equipment to be identified; step 3: the image of the equipment to be identified and prompt words customized for defects of the power generation equipment are jointly input into a Qwen2.5-VL multi-modal large model to perform defect type identification, positioning and severity evaluation; and step 4: the boundary box coordinates output in step 1 are fused with the defect semantic information output in step 3 to generate a global defect detection report. The application improves the adaptability, precision and efficiency of detection, reduces the system operation and maintenance cost, and provides reliable technical support for safe operation and maintenance of the power generation equipment.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, specifically to a method and system for identifying defects in power generation equipment based on open sets and multimodal models. Background Technology

[0002] Power generation equipment is the core carrier for the safe and stable operation of the power system, and its operating status directly determines the reliability of power supply. Traditional power generation equipment inspection relies on manual inspection, which has the following drawbacks: (1) Low efficiency: It requires manual inspection of each piece of equipment. When facing complex scenarios such as large wind farms and thermal power plants, the inspection cycle can take several days; (2) High risk: Some equipment (such as wind turbine towers and high-voltage transformers) are located in high-altitude or high-voltage environments, and manual inspection can easily lead to safety accidents; (3) Limited accuracy: It relies on manual experience to judge defects (such as blade cracks and bolt corrosion), which can easily lead to missed or false detections.

[0003] Existing machine vision detection solutions are mostly based on closed set target detection models (such as the traditional YOLO series and Faster R-CNN), which have significant limitations: (1) poor adaptability: they can only detect devices of predefined categories and cannot cope with new types of equipment in power generation scenarios (such as different models of photovoltaic panels and new turbine components); (2) weak defect recognition capability: most models can only achieve device localization and cannot further identify defects such as cracks, deformations, and corrosion on the surface of the device. They need to deploy additional dedicated defect detection models, resulting in high system complexity; (3) dependence on labeled data: closed set models require a large number of labeled images of power generation equipment for training, while defect samples are scarce in power generation scenarios, resulting in poor model generalization ability.

[0004] In recent years, open-set object detection models and multimodal large models (such as Qwen2.5-VL) have offered technical possibilities for solving the above problems: open-set object detection models support detection in three open scenarios—text-provided, visual-provided, and unprovided—and can flexibly adapt to diverse equipment in power generation scenarios; Qwen2.5-VL has cross-modal image-text understanding capabilities and can achieve defect identification based on a small number of samples. However, currently there is no technical solution to combine the two to build an end-to-end power generation equipment detection system encompassing "equipment localization-image cropping-defect identification," thus failing to fully leverage the technical advantages of both.

[0005] In view of this, it is necessary to improve the existing technology to solve the above problems. Summary of the Invention

[0006] This invention provides a method and system for identifying defects in power generation equipment based on open sets and multimodal models, aiming to solve the problems of "poor adaptability, low defect identification accuracy, and high system complexity" in existing power generation equipment detection schemes.

[0007] The model of this invention is mainly a method and system for detecting equipment defects in power generation scenarios that integrates the open set target detection capability of the open set target detection model with the multimodal defect recognition capability of Qwen2.5-VL. It is applicable to the real-time detection and defect anomaly recognition of core equipment (such as wind turbine blades, steam turbines, photovoltaic inverters, transformers, etc.) in scenarios such as thermal power generation, wind power generation, and photovoltaic power generation.

[0008] The core technology of this invention lies in the following: First, an open-set target detection model optimized for power generation scenarios is used to achieve open-set detection and localization of equipment. It innovatively applies a reparameterizable region-text alignment module to achieve precise alignment between equipment features and text prompts. A semantically activated visual cue encoder module supports visual cue retrieval, and a lazy region-cue comparison mechanism enables equipment recognition in cue-free scenarios. This supports the inspection and detection of power generation equipment of different types, including wind, solar, hydro, and thermal power. Then, based on the detection results, the equipment region is cropped and input along with specially designed defect identification prompts into the Qwen2.5-VL multimodal large model to complete defect type identification, localization, and severity assessment. Qwen2.5-VL, after specific fine-tuning with a power generation equipment defect knowledge base, possesses powerful cross-modal understanding and structured output capabilities. Compared to traditional solutions, this invention significantly improves the adaptability, accuracy, and efficiency of detection, while reducing system complexity and maintenance costs, providing reliable technical support for the safe operation and maintenance of power generation equipment.

[0009] The technical solution of this invention can be summarized as follows: Step A: Building the Dataset for Power Generation Scenarios Collect equipment images from various power generation scenarios (wind power, thermal power, and photovoltaic), covering different environments (such as rainy days, sandstorms, and nighttime); label equipment categories (such as wind turbine blades, steam turbine rotors, photovoltaic panels, and transformer bushings), equipment bounding box coordinates, and defect samples (such as cracks, corrosion, loose bolts, and damaged insulators); and build a dedicated vocabulary for power generation equipment, containing 200+ core equipment categories for open set detection adaptation.

[0010] Step B: Deployment and optimization of open set target detection model and power equipment detection model Based on the open set target detection model, customized adjustments are made for power generation scenarios to achieve real-time detection and localization of equipment; the bounding box coordinates (x1, y1, x2, y2) and equipment category are output for subsequent image cropping.

[0011] Step C: Crop the device bounding box coordinate image Based on the bounding box coordinates output by the open set target detection model, the device bounding box coordinates are extracted from the original image using an image cropping algorithm (such as the crop function in OpenCV). The bounding box coordinates are preprocessed: resized to the Qwen2.5-VL fit size and normalized (pixel values ​​are normalized to [0,1]) to eliminate the influence of image size differences on defect recognition.

[0012] Step D: Deployment and Inference of the Qwen2.5-VL Defect Identification Model Design a dedicated prompt for defects in power generation equipment, formatted as follows: "Please analyze whether there are any open cabinet doors in the provided image. An open cabinet door means: the cabinet door is at a significant angle to the cabinet body (not flush), the inside of the cabinet door or the interior space is visible, or the cabinet door is significantly deviated from its normal closed position. If at least one cabinet door in the image meets the above description, please return the result in JSON format: [{"door_open":true}]; if all cabinet doors in the image are closed (the cabinet door is flush with the cabinet body, without a significant angle) or there are no cabinet doors, please return: [{"door_open":false}]." Input the preprocessed bounding box coordinates and the dedicated prompt into Qwen2.5-VL, and the model outputs the defect identification results; perform post-processing on the identification results: filter defect predictions with a confidence level below 0.7, and merge overlapping defect bounding boxes.

[0013] Step E: Output and Visualization of Detection Results The device location results of the open set target detection model are integrated with the defect identification results of Qwen2.5-VL to generate a detection report (including device category, defect type, location, severity, and detection time); the device bounding box and defect annotations are overlaid on the original image to achieve a visual display.

[0014] Specifically, the technical solution of the present invention is as follows: A method for identifying defects in power generation equipment based on open sets and multimodal models includes the following steps: Step S1: Device Open Set Detection and Localization: An open set target detection model optimized for power generation scenarios is used to realize the open set detection and localization of devices, and to obtain the device category label and the corresponding bounding box coordinates; the open set target detection model supports device recognition under three paradigms: text prompts, visual prompts, and no prompts; Step S2: Refined processing of device images: The device bounding boxes output in step S1 are subjected to coordinate correction and image cropping, and sorted according to the priority of device categories to generate images of the devices to be identified that are adapted to the input requirements of the multimodal model. Step S3: Multimodal defect recognition and reasoning: Input the image of the equipment to be identified and the prompt words customized for defects in power generation equipment into the pre-fine-tuned Qwen2.5-VL multimodal large model to perform defect type identification, localization and severity assessment; Step S4: Result Fusion Output: The bounding box coordinates output in Step S1 are used as a spatial reference and fused with the defect semantic information output in Step S3 to generate a global defect detection report.

[0015] As a further improvement of the present invention, the open set target detection model in step S1 applies a reparameterizable region-text alignment module, the core mechanism of which includes: An initial text embedding P is generated using a pre-trained text encoder, and P is optimized using a lightweight auxiliary network to enhance the matching degree between text semantics and device visual features. After training, the parameters of the auxiliary network are reparameterized into the classification head of the detector through a parameter remapping function to form an inference model consistent with the original YOLO structure, so as to achieve visual-text feature alignment with zero inference overhead.

[0016] As a further improvement of the present invention, the reparameterization operation follows the formula: Among them, K T The original classification head convolution kernel; Dimension C represents the number of categories, and D represents the feature dimension. The result of the SwigluffN network processing text embedding P; R is the parameter remapping function, which converts the auxiliary network parameters into convolutional kernel form; * indicates a convolution operation. This is the reconstructed classification head convolutional kernel (used directly during inference, without needing to retain the auxiliary network).

[0017] As a further improvement of the present invention, the visual cue detection in step S1 is achieved through a semantic activation visual cue encoder module, and the implementation steps include: Extract common semantic features of devices from images using semantic branches; By activating branches to fuse input visual cue samples with multi-scale image features, cue perception weights are generated. By concatenating and fusing general semantic features with cue perception weights, a visual cue embedding is generated for accurate device positioning and classification.

[0018] As a further improvement of the present invention, the generation formula for the visual cue embedding P is: S represents the general semantic features (dimensions) output by the semantic branch. ), To activate the i-th group of cue perception weights in the branch output, A is the number of groups (A=8 in the power generation scenario to reduce computation), Concat() is the feature concatenation operation, and P is the final visual cue embedding.

[0019] As a further improvement of the present invention, the device identification in the unnoticed scenario in step S1 is achieved through a lazy region-notification comparison mechanism, the steps of which include: By utilizing learned general device features, a dedicated cue embedding is generated as a feature representation of the device abstract concept; During inference, only device anchors with a similarity to the dedicated cue embedding higher than a preset threshold δ are retained, and the formula is as follows: Where o is the candidate anchor point feature embedding (dimension D) generated by the open set object detection model. The transpose of the dedicated prompt embedding, δ is the filtering threshold, which ranges from 0.005 to 0.01; The filtered anchor points are compared with the category embeddings in the built-in vocabulary for power generation equipment, and the most matching category is output.

[0020] As a further improvement of the present invention, the coordinate correction and image cropping in step S2 specifically include: Expand the boundary frame outward by 5% to 10% in all directions to avoid missing equipment edge defects, with photovoltaic panels expanded by 5% and turbine bearings by 10%. When multiple devices are detected in a single image, they are sorted by device category priority, with wind turbine blades having higher priority than supports in wind power scenarios. The cropped image is resized to the optimal input size of Qwen2.5-VL while maintaining the aspect ratio.

[0021] As a further improvement of the present invention, the Qwen2.5-VL fine-tuning training in step S3 includes: based on a multimodal power inspection dataset containing power generation scene images, text descriptions and annotation information, the model is trained to have the ability to identify equipment types, detect defects and understand semantics by combining supervised fine-tuning of SFT with contrastive learning.

[0022] As a further improvement of the present invention, the prompt word design in step S3 is tailored to different device types, including customized JSON format return requirements, such as: For the main transformer, the prompt requires analyzing the building's exterior appearance and returning a Boolean value indicating whether the appearance is normal or not. For reactors or capacitors, the prompt requires analysis of whether there are bird nests or foreign objects present. For couplings and high-speed brake discs, the prompt requires a comparative analysis of two images to determine whether the protective shell is loose or has detached.

[0023] Based on the above method, this invention also proposes a power generation equipment defect identification system based on open sets and multimodal models, comprising: Equipment location module: used to execute the power generation equipment defect identification method based on open set and multimodal model described above, and output the equipment bounding box; Image preprocessing module: used to perform coordinate correction, sorting and size adaptation as described in the above methods; Multimodal recognition module: used to deploy the fine-tuned Qwen2.5-VL model to identify defects based on customized prompt words; Data fusion module: used to convert local defect identification results back to global coordinates of the equipment and output a report.

[0024] Beneficial effects: This invention proposes a method and system for defect identification of power generation equipment based on open sets and multimodal models. By integrating lightweight open set target detection technology with the Qwen2.5-VL multimodal model, it achieves flexible detection of various types of power generation equipment in multiple scenarios such as thermal power, wind power, and photovoltaic power. New equipment can be quickly adapted by simply updating the text prompts, without retraining the model. At the same time, it constructs an end-to-end automated process of "equipment detection - coordinate clipping - defect identification", which significantly improves detection efficiency and eliminates manual intervention. For fuzzy defects in complex environments such as cracks covered by oil and microcracks in photovoltaic systems, it achieves accurate identification with multimodal understanding capabilities, effectively reducing the risk of missed detection. The detection module supports real-time deployment at the edge, and Qwen2.5-VL can flexibly adapt to cloud or edge computing conditions, with flexible deployment methods. Through automatic defect level judgment and alarm mechanism, it can quickly respond to serious defects, buy time for equipment repair, and improve the operational stability of the power generation system. Moreover, it can seamlessly switch between multiple power generation scenarios without adjusting the core algorithm framework, which is highly versatile and greatly reduces the technical adaptation cost and equipment failure risk of operation and maintenance in multiple scenarios. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method steps of the present invention; Figure 2 This is a schematic diagram illustrating the alarm details when a defect occurs in the equipment box in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the alarm details when a defect occurs in the windmill brake disc in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the alarm details when the temperature of a photovoltaic panel is too high during infrared thermal imaging inspection, as described in this embodiment of the invention. Figure 5 This is a schematic diagram illustrating the alarm details when a defect occurs in a photovoltaic panel, as shown in this embodiment of the invention. Figure 6 This is a schematic diagram showing the alarm details when a defect occurs in the wind turbine blades in an embodiment of the present invention. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific implementation examples.

[0027] like Figure 1-6 As shown, this invention discloses a method and system for identifying defects in power generation equipment based on open sets and multimodal models. Leveraging its "single model supporting multiple open prompt mechanisms," it adapts to the detection needs of various types of equipment in power generation scenarios, such as wind turbine blades, steam turbines, photovoltaic panels, and transformers. It eliminates the need to retrain the model for new equipment types, enabling equipment localization through text prompts, visual prompts, or no prompts, thus solving the problem of poor adaptability of traditional closed-set detection models. For text-prompt detection scenarios in power generation equipment, a reparameterized region-text alignment module can be used to construct a dedicated text vocabulary for power generation equipment. The pre-trained text embedding of this vocabulary (based on a MobileCLIP-B(LT) encoder) is optimized through a lightweight auxiliary network (a feedforward network with a SwiGLU activation function) of this module, enhancing the alignment accuracy between visual features and text semantics of power generation equipment. Furthermore, the trained auxiliary network can be reparameterized to the detector classification head, achieving zero inference overhead and meeting the real-time inspection requirements of power generation scenarios. Specifically, this invention discloses a method for identifying defects in power generation equipment based on open sets and multimodal models, comprising the following steps: Step S1: Device Open Set Detection and Localization: An open set target detection model optimized for power generation scenarios is used to realize the open set detection and localization of devices, and to obtain the device category label and the corresponding bounding box coordinates; the open set target detection model supports device recognition under three paradigms: text prompts, visual prompts, and no prompts; Step S2: Refined processing of device images: The device bounding boxes output in step S1 are subjected to coordinate correction and image cropping, and sorted according to the priority of device categories to generate images of the devices to be identified that are adapted to the input requirements of the multimodal model. Step S3: Multimodal defect recognition and reasoning: Input the image of the equipment to be identified and the prompt words customized for defects in power generation equipment into the pre-fine-tuned Qwen2.5-VL multimodal large model to perform defect type identification, localization and severity assessment; Step S4: Result Fusion Output: The bounding box coordinates output in Step S1 are used as a spatial reference and fused with the defect semantic information output in Step S3 to generate a global defect detection report.

[0028] Furthermore, the open set object detection model in step S1 applies a reparameterizable region-text alignment module, the core mechanism of which includes: An initial text embedding P is generated using a pre-trained text encoder, and P is optimized using a lightweight auxiliary network to enhance the matching degree between text semantics and device visual features. After training, the parameters of the auxiliary network are reparameterized into the classification head of the detector through a parameter remapping function to form an inference model consistent with the original YOLO structure, so as to achieve visual-text feature alignment with zero inference overhead.

[0029] Furthermore, the reparameterization operation follows the formula: Among them, K T The original classification head convolution kernel; Dimension C represents the number of categories, and D represents the feature dimension. The result of the SwigluffN network processing text embedding P; R is the parameter remapping function, which converts the auxiliary network parameters into convolutional kernel form; * indicates a convolution operation. This is the reconstructed classification head convolutional kernel (used directly during inference, without needing to retain the auxiliary network).

[0030] Furthermore, in step S1, visual cue detection is achieved through a semantic activation visual cue encoder module, the implementation steps of which include: Extract common semantic features of devices from images using semantic branches; By activating branches to fuse input visual cue samples with multi-scale image features, cue perception weights are generated. By concatenating and fusing general semantic features with cue perception weights, a visual cue embedding is generated for accurate device positioning and classification.

[0031] Furthermore, the generation formula for the visual cue embedding P is: S represents the general semantic features (dimensions) output by the semantic branch. ), To activate the i-th group of cue perception weights in the branch output, A is the number of groups (A=8 in the power generation scenario to reduce computation), Concat() is the feature concatenation operation, and P is the final visual cue embedding.

[0032] Furthermore, in step S1, device identification in the unnoticed scenario is achieved through a lazy region-notification comparison mechanism, the steps of which include: By utilizing learned general device features, a dedicated cue embedding is generated as a feature representation of the device abstract concept; During inference, only device anchors with a similarity to the dedicated cue embedding higher than a preset threshold δ are retained, and the formula is as follows: Where o is the candidate anchor point feature embedding (dimension D) generated by the open set object detection model. The transpose of the dedicated prompt embedding, δ is the filtering threshold, which ranges from 0.005 to 0.01; The filtered anchor points are compared with the category embeddings in the built-in vocabulary for power generation equipment, and the most matching category is output.

[0033] Furthermore, the coordinate correction and image cropping in step S2 specifically include: Expand the boundary frame outward by 5% to 10% in all directions to avoid missing equipment edge defects, with photovoltaic panels expanded by 5% and turbine bearings by 10%. When multiple devices are detected in a single image, they are sorted by device category priority, with wind turbine blades having higher priority than supports in wind power scenarios. The cropped image is resized to the optimal input size of Qwen2.5-VL while maintaining the aspect ratio.

[0034] Furthermore, the Qwen2.5-VL fine-tuning training in step S3 includes: based on a multimodal power inspection dataset containing power generation scene images, text descriptions, and annotation information, the model is trained to have the ability to identify equipment types, detect defects, and understand semantics by combining supervised fine-tuning of SFT with contrastive learning.

[0035] Furthermore, the prompt word design in step S3 includes customized JSON format return requirements for different device types, including: For the main transformer, the prompt requires analyzing the building's exterior appearance and returning a Boolean value indicating whether the appearance is normal or not. For reactors or capacitors, the prompt requires analysis of whether there are bird nests or foreign objects present. For couplings and high-speed brake discs, the prompt requires a comparative analysis of two images to determine whether the protective shell is loose or has detached.

[0036] Based on the above method, this invention also proposes a power generation equipment detection and defect identification system based on open set target detection and multimodal model, comprising: Equipment location module: used to execute the power generation equipment defect identification method based on open set and multimodal model described above, and output the equipment bounding box; Image preprocessing module: used to perform coordinate correction, sorting and size adaptation as described in the above methods; Multimodal recognition module: used to deploy the fine-tuned Qwen2.5-VL model to identify defects based on customized prompt words; Data fusion module: used to convert local defect identification results back to global coordinates of the equipment and output a report.

[0037] Example 1 Overall implementation process; The power generation equipment defect identification method and system based on open sets and multimodal models provided in this embodiment are mainly applied to power generation scenarios such as thermal power, wind power, and photovoltaic power generation. Figure 1 As shown, this method mainly consists of two stages: Phase 1 (S1-S2): Addressing the questions of "where" and "what kind of device." Through an optimized open-set object detection model, new or non-standard devices can be identified without retraining.

[0038] The second phase (S3-S4) addresses the question of "what are the defects?". A deep semantic analysis is performed on the trimmed local diagram of the equipment using a specially fine-tuned multimodal large model (Qwen2.5-VL).

[0039] 1. Step S1: Depth Implementation of Device Open Set Detection and Positioning In this embodiment, the open set detection model achieves accurate detection of power generation equipment through the following three logical mechanisms: (1) RepRTA (Reparameterizable Region-Text Alignment) mechanism To achieve real-time detection at the edge of power inspection (such as drones and mobile centralized control terminals), this solution adopts reparameterization technology.

[0040] Training phase: A lightweight auxiliary network f is introduced, consisting of a single-layer feedforward network (FFN) with SwiGLU activation function. θ By leveraging the non-linear expressive power of SwiGLU, the semantic mapping depth between the text cue embedding P and the visual feature anchors is enhanced during training.

[0041] Among them, RepRTA's auxiliary network (containing a single SwiGLUFFN block) only works during training, and improves detection accuracy by optimizing the alignment quality of text embeddings and visual anchors through nonlinear transformations.

[0042] Remove non-linear layers during inference: After training, the auxiliary network (including the optimization effect of SwiGLU) is merged with the original network convolution kernel using the reparameterization technique to generate a new linear convolution kernel.

[0043] The key to zero inference overhead: after merging, there is no need to retain the SwiGLU layer. Inference only performs a linear convolution once, which is completely consistent with the original YOLO logic and has no additional computation. This is not a simplification or approximation of nonlinearity, but rather the solidification of the effect of nonlinear optimization in training into the parameters. Reparameterization is a strict algebraic operation, and the effect is equivalent and error-free.

[0044] Inference phase: To eliminate the computational overhead caused by nonlinear operations, the trained parameters in the auxiliary network are solidified and merged into the convolutional kernel K of the original network through rigorous algebraic operations. T In this process, the nonlinear SwiGLU layer is completely removed. The transformation formula is expressed as: The final generated K ’ It uses a pure linear convolution kernel, which makes the inference logic completely consistent with the original YOLO structure, achieving "zero additional inference overhead".

[0045] (2) Semantic activation of visual cues (Visual Prompt) logic For novel devices that cannot be described in words (such as a specific model of domestic inverter), the system supports "image search" detection. By using a semantically activated visual cue encoder, the number of groups A for the activated branches is set to 8. Experiments show that this number of groups effectively focuses the model on key structural features of the power generation equipment (such as heat dissipation grids and terminal block distribution) while maintaining low computational load, resulting in an approximately 12% improvement in detection accuracy compared to no grouping.

[0046] (3) Lazy area - prompt comparison (no prompt mode) When no textual or visual cues are provided, the system initiates a "self-discovery" mode. A filtering threshold δ=0.001 is introduced using a pre-generated vocabulary vector of power generation equipment from MobileCLIP-B(LT). This threshold, tested extensively in power scenarios, represents the optimal balance between average accuracy (AP) and inference speed (FPS), effectively filtering background noise from the ground and sky, retaining only candidate regions with extremely high similarity to the features of power generation equipment.

[0047] 2. Step S2: Image Refinement and Adaptive Correction After the detection model outputs the bounding box coordinates [x1, y1, x2, y2], it does not directly perform clipping, but instead executes adaptive correction logic: Outward expansion strategy: Considering that defects in power equipment often occur at the edges (such as cracks at the tips of wind turbine blades and corrosion on the edges of photovoltaic panels), the system automatically expands the boundary frame outward by 5% to 10% in each direction.

[0048] Differentiated settings: For photovoltaic panels with relatively regular structures, the expansion ratio is set to 5%; for turbine bearings or transformer bodies with complex structures, the expansion ratio is set to 10%.

[0049] Results: In actual inspection tests, this strategy reduced the missed detection rate of edge defects by approximately 30%. The cropped image is resized to the optimal input size of the multimodal model while maintaining the aspect ratio.

[0050] 3. Step S3: Multimodal defect identification based on Qwen2.5-VL The system inputs the cropped partial device diagram and customized prompts into the SFT (Supervised Fine-tuning) Qwen2.5-VL model.

[0051] Example of prompt words: Scenario A (Reactor): "Please analyze whether there are foreign objects (such as bird nests) in the image. If they exist, please return them in JSON format: {"foreign_object":true}, otherwise return false."

[0052] Scene B (Protective Case): "Comparison" Figure 1 Normal state, analysis Figure 2 If there is any loosening or detachment, return: {"scp":true / false}.

[0053] Technical advantages: In this solution, Qwen2.5-VL is only responsible for semantic determination (i.e., determining whether there is a defect) and does not involve complex pixel-level coordinate regression. This allows the large model to leverage its robustness under extreme weather conditions (such as glare and fog) and maintain a recognition accuracy of over 85%.

[0054] 4. Step S4: Result Fusion and Global Report Generation Finally, the system performs a "mounting" fusion of spatial coordinates and semantic information. The device's global physical coordinates obtained in the first stage are used as the "base map location," and the Boolean values ​​or decision types obtained in the second stage are used as the "attribute content."

[0055] Fusion logic: Global defect location = [coordinates of the top left corner of the device bounding box] + [local semantic judgment attribute].

[0056] Report output: Generates a structured JSON report containing equipment name, defect type, severity level, timestamp, and coordinate labels, and synchronizes it to the smart power plant operation and maintenance system.

[0057] 5. Experimental Conclusions and Comparisons The proposed solution was compared with traditional closed set detection models (such as the standard YOLOv8). The proposed solution greatly improves adaptability. The traditional model has a 0% recognition rate for new devices, while the proposed solution can achieve a recognition rate of about 85% with text prompts.

[0058] In response to the above methods, this invention also proposes a power generation equipment defect identification system based on open sets and multimodal models, comprising: Device localization module: used to perform the open set target detection described above and output the device bounding box; Image preprocessing module: used to perform the coordinate correction, sorting and size adaptation mentioned above; Multimodal recognition module: used to deploy the fine-tuned Qwen2.5-VL model to identify defects based on customized prompt words; Data fusion module: used to convert local defect identification results back to global coordinates of the equipment and output a report.

[0059] The following is an explanation of specific terms and more detailed steps for a method for identifying defects in power generation equipment based on open sets and multimodal models: Text prompts: Define equipment category lists for different power generation scenarios (e.g., thermal power scenario: "steam turbine, boiler water-cooled wall, valve, bearing"; wind power scenario: "wind turbine blade, gearbox, hub, control cabinet"; photovoltaic scenario: "photovoltaic panel, inverter, combiner box, bracket"). Utilize a reparameterizable region-text alignment strategy and optimize pre-trained text embeddings through a lightweight auxiliary network to improve the alignment accuracy between equipment categories and visual features, without additional overhead during inference. Visual cues: For devices with similar appearances but different functions, a semantic activation visual cue word encoder is used. The bounding box / mask of the device sample image is used as the visual cue word. Through the decoupled semantic branch (extracting common semantic features of the device) and activation branch (generating cue word perception weights), the device can be accurately located and adapted to niche or new devices. No prompt word required: For unknown devices, a lazy region-prompt word comparison strategy is enabled. By utilizing the built-in general vocabulary of power generation equipment, the device region in the image is filtered out through special embedding, without relying on an external language model, thus reducing deployment complexity.

[0060] The core mechanism of the reparameterizable region-text alignment module is to achieve efficient alignment of visual and text features in text prompt scenarios by "introducing a lightweight auxiliary network to optimize text embedding during the training phase and reparameterizing it to the main network during the inference phase", while maintaining the high inference efficiency of the open set object detection model.

[0061] Text embedding optimization: For equipment text prompts in power generation scenarios (such as "wind turbine blades" and "photovoltaic panels"), a pre-trained text encoder (such as MobileCLIP-B(LT)) is used to generate an initial text embedding P (with dimension D). Then, a lightweight auxiliary network (containing only one SwigLUFFN block) is used to optimize P to enhance the matching degree between text semantics and device visual features.

[0062] Reparameterization operation: After training, the parameters of the auxiliary network are "absorbed" into the classification head of the open set object detection model, forming an inference model consistent with the original YOLO structure, thus avoiding additional inference overhead.

[0063] Cross-modal alignment enhancement: Improves device classification accuracy under text cues by calculating the similarity between the optimized text embedding and device visual features (output by the target embedding head of the open-set object detection model). The formula is: Among them, K T For the original classification head convolution kernel (dimension) (C represents the number of categories, D represents the feature dimension) The result of the SwiGLUFFN network processing text embedding P. The parameter remapping function converts the auxiliary network parameters into convolutional kernel form. For convolution operations, This is the reconstructed classification head convolutional kernel (used directly during inference, without needing to retain the auxiliary network).

[0064] The semantic activation visual cue encoder module generates visual cue embeddings (such as cue based on "defect-free wind turbine blade" samples) with low computational complexity by "decoupling semantic branches and activation branches," adapting to the needs of power generation scenarios that rely on standard equipment samples for detection.

[0065] The core mechanism is as follows: Semantic branch: Independent of visual cues, extract general semantic features of the device in the image (such as the streamlined structure of the blades and the texture of the metal parts), without relying on the input visual cues, to ensure the stability of basic feature extraction.

[0066] Activation branch: Fuse input visual cues (such as standard device samples) with multi-scale image features to generate "cue perception weights" that dynamically enhance visual feature responses related to cues (such as highlighting areas that match the shape of standard blades).

[0067] Cue Embedding Generation: The general features of the semantic branches are fused with the cue weights of the activation branches to generate high-quality visual cue embeddings for subsequent device localization and classification. The formula is as follows: in, S represents the general semantic features (dimensions) output by the semantic branch. ), To activate the i-th group of cue perception weights in the branch output, A is the number of groups (A=8 in the power generation scenario to reduce computation), Concat() is the feature concatenation operation, and P is the final visual cue embedding.

[0068] The lazy region-cue comparison mechanism is used for autonomous inspection scenarios without preset prompts. It achieves efficient identification of power generation equipment through "dedicated prompt embedding + built-in vocabulary + lazy matching" and avoids the high overhead caused by relying on large language models.

[0069] Specialized cue embedding training: During the training phase, all power generation equipment is treated as a single category. The training learns common characteristics of the equipment (such as metal material and mechanical structure) to generate specialized cue embeddings. (Dimension D), representing the features of the abstract concept of "device". Anchor point filtering: During reasoning, only anchor points related to... Device anchors with similarity above a threshold (filtering background areas) reduce invalid calculations.

[0070] Lazy matching: The filtered anchor points are compared with the category embeddings in the built-in vocabulary for power generation equipment (such as "transformer" and "steam turbine"), and the best-matching category is output, achieving equipment identification without prompting. The formula is: Where o is the candidate anchor point feature embedding (dimension D) generated by the open set object detection model. The transpose of the embedded prompt is used for special purposes, and δ is the filtering threshold (δ=0.005~0.01 in the power generation scenario to reduce false detections of backgrounds such as sky and ground). Model fine-tuning; Dataset construction: Collect publicly available datasets of power generation equipment (such as wind turbine blade defect datasets and photovoltaic panel defect datasets) and self-built datasets (containing equipment images under different scenes and different lighting / shading conditions, labeled with equipment categories and bounding boxes). Training parameters: 30 epochs for text-based prompts, 2 epochs for visual prompts (with the backbone network frozen, training only the semantic activation visual cue encoder module), and 1 epoch for scenarios without prompts (training dedicated embeddings). The AdamW optimizer was used (initial learning rate 0.002, batch size = 128, weight decay 0.025). Data augmentation included mosaic, random horizontal flipping, and color jittering to adapt to the complex environment of power generation scenarios.

[0071] Equipment detection and inference; Input: Real-time images of the power generation scene (from bullet cameras, zoom gimbal dome cameras, dual-spectrum bullet cameras, drone inspection equipment, etc.); Output: Device category label and corresponding bounding box coordinates (format: [x1,y1,x2,y2], where (x1,y1) is the top-left corner coordinate and (x2,y2) is the bottom-right corner coordinate).

[0072] Coordinate correction: Considering potential edge occlusion in the image of power generation equipment (such as wind turbine blade edges being obscured by the tower), the bounding box is expanded by 5%~10% in each direction (adaptive adjustment can be made according to the equipment type, such as 5% for photovoltaic panels and 10% for steam turbine bearings) to avoid missing defects at the edges of the equipment. Multi-device cropping and sorting: If multiple devices are detected in a single image (such as multiple panels in a photovoltaic array), they are sorted according to the priority of device category (core devices take priority, such as "wind turbine blades" taking priority over "support" in a wind power scenario), and cropped in sequence to generate single-device images; Size adaptation: The cropped image is resized to the optimal input size of Qwen2.5-VL to maintain the image aspect ratio and avoid defects and deformations caused by stretching.

[0073] Prompt word design: Customize prompt words for different types of power generation equipment, as shown in the following example: (1) Main body: Please analyze the appearance of the building in the image. If there is an accident such as the collapse of the building in the image, please return the result in JSON format: [{\"appearance\":true}]; if the building in the image is in a normal state, please return: [{\"appearance\":false}]; (2) Whether there is a bird's nest in the reactor / capacitor: "Please analyze whether there is a foreign object (such as a bird's nest) in the image. If there is a foreign object (such as a bird's nest) in the image, please return the result in JSON format: [{\"foreign_object\":true}]; If there is no foreign object in the image, please return: [{\"foreign_object\":false}]; (3) Couplings and high-speed brake discs: Please provide analysis and details. Figure 1 If the fan casing is in normal condition, please analyze. Figure 2 If the protective case is in a secure, undamaged state, please return the result in JSON format: [{\"scp\":true}]; if the protective case is in a normal state, please return: [{\"scp\":false}].

[0074] Defect identification reasoning: Input: Cropped device image + customized prompt words; Output: Defect type, defect level (e.g., "[{'appearance':False}]").

[0075] Result fusion: The device detection results (device category, global coordinates) of the open set target detection model are fused with the defect identification results (defect type) of Qwen2.5-VL and converted into global defect coordinates of the device (local coordinates + coordinate offset of the upper left corner of the device bounding box) to generate a detection report.

[0076] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0080] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for identifying defects in power generation equipment based on open sets and multimodal models, characterized in that, Includes the following steps: Step S1: Device Open Set Detection and Localization: An open set target detection model optimized for power generation scenarios is used to realize open set detection and localization of devices, and to obtain device category labels and corresponding bounding box coordinates; the open set target detection model supports device recognition under three paradigms: text prompts, visual prompts, and no prompts; In step S1, visual cue detection is achieved through a semantic activation visual cue encoder module, and the implementation steps include: Extract common semantic features of devices from images using semantic branches; By activating branches to fuse input visual cue samples with multi-scale image features, cue perception weights are generated. By concatenating and fusing general semantic features with cue perception weights, a visual cue embedding is generated for accurate device positioning and classification. The formula for generating the visual cue embedding P is: S represents the general semantic feature output by the semantic branch, with dimensions... , The i-th group of cue perception weights is used to activate the branch output, where A is the number of groups, Concat() is the feature concatenation operation, and P is the final visual cue embedding. In step S1, device identification in the unnoticed scenario is achieved through a lazy region-noticed comparison mechanism, the steps of which include: By utilizing learned general device features, a dedicated cue embedding is generated as a feature representation of the device abstract concept; During inference, only device anchors with a similarity to the dedicated cue embedding higher than a preset threshold δ are retained, and the formula is as follows: Where o is the candidate anchor point feature embedding generated by the open set object detection model, The transpose of the dedicated prompt embedding, δ is the filtering threshold, which ranges from 0.005 to 0.01; The filtered anchor points are compared with the category embeddings in the built-in vocabulary for power generation equipment, and the most matching category is output. Step S2: Refined processing of device images: The device bounding boxes output in step S1 are subjected to coordinate correction and image cropping, and sorted according to the priority of device categories to generate images of the devices to be identified that are adapted to the input requirements of the multimodal model. The coordinate correction and image cropping in step S2 specifically include: Expand the boundary frame outward by 5% to 10% in all directions to avoid missing equipment edge defects, with photovoltaic panels expanded by 5% and turbine bearings by 10%. When multiple devices are detected in a single image, they are sorted by device category priority, with wind turbine blades having higher priority than supports in wind power scenarios. Resize the cropped image to the optimal input size of Qwen2.5-VL while maintaining the aspect ratio; Step S3: Multimodal defect recognition and reasoning: Input the image of the equipment to be identified and the prompt words customized for defects in power generation equipment into the pre-fine-tuned Qwen2.5-VL multimodal large model to perform defect type identification, localization and severity assessment; The Qwen2.5-VL fine-tuning training in step S3 includes: based on a multimodal power inspection dataset containing power generation scene images, text descriptions, and annotation information, the model is trained to have the ability to identify equipment types, detect defects, and understand semantics by combining supervised fine-tuning of SFT with contrastive learning. Step S4: Result Fusion Output: The bounding box coordinates output in Step S1 are used as a spatial reference and fused with the defect semantic information output in Step S3 to generate a global defect detection report.

2. The method for identifying defects in power generation equipment based on open sets and multimodal models according to claim 1, characterized in that, The open set object detection model in step S1 applies a reparameterizable region-text alignment module, the core mechanism of which includes: An initial text embedding P is generated using a pre-trained text encoder, and P is optimized using a lightweight auxiliary network to enhance the matching degree between text semantics and device visual features. After training, the parameters of the auxiliary network are reparameterized into the classification head of the detector through the parameter remapping function to form an inference model consistent with the original YOLO structure, so as to achieve visual-text feature alignment with zero inference overhead.

3. The method for identifying defects in power generation equipment based on open sets and multimodal models according to claim 2, characterized in that, The reparameterization operation follows the formula: Among them, K T The original classification head convolution kernel; Dimension C represents the number of categories, and D represents the feature dimension. The processing results of text embedding P for the auxiliary network SwigLUFFN; R is the parameter remapping function, which converts the auxiliary network parameters into convolutional kernel form; * indicates a convolution operation, K ’ This is the reconstructed classification head convolution kernel.

4. The method for identifying defects in power generation equipment based on open sets and multimodal models according to claim 1, characterized in that, The prompt word design in step S3 is tailored to different device types, requiring customized JSON format returns, including: For the main transformer, the prompt requires analyzing the building's exterior appearance and returning a Boolean value indicating whether the appearance is normal or not. For reactors or capacitors, the prompt requires analysis of whether there are bird nests or foreign objects present. For couplings and high-speed brake discs, the prompt requires a comparative analysis of two images to determine whether the protective shell is loose or has detached.

5. A defect identification system for power generation equipment based on open sets and multimodal models, characterized in that, include: Equipment location module: used to execute the power generation equipment defect identification method based on open set and multimodal model as described in any one of claims 1 to 3, and output the equipment bounding box; Image preprocessing module: used to perform coordinate correction, sorting and size adaptation as described in claim 1; Multimodal recognition module: used to deploy the fine-tuned Qwen2.5-VL model to identify defects based on customized prompt words; Data fusion module: used to convert local defect identification results back to global coordinates of the equipment and output a report.

Images