Global video intelligent inspection method for wind power station based on multispectral fusion
By combining multispectral fusion and deep learning technologies with the three-dimensional geographic information of wind power stations and digital twin models of equipment, intelligent inspection of wind power station equipment has been achieved. This solves the problems of low efficiency and insufficient accuracy of traditional manual inspection, and provides efficient and accurate equipment status monitoring and fault prediction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GUODIAN GUANGXI NEW ENERGY DEV CO LTD
- Filing Date
- 2026-03-10
- Publication Date
- 2026-07-14
AI Technical Summary
Traditional manual inspection methods in wind power stations are characterized by high cost, low efficiency, susceptibility to human factors, and the tendency to overlook equipment malfunctions, making it difficult to ensure safe operation and efficient maintenance of equipment in different environments.
A comprehensive video intelligent inspection method for wind power stations based on multispectral fusion is adopted. By deploying a distributed multispectral video acquisition network, combining video streams from visible light, infrared, and ultraviolet bands, multi-dimensional feature fusion and deep learning are performed. Combined with high-precision three-dimensional geographic information of the wind power station and digital twin models of equipment, the method can accurately identify, classify, and locate equipment anomalies. The inspection report is optimized through cross-validation of multi-source data.
It enables accurate identification and classification of abnormal characteristics of wind power generation equipment, improves the accuracy and comprehensiveness of inspections, ensures the accuracy and reliability of inspection reports, and can efficiently monitor equipment status and predict faults in different environments.
Smart Images
Figure CN122391938A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of wind power equipment inspection technology, specifically to a full-area video intelligent inspection method for wind power stations based on multispectral fusion. Background Technology
[0002] With the increasing global demand for renewable energy, wind power, as an important clean energy source, has been widely used in various geographical areas, especially in regions rich in wind resources. To ensure the efficient operation and maintenance of wind power plants, the inspection and monitoring of wind power equipment has become particularly important. However, traditional manual inspection methods suffer from high costs, low efficiency, and limitations due to human factors, and are prone to overlooking potential equipment failures and anomalies. Therefore, how to improve the inspection efficiency of wind power plants and ensure the safe operation of equipment in different environments has become an urgent problem to be solved in the current wind power industry.
[0003] Against this backdrop, video intelligent inspection technology has emerged. By combining multispectral technology, it collects multispectral video streams in the visible, infrared, and ultraviolet bands of wind power stations and performs real-time analysis through intelligent algorithms. This can significantly improve the accuracy and automation of inspections. Furthermore, the combination of multispectral video fusion technology and deep learning algorithms can more comprehensively and accurately identify abnormal equipment conditions and perform intelligent processing and classification. Summary of the Invention
[0004] The purpose of this invention is to solve the problems mentioned above by proposing a full-area video intelligent inspection method for wind power stations based on multispectral fusion.
[0005] The objective of this invention can be achieved through the following technical solutions: A multispectral fusion-based intelligent video inspection method for wind power stations covering the entire area includes: Based on the overall topology of the wind power station, a distributed multispectral video acquisition network is deployed to simultaneously acquire multispectral raw video streams containing visible light, infrared and ultraviolet bands, resulting in a multidimensional multispectral raw video set. The original multispectral video set is optimized using a band-adaptive preprocessing strategy to obtain a standardized multispectral video dataset. The standardized multispectral video dataset is subjected to cross-band feature deep fusion using a weighted multispectral fusion model to generate a multi-dimensional fused video feature set that combines spatial details and spectral differences. The multi-dimensional fused video feature set is input into a deep learning inspection model based on an attention mechanism for training and inference, to identify and classify abnormal features of wind power generation equipment. The identified abnormal features are fused with the high-precision three-dimensional geographic information of the wind power station and the digital twin model of the equipment to perform spatial coordinate calibration and positioning, generate an inspection report, verify and optimize the inspection report, and obtain the final inspection results.
[0006] Preferably, the original multispectral video set is optimized using a band-adaptive preprocessing strategy to obtain a standardized multispectral video dataset, including: The original multispectral video set is subjected to frame extraction and deduplication to obtain single-frame image sequences corresponding to each band, forming a multiband image set. Differentiated noise reduction processing is adopted based on the noise characteristics of images in each band: the visible light band adopts the noise intensity adaptive median filtering algorithm, the infrared band adopts the improved wavelet threshold noise reduction algorithm, and the ultraviolet band adopts the bilateral filtering and mean shift combined algorithm to accurately suppress noise. An improved scale-invariant feature transform algorithm is used to extract key feature points of images in each band, generate directional feature descriptors, and perform high-precision spatial alignment of multi-band images through feature matching and random sampling consensus algorithm. Pixel-level scale normalization is performed on the aligned multi-band image, mapping the pixel values of each band to the [0,255] range, and unifying the image resolution to the preset standard size; Adaptive histogram equalization combined with the Retinex algorithm is used to compensate for the uneven illumination regions in the normalized image, resulting in a multispectral image sequence with consistent illumination. Add timestamps, device numbers, and acquisition location metadata to the processed multispectral image sequences to form a standardized multispectral video dataset.
[0007] Preferably, a weighted multispectral fusion model is used to perform cross-band feature deep fusion on the standardized multispectral video dataset to generate a multi-dimensional fused video feature set that combines spatial details and spectral differences, including: A pixel-level and feature-level dual-layer weighted fusion framework is constructed to perform hierarchical progressive fusion on the standardized multispectral video dataset. Weighting coefficients are calculated based on the information entropy, signal-to-noise ratio, and edge sharpness of each band image. An improved weighted average algorithm is then used to sum the pixel values of the standardized multi-band images to obtain the initial fused image. The formula for the weighted average algorithm is as follows: In the formula, To merge images, For the first Each band image at location pixel values on It is the first Weighting coefficients for each band, It is the number of bands. It is the first Information entropy of an image band Information entropy for each band; Deep semantic features are extracted from the initial fused image based on a convolutional neural network, while shallow visual features of texture and edges of each single-band image are extracted through gray-level co-occurrence matrix and LBP operator. An attention mechanism is used to weightedly fuse deep semantic features and shallow visual features to enhance the feature response of abnormal regions; The kernel principal component analysis algorithm is used to reduce the dimensionality of the fused features, retaining the principal component features with a cumulative contribution rate of over 90%, and generating a multi-dimensional fused video feature set.
[0008] Preferably, the multi-dimensional fused video feature set is input into a deep learning inspection model based on an attention mechanism for training and inference, to identify and classify abnormal features of wind power generation equipment, including: A deep learning inspection model based on the combination of improved YOLOv8 and VisionTransformer is constructed. The model includes a feature extraction layer, a feature enhancement layer, an anomaly detection layer, and a classification layer. The multi-dimensional fused video feature set was divided into a training set, a validation set, and a test set in a 7:2:1 ratio, and the training set samples were expanded using a data augmentation strategy. Set the model training parameters, input the training set into the model for training, monitor the model performance through the validation set, and use an early stopping strategy to avoid overfitting. The test set is input into the trained model for inference, identifying the anomaly types of wind power generation equipment and outputting the anomaly confidence level, thereby identifying and classifying the anomaly features.
[0009] Preferably, the identified anomaly features are fused with the high-precision three-dimensional geographic information of the wind power station and the digital twin model of the equipment for spatial coordinate calibration and positioning, generating an inspection report, including: Acquire a high-precision 3D geographic information model and digital twin model of the wind power station, and extract the fused video frames containing abnormal features and the positioning and attitude parameters of the corresponding acquisition equipment; Based on the intrinsic parameters, extrinsic parameters, and distortion parameters of the acquisition device, the pixel coordinates of abnormal features are converted into world coordinates, and a coordinate mapping relationship with the 3D geographic information model is established. By combining the digital twin model of the equipment and using the iterative nearest point algorithm to calibrate the world coordinates, the precise spatial location coordinates of the anomaly feature are obtained. Integrate anomaly type, precise coordinates, data collection time, and confidence level information to generate a standardized inspection report.
[0010] Preferably, the inspection report is verified and optimized to obtain the final inspection result, including: Retrieve historical inspection data, real-time equipment operation data, and environmental monitoring data of the abnormal area, and perform multi-source cross-validation with the inspection report; If the cross-validation consistency exceeds the preset threshold, the inspection report is confirmed to be valid; if the consistency does not meet the standard, the corresponding area multispectral equipment is activated for secondary acquisition, and the inspection report is regenerated. The information of the verified inspection reports is supplemented and the format is standardized to obtain the final inspection results.
[0011] Preferably, the method further includes an adaptive scheduling strategy for the multispectral acquisition network, specifically including: Based on the three-dimensional geographic information model of the wind power station, core and general inspection areas are divided, and a redundant data acquisition network is constructed. Real-time data collection of device operating status, battery level, and environmental information; and the construction of a device scheduling model using deep reinforcement learning algorithms. Based on the scheduling strategy output by the equipment scheduling model, the working mode, acquisition parameters and angle of the acquisition equipment are adjusted, and redundant backup and protection modes are activated in the event of equipment failure or severe weather.
[0012] Preferably, real-time data collection of device operating status, battery level, and environmental information is used to construct a device scheduling model using deep reinforcement learning algorithms, specifically including: The device's operating status, battery level, and environmental information are converted into quantitative indicators. One-hot encoding is used to process the categorized data to form a standardized state vector. Define the state space, action space, and reward function of the deep reinforcement learning model. The state space includes standardized state vectors and inspection area priorities. The action space includes switching of device working modes, adjusting acquisition parameters, fine-tuning of shooting angle, and starting and stopping redundant devices. The reward function adopts a weighted summation method, and the weight coefficients are optimized through grid search. A deep reinforcement learning model is constructed using a proximal policy optimization algorithm. The model consists of an input layer, two fully connected hidden layers, and an output layer. The input layer receives a standardized state vector, and the output layer outputs the probability distribution of each action. A training dataset is constructed based on historical scheduling data, equipment operation logs, and simulated extreme environment data. It is divided into a training set and a validation set in an 8:2 ratio. The training set is input into the model for training, and the model reward value is monitored in real time through the validation set. An early stop strategy is adopted. If the validation set reward value does not improve for 20 consecutive rounds, training is stopped to avoid model overfitting. After training, the model outputs the optimal scheduling action based on the real-time state vector.
[0013] Preferably, after obtaining the precise spatial coordinates of the anomalous features, the method further includes predicting the anomalous development trend, specifically including: Based on the digital twin model of the equipment, historical operation, inspection and maintenance data of the abnormal location are retrieved to construct an anomaly development time series dataset; A prediction model is constructed using LSTM combined with an attention mechanism. The model is input into a time series dataset to predict the future development and impact of anomalies. The prediction results are then incorporated into the final inspection results to provide a priority basis for operation and maintenance.
[0014] Preferably, the step of constructing a prediction model using LSTM combined with an attention mechanism, and inputting a time-series dataset to predict the future development degree and impact range of the anomaly, specifically includes: A prediction model combining LSTM and self-attention mechanism is constructed. The model includes an input layer, an LSTM feature extraction layer, a self-attention enhancement layer, a fully connected prediction layer, and an output layer. The LSTM feature extraction layer has 3 hidden layers, and the self-attention layer uses a multi-head attention mechanism to enhance key temporal features. The model training parameters were set, an adaptive moment estimation optimizer was used, the initial learning rate was set to 0.001, the mean squared error was used as the loss function, the time series dataset was input into the model for training, and the prediction accuracy was monitored by the validation samples. Input the time series data of the anomaly to be predicted into the trained model, output the quantitative value of the anomaly's development degree and the boundary parameters of its impact range within a preset future period, and obtain the anomaly development trend prediction result.
[0015] Compared with the prior art, the beneficial effects of the present invention are: 1. By fusing multispectral video data from visible, infrared, and ultraviolet bands and employing a weighted multispectral fusion model, deep fusion of cross-band features is achieved. This technology can comprehensively capture abnormal features of wind power generation equipment, including abnormal surface temperatures and cracks that are difficult to detect with a single band, thereby significantly improving the accuracy and comprehensiveness of inspections.
[0016] 2. By combining the improved YOLOv8 and VisionTransformer models and introducing an attention mechanism, multi-dimensional fused video features are trained and inferred to achieve accurate identification and classification of abnormal features of wind power generation equipment. This innovation improves the accuracy of equipment fault detection and reduces the risk of misjudgment caused by overfitting.
[0017] 3. By combining high-precision 3D geographic information of wind power stations with digital twin models of equipment, spatial coordinate calibration technology is used to accurately locate abnormal equipment features. At the same time, multi-source data cross-validation and dynamic optimization strategies are used to ensure the accuracy and effectiveness of inspection reports. Secondary data collection is automatically initiated for verification and optimization, further improving the credibility and operability of inspection results. Attached Figure Description
[0018] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.
[0019] Figure 1 This is a flowchart of the method steps of the present invention. Detailed Implementation
[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0022] Please see Figure 1 As shown, the intelligent video inspection method for the entire wind power station based on multispectral fusion includes: Step 1.1: Acquisition of Multispectral Raw Video Sets. First, a full-area topology survey of the wind power station is conducted to determine the location, distribution density, and inspection priority of core facilities such as wind turbines, substations, transmission lines, and energy storage equipment. A full-area topology map is constructed, and a distributed multispectral video acquisition network is deployed based on this map. Multispectral cameras are installed at locations such as the top of the nacelle, the root of the blades, the middle and bottom of the tower of each wind turbine, the high and low voltage equipment area of the substation, and key nodes of the transmission line. Each camera supports simultaneous acquisition of visible light, infrared, and ultraviolet light bands, and the sampling frame rate can be adaptively adjusted within the range of 5-30fps. Clock synchronization of all acquisition devices is achieved through a time synchronization protocol to ensure the time consistency of multi-band and multi-device acquisition data. 7×24-hour multispectral raw video streams are acquired synchronously, and a multi-dimensional multispectral raw video set is obtained.
[0023] Step 1.2: Preprocessing of the multispectral raw video dataset. The acquired multispectral raw video dataset is optimized using a band-adaptive preprocessing strategy. Differentiated processing procedures are designed for the signal characteristics of different bands to eliminate interference from noise, illumination, scale differences, etc., and to obtain a standardized multispectral video dataset.
[0024] Step 1.3: Deep fusion of multispectral features. Construct a weighted multispectral fusion model. Adopt a two-layer progressive fusion method at the pixel level and feature level. First, weighting coefficients are assigned based on the information entropy, signal-to-noise ratio and other indicators of each band image to achieve initial pixel-level fusion. Then, deep semantic features and shallow visual features are extracted respectively. The abnormal region features are strengthened through the attention mechanism. Finally, a multi-dimensional fused video feature set is generated after dimensionality reduction processing.
[0025] Step 1.4: Anomaly feature identification and classification. Construct a deep learning inspection model based on the combination of improved YOLOv8 and VisionTransformer. Divide the fused video feature set into training set, validation set and test set according to the proportion. After model training and inference, identify anomaly types such as blade wear, gearbox leakage, circuit overheating and fastener loosening, and output the confidence level of each anomaly.
[0026] Step 1.5: Anomaly location, report generation and optimization. Obtain a 1:500 high-precision 3D geographic information model and equipment digital twin model of the wind power station. Combine the collected equipment parameters to convert the anomaly pixel coordinates into world coordinates and calibrate them to generate an inspection report containing anomaly type, location, time and confidence level. Optimize the report through cross-validation of multi-source data to obtain accurate inspection results. At the same time, an anomaly development trend prediction step can be added to provide priority basis for operation and maintenance.
[0027] Band-adaptive preprocessing of multispectral raw video datasets, specifically including: Step 2.1: Frame extraction and deduplication. For each video in the multispectral original video set, frames are extracted using a fixed time interval method to obtain the single-frame image sequence corresponding to each band. The extracted frame images are deduplicated based on the structural similarity algorithm (SSIM). The SSIM threshold is set to 0.95. If the SSIM value of two adjacent frames is greater than the threshold, they are determined to be duplicate frames. The previous frame is retained and the next frame is deleted, thus forming a multi-band image set.
[0028] Step 2.2: Differentiated noise reduction processing. Dedicated noise reduction algorithms are designed for the noise characteristics of images in different bands: The visible light band is mainly affected by salt-and-pepper noise, so an adaptive median filtering algorithm based on noise intensity is used. The filter window size is dynamically adjusted according to the noise density of the pixel neighborhood. A 7×7 window is used when the noise density is higher than 30%, and a 3×3 window is used when it is lower than 10%, achieving precise noise suppression. The infrared band is mainly affected by Gaussian noise, so an improved wavelet thresholding algorithm is used, replacing the traditional hard threshold with a linear soft threshold function to reduce signal distortion caused by thresholding. The wavelet basis is db4, and the decomposition level is set to 3. The ultraviolet band has complex noise types, so a combined bilateral filtering and mean shift algorithm is used. First, bilateral filtering preserves edge information while removing high-frequency noise, and then mean shift smooths low-frequency noise. The spatial standard deviation of the bilateral filtering is set to 2, the grayscale standard deviation is set to 20, and the bandwidth of the mean shift is set to 5.
[0029] Step 2.3: Multi-band image spatial alignment. A modified scale-invariant feature transform (SIFT) algorithm is used to extract key feature points from each band image. An adaptive contrast threshold adjustment mechanism is introduced during the feature point detection stage, dynamically setting the contrast threshold based on the image grayscale variance to generate a 128-dimensional feature descriptor with orientation. Feature matching is performed using the K-nearest neighbor algorithm, with a matching distance threshold of 1.2, yielding initial matching point pairs. If the number of initial matching point pairs exceeds 50, a random sampling consensus algorithm is used to remove abnormal matching points, with 1000 iterations and an interior point threshold of 2 pixels, ultimately obtaining reliable matching point pairs. Based on the reliable matching point pairs, an affine transformation matrix is calculated to complete high-precision spatial alignment of the multi-band images, with the alignment error controlled within 1 pixel.
[0030] Step 2.4: Pixel-level scale normalization. Perform double normalization of pixel values and resolution on the aligned multi-band images: use the min-max normalization algorithm to map the pixel values of each band to the [0,255] interval, and use the bilinear interpolation algorithm to unify the resolution of all images to the preset standard size to ensure scale consistency in subsequent processing.
[0031] Step 2.5: Illumination compensation. Adaptive histogram equalization combined with the Retinex algorithm is used to perform illumination compensation on the normalized image. The CLAHE algorithm's cropping limit is set to 2.0, and the grid size is set to 8×8 to enhance local contrast. The Retinex algorithm uses multi-scale Gaussian filtering with scale parameters set to 15, 80, and 250 to separate reflected light from incident light and eliminate image grayscale deviations caused by uneven illumination, ultimately obtaining a multispectral image sequence with consistent illumination.
[0032] Step 2.6: Metadata addition. Standardized metadata is added to the processed multispectral image sequence, including timestamps, device numbers, acquisition locations, and acquisition parameter information. This is encapsulated in JSON format to form a standardized multispectral video dataset.
[0033] In some embodiments, multispectral cross-band feature deep fusion specifically includes: Step 3.1: Construct a two-layer weighted fusion framework. Design a pixel-level and feature-level two-layer weighted fusion framework. Pixel-level fusion focuses on the complementarity of pixel information in each band of the image, while feature-level fusion focuses on the semantic complementarity of features at different levels, so as to realize full-dimensional feature fusion from the bottom pixel to the high-level semantic.
[0034] Step 3.2: Pixel-level weighted fusion. Weighting coefficients are calculated based on the information entropy, signal-to-noise ratio (SNR), and edge sharpness of each band image. The entropy weighting method is used to determine the weights of the three indicators: information entropy weight 0.4, SNR weight 0.3, and edge sharpness weight 0.3. Information entropy H is calculated using Shannon entropy, reflecting the richness of image information. SNR is calculated as the ratio of signal power to noise power, reflecting the image noise suppression effect. Edge sharpness E is calculated by extracting edges using the Sobel operator and then calculating the average gradient magnitude, reflecting the degree of image detail preservation. An improved weighted average algorithm is used to weight and sum the pixel values of the standardized multi-band images to obtain the initial fused image. Step 3.3: Feature-level fusion. A convolutional neural network is used to extract deep semantic features from the initial fused image. ResNet50 is selected as the backbone network. The last fully connected layer is removed, and a 2048-dimensional deep feature vector is output. At the same time, the texture features of each single-band image are extracted through the gray-level co-occurrence matrix, and the edge features are extracted through the LBP operator. The texture features and edge features are concatenated to form a 512-dimensional shallow visual feature vector.
[0035] Step 3.4: Weighted Fusion with Attention Mechanism. A channel attention mechanism is introduced to weightedly fuse deep semantic features and shallow visual features. Through adaptive learning, weights are assigned to different feature channels to enhance the feature response of abnormal regions and suppress background noise features. Specifically, deep features and shallow features are concatenated into a 2560-dimensional feature vector, which is then input into the SE module. After global average pooling, a fully connected layer, and Sigmoid activation, 2560-dimensional attention weights are obtained. The weights are then multiplied element-wise with the concatenated features to obtain the fused feature vector.
[0036] Step 3.5: Feature dimensionality reduction. The kernel principal component analysis algorithm is used to reduce the dimensionality of the fused feature vector. The radial basis function is selected as the kernel function and the kernel parameter is set to 0.01. The contribution rate of each principal component is calculated, and the principal component features with a cumulative contribution rate of more than 90% are retained. Finally, a multi-dimensional fused video feature set with reduced dimensionality and complete information is generated.
[0037] In some embodiments, anomaly feature identification and classification based on deep learning specifically includes: Step 4.1: Construction of Deep Learning Inspection Model. A deep learning inspection model based on the combination of improved YOLOv8 and VisionTransformer is constructed. The model is divided into four layers: the feature extraction layer adopts the C2f module and SPPF module of improved YOLOv8 to enhance the correlation between low-level features and high-level features; the feature enhancement layer introduces the multi-head attention mechanism of ViT to divide the feature map into 16×16 image patches and generate 768-dimensional image patch embedding vectors, and captures long-distance feature dependencies through multi-head attention; the anomaly detection layer adopts the detection head of YOLOv8 and outputs the bounding box and confidence of the abnormal target; the classification layer adopts a fully connected layer and outputs the classification result.
[0038] Step 4.2: Dataset partitioning and augmentation. The multi-dimensional fused video feature set is randomly divided into training, validation, and test sets in a ratio of 7:2:1. The training set is used for model parameter learning, the validation set is used to monitor the training process and adjust parameters, and the test set is used to evaluate the final performance of the model. Data augmentation strategies are used to expand the training set samples, including random cropping, horizontal flipping, vertical flipping, color gamut transformation, and Gaussian noise addition, which doubles the size of the training set samples and improves the model's generalization ability.
[0039] Step 4.3: Model Training. Set the model training parameters: The optimizer uses adaptive moment estimation, the initial learning rate is set to 0.001, the learning rate is adjusted using cosine annealing, and the weight decay coefficient is set to 0.0005; the batch size is set to 32, and the number of iterations is set to 300 rounds; the loss function is a combination of CIoU loss, cross-entropy loss, and Focal loss; the model's precision, recall, and F1 score are monitored through the validation set, and an early stopping strategy is adopted, stopping training if the F1 score on the validation set does not improve for 15 consecutive rounds to avoid model overfitting.
[0040] Step 4.4: Model Inference and Anomaly Classification. The test set is input into the trained model for inference. The model outputs the bounding box coordinates, anomaly type, and confidence level for each anomaly target. The confidence level threshold is set to 0.7. If the output confidence level is greater than the threshold, it is determined to be a valid anomaly, and its type and location information are retained. If the confidence level is less than the threshold, it is determined to be a suspected anomaly, marked, and included in the secondary review. Finally, the accurate identification and classification of anomaly features of wind power generation equipment is achieved.
[0041] In some embodiments, the calibration and positioning of anomaly feature spatial coordinates and the generation of inspection reports specifically include: Step 5.1: Basic Model and Parameter Extraction. Obtain a 1:500 high-precision 3D geographic information model of the wind power station. This model includes the precise 3D coordinates of facilities such as terrain, roads, wind turbines, and substations, using a geodetic coordinate system. Obtain the digital twin model of the equipment. This model is constructed based on the equipment's CAD drawings and actual measurement data, and includes the 3D structure, material parameters, and operating thresholds of core components such as blades, gearboxes, and nacelles. Extract the fused video frames containing abnormal features from the identification results, and simultaneously retrieve the positioning and attitude parameters of the acquisition equipment corresponding to that frame.
[0042] Step 5.2: Pixel coordinates to world coordinates. Based on the intrinsic, extrinsic, and distortion parameters of the acquisition device, the conversion of anomalous feature pixel coordinates to world coordinates is completed. The device intrinsic parameters are obtained through Zhang Zhengyou's calibration method, including focal length, principal point coordinates, and distortion coefficients. The extrinsic parameters include the rotation matrix R and the translation vector T, which are determined through the calibration data during device installation. The conversion process consists of two steps: First, the lens distortion effect is eliminated through the distortion correction formula to obtain the corrected pixel coordinates; then, the corrected pixel coordinates are converted to world coordinates through the camera imaging model to establish a coordinate mapping relationship with the 3D geographic information model.
[0043] Step 5.3: World coordinate calibration. Using the device's digital twin model, the iterative nearest point algorithm is employed to accurately calibrate the transformed world coordinates. One hundred feature points on the device surface surrounding the anomaly are selected as the target point set, and 100 corresponding 3D points are extracted from the digital twin model as the source point set. The ICP algorithm iteratively optimizes the rotation matrix and translation vector to minimize the average distance between the source and target point sets. An upper limit of 500 iterations is set, and a convergence threshold of 0.01 meters is used to finally obtain the precise spatial coordinates of the anomaly.
[0044] Step 5.4: Inspection report generation. Integrate information such as anomaly type, precise spatial coordinates, acquisition time, anomaly confidence level, acquisition device number, and image evidence. According to the designed standardized report template, generate a standardized inspection report containing text description, data tables, and anomaly image annotations. Supports PDF format export and system upload.
[0045] In some embodiments, the inspection report verification optimization specifically includes: Step 6.1: Cross-validation of multi-source data. Three main categories of data are retrieved and cross-validated with the inspection report: First, historical inspection data for the abnormal area, including historical anomaly types, locations, handling results, and recurrence rates; second, real-time equipment operation data, obtained from the SCADA system, including parameters such as equipment speed, temperature, vibration, and oil pressure; and third, environmental monitoring data, including wind speed, wind direction, temperature, and humidity during the inspection period. A correlation analysis algorithm is used to calculate the consistency between the anomaly information in the inspection report and the three types of data. Consistency indicators include type matching degree, location overlap degree, correlation of abnormal operating parameters, and correlation of environmental impact.
[0046] Step 6.2: Verification result judgment and secondary acquisition. Set the cross-validation consistency threshold to 85%. If the consistency index exceeds the threshold, the inspection report is confirmed to be valid and proceeds to the subsequent optimization stage. If the consistency index is lower than the threshold, it is judged as a suspected false alarm or missed alarm. The secondary acquisition process of the multispectral equipment in the corresponding area is started, repeating feature fusion, anomaly identification and coordinate positioning, and regenerating the inspection report. If the consistency between the secondary report and the original report is still lower than the threshold, manual review is triggered.
[0047] Step 6.3: Report Optimization and Final Result Output. For verified inspection reports, supplement information such as the service life of equipment in the abnormal area, historical maintenance records, and the operating status of related equipment. Standardize the reports according to the format requirements of the operation and maintenance management system, unifying report numbers, field names, and data formats. Integrate the optimized reports with the anomaly development trend prediction results to form the final inspection results, which are then pushed to the operation and maintenance management platform to provide operation and maintenance personnel with accurate fault handling basis.
[0048] In some embodiments, the adaptive scheduling of the multispectral acquisition network specifically includes: Step 7.1: Inspection Area Division and Redundant Network Construction. Based on the 3D geographic information model of the wind power station, the K-means clustering algorithm is used to divide the core inspection area into a core inspection area and a general inspection area. The core area includes areas prone to anomalies and with significant impact, such as wind turbine blades, gearboxes, nacelle control cabinets, substation main transformer equipment, and transmission line joints. The division density is one data acquisition device per 100 square meters. The general area includes the outside of the tower, fenced areas, auxiliary facilities, etc. The division density is one data acquisition device per 500 square meters. A redundant data acquisition network is constructed. The core area adopts a 1-primary-2-backup device configuration, and the general area adopts a 1-primary-1-backup configuration to ensure that the area coverage is not affected when a single device fails.
[0049] Step 7.2: Scheduling model construction and parameter acquisition. Three types of data are collected in real time: device operating status, battery level, and environmental information; a deep reinforcement learning algorithm is used to construct the device scheduling model.
[0050] Step 7.3: Execution and dynamic adjustment of scheduling strategy. Based on the optimal scheduling strategy output by the equipment scheduling model, the working mode, acquisition parameters and shooting angle of the acquisition equipment are dynamically adjusted. When the equipment fails or encounters severe weather, the redundant backup equipment is immediately activated and switched to protection mode to ensure the continuity of inspection and data security.
[0051] In some embodiments, the construction of a deep reinforcement learning device scheduling model specifically includes: Step 8.1: Data preprocessing and state vector construction. The collected equipment operating status, battery level and environmental information are converted into quantitative indicators: the equipment operating status is encoded using one-hot encoding, and the battery level is directly quantified according to the remaining percentage; in the environmental information, wind speed, temperature, humidity and light intensity are quantified according to actual values. All quantitative indicators are normalized to the [0,1] interval using min-max normalization. Combined with the inspection area priority, a 10-dimensional standardized state vector is constructed.
[0052] Step 8.2: Definition of State Space, Action Space, and Reward Function. The state space is a continuous space composed of 10-dimensional standardized state vectors, covering equipment, energy consumption, environmental, and regional priority information; the action space includes four types of actions: switching equipment operating modes, adjusting acquisition parameters, fine-tuning shooting angles, and starting and stopping redundant equipment; the reward function adopts a weighted summation method, and the weight coefficients are optimized through grid search.
[0053] Step 8.3: PPO model construction. A deep reinforcement learning model is constructed using the proximal policy optimization algorithm. The model includes an input layer, two fully connected hidden layers, and an output layer.
[0054] Step 8.4: Model Training and Optimization. A training dataset is constructed based on historical scheduling data from the past 3 months, equipment operation logs, and simulated extreme environment data. The dataset is divided into training and validation sets in an 8:2 ratio. Training parameters are set as follows: initial learning rate of 0.001, cosine annealing strategy is used to adjust the learning rate, 500 iterations, and batch size of 64. The training set is input into the model for training. The reward value changes are monitored in real time through the validation set. An early stopping strategy is adopted to avoid model overfitting. After training, the model can output the optimal scheduling action based on the real-time state vector. At the same time, the model parameters are fine-tuned every 7 days using newly added actual scheduling data to improve the model's adaptive capability.
[0055] In some embodiments, the prediction of abnormal development trends specifically includes: Step 9.1: Construction of the time series dataset for abnormal development. Based on the interface of the equipment digital twin model, historical data of the abnormal location is retrieved, including the operation data of the past 24 months, the inspection data of each inspection, and the maintenance records. The retrieved data is cleaned and preprocessed: outliers are removed using the 3σ criterion, missing values are filled using linear interpolation, and the data is sorted by time series to construct the initial time series dataset.
[0056] Step 9.2: Prediction model construction and training. The prediction model is constructed using LSTM combined with an attention mechanism.
[0057] Step 9.3: Output and application of prediction results. Input the preprocessed time series dataset into the trained prediction model, output the quantitative value of the development degree of the anomaly within the preset period and the boundary parameters of the impact range, incorporate the prediction results into the final inspection results, divide the operation and maintenance priorities according to the anomaly development speed and impact range, arrange maintenance for high-priority anomalies immediately, and formulate regular tracking plans for medium and low-priority anomalies.
[0058] In some embodiments, the construction and inference of the LSTM-attention prediction model specifically includes: Step 10.1: Prediction Model Construction. A prediction model combining LSTM and self-attention mechanism is constructed. The model structure consists of five layers: Input layer, which receives time-series data with a 12-dimensional input dimension, corresponding to 12 key parameters; LSTM feature extraction layer, with 3 hidden layers, each with 128, 64, and 32 neurons, using the tanh activation function and a dropout rate of 0.2, used to extract time-series features; Self-attention enhancement layer, using a multi-head attention mechanism with 8 heads and a 32-dimensional hidden layer, used to enhance the feature weights of key time-series nodes; Fully connected prediction layer, with 2 fully connected layers, each with 32 and 16 neurons, using the ReLU activation function; Output layer, with 3 neurons, outputting the quantified values of the abnormal development degree and the boundary parameters of the influence range for 1 month, 3 months, and 6 months, respectively.
[0059] Step 10.2: Model training. Set the model training parameters: the optimizer uses adaptive moment estimation, the initial learning rate is set to 0.001, and the weight decay coefficient is set to 0.0001; the loss function uses mean squared error to measure the deviation between the predicted value and the actual value; the constructed abnormal development time series dataset is divided into training set and validation set in a 7:3 ratio, the batch size is set to 16, and the number of iterations is set to 200 rounds; the prediction accuracy is monitored through the validation set, and an early stopping strategy is adopted. If the MAE of the validation set does not decrease for 15 consecutive rounds, training is stopped to avoid model overfitting.
[0060] Step 10.3: Model Inference and Result Optimization. Input the time series data of the anomaly to be predicted into the trained model, and output the quantitative value of the development degree of the anomaly within the preset period and the boundary parameters of the impact range. Combine the structural parameters and operating threshold of the device digital twin model to verify the rationality of the prediction results. If the predicted value exceeds the 95% confidence interval of the model, an expert experience correction coefficient is introduced to adjust the prediction results, and finally an accurate prediction result of the anomaly development trend is obtained.
[0061] In summary, the advantages of this invention are: The inspection method based on multispectral fusion combines information from different bands such as visible light, infrared, and ultraviolet, which helps to comprehensively capture the operating status and abnormal characteristics of equipment. Compared with traditional manual inspection, this intelligent inspection method can improve inspection efficiency and monitor around the clock. By using deep learning models, especially the improved deep learning algorithm that combines attention mechanisms and YOLOv8 with VisionTransformer, abnormal features of equipment can be accurately identified. These features include cracks on the equipment surface, abnormal temperatures, and other potential faults, which can greatly improve the accuracy and timeliness of fault prediction for wind power generation equipment. By combining high-precision 3D geographic information of wind power stations with digital twin models of equipment and using iterative nearest point algorithms for spatial coordinate calibration, accurate equipment locations can be provided for inspection reports. This not only improves the quality of inspection reports but also enables more precise equipment positioning, facilitating subsequent maintenance and repair. After the inspection report is generated, the report is optimized using a cross-validation method based on multi-source data. If the accuracy of the report is insufficient, the system can automatically initiate secondary data collection to ensure the authenticity and validity of the inspection report. This verification and optimization mechanism further enhances the credibility of the inspection results. Based on the three-dimensional geographic information model of the wind power station, the system can dynamically adjust the working status and acquisition parameters of the acquisition equipment. Through deep reinforcement learning algorithms, the system can intelligently schedule the acquisition equipment according to the real-time status and automatically switch to backup mode when the equipment fails or the environment changes, thereby ensuring the efficient and stable operation of the system. The prediction model, which combines LSTM and attention mechanism, can predict anomalies in wind power equipment in a timely manner, provide early warning of the development trend and impact range of potential faults, and provide more accurate priority basis for equipment maintenance and management, thus avoiding the escalation of equipment faults and delayed response. Based on weighted fusion of multispectral data and multi-level deep learning algorithms, the system has high robustness and can adapt to the inspection needs of wind power stations under different environmental conditions. At the same time, the adaptive scheduling strategy and intelligent data augmentation method enable the system to operate efficiently in various complex environments.
[0062] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims
1. A method for intelligent video inspection of wind power stations based on multispectral fusion, characterized in that, include: Based on the overall topology of the wind power station, a distributed multispectral video acquisition network is deployed to simultaneously acquire multispectral raw video streams containing visible light, infrared and ultraviolet bands, resulting in a multidimensional multispectral raw video set. The original multispectral video set is optimized using a band-adaptive preprocessing strategy to obtain a standardized multispectral video dataset. The standardized multispectral video dataset is subjected to cross-band feature deep fusion using a weighted multispectral fusion model to generate a multi-dimensional fused video feature set that combines spatial details and spectral differences. The multi-dimensional fused video feature set is input into a deep learning inspection model based on an attention mechanism for training and inference, to identify and classify abnormal features of wind power generation equipment. The identified abnormal features are fused with the high-precision three-dimensional geographic information of the wind power station and the digital twin model of the equipment to perform spatial coordinate calibration and positioning, generate an inspection report, verify and optimize the inspection report, and obtain the final inspection results.
2. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 1, characterized in that, The original multispectral video dataset is optimized using a band-adaptive preprocessing strategy to obtain a standardized multispectral video dataset, including: Frame extraction and deduplication are performed on the original multispectral video set to obtain single-frame image sequences corresponding to each band, forming a multiband image set. Differentiated noise reduction processing is adopted based on the noise characteristics of images in each band: the visible light band adopts the noise intensity adaptive median filtering algorithm, the infrared band adopts the improved wavelet threshold noise reduction algorithm, and the ultraviolet band adopts the bilateral filtering and mean shift combined algorithm to accurately suppress noise. An improved scale-invariant feature transform algorithm is used to extract key feature points of images in each band, generate directional feature descriptors, and perform high-precision spatial alignment of multi-band images through feature matching and random sampling consensus algorithm. Pixel-level scale normalization is performed on the aligned multi-band image to map the pixel values of each band to a unified range and unify the image resolution to a preset standard size. Illumination compensation is performed on the uneven illumination regions of the normalized image to obtain a multispectral image sequence with uniform illumination. Add timestamps, device numbers, and acquisition location metadata to the processed multispectral image sequences to form a standardized multispectral video dataset.
3. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 2, characterized in that, The standardized multispectral video dataset is subjected to cross-band feature deep fusion using a weighted multispectral fusion model to generate a multi-dimensional fused video feature set that combines spatial details and spectral differences, including: A pixel-level and feature-level dual-layer weighted fusion framework is constructed to perform hierarchical progressive fusion on the standardized multispectral video dataset. Weighting coefficients are calculated based on the information entropy, signal-to-noise ratio, and edge sharpness of each band image. An improved weighted average algorithm is then used to sum the pixel values of the standardized multi-band images to obtain the initial fused image. The formula for the weighted average algorithm is as follows: ; In the formula, To merge images, For the first Each band image at location pixel values on It is the first Weighting coefficients for each band, It is the number of bands. It is the first Information entropy of an image band Information entropy for each band; Deep semantic features are extracted from the initial fused image based on a convolutional neural network, while shallow visual features of texture and edges of each single-band image are extracted through gray-level co-occurrence matrix and LBP operator. An attention mechanism is used to weightedly fuse deep semantic features and shallow visual features to enhance the feature response of abnormal regions. The kernel principal component analysis algorithm is used to reduce the dimensionality of the fused features, generating a multi-dimensional fused video feature set.
4. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 3, characterized in that, The multi-dimensional fused video feature set is input into a deep learning inspection model based on an attention mechanism for training and inference, to identify and classify abnormal features of wind power generation equipment, including: Construct a deep learning inspection model, which includes a feature extraction layer, a feature enhancement layer, an anomaly detection layer, and a classification layer; The multi-dimensional fused video feature set was divided into a training set, a validation set, and a test set in a 7:2:1 ratio, and the training set samples were expanded using a data augmentation strategy. Set the model training parameters, input the training set into the model for training, monitor the model performance through the validation set, and use an early stopping strategy to avoid overfitting. The test set is input into the trained model for inference, identifying the anomaly types of wind power generation equipment and outputting the anomaly confidence level, thereby identifying and classifying the anomaly features.
5. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 4, characterized in that, The identified anomalies are fused with high-precision 3D geographic information of the wind power station and a digital twin model of the equipment for spatial coordinate calibration and positioning, generating an inspection report, including: Acquire a high-precision 3D geographic information model and digital twin model of the wind power station, and extract the fused video frames containing abnormal features and the positioning and attitude parameters of the corresponding acquisition equipment; Based on the intrinsic parameters, extrinsic parameters, and distortion parameters of the acquisition device, the pixel coordinates of abnormal features are converted into world coordinates, and a coordinate mapping relationship with the 3D geographic information model is established. By combining the digital twin model of the equipment and using the iterative nearest point algorithm to calibrate the world coordinates, the precise spatial location coordinates of the anomaly feature are obtained. Integrate anomaly type, precise coordinates, data collection time, and confidence level information to generate a standardized inspection report.
6. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 5, characterized in that, The inspection report is verified and optimized to obtain the final inspection results, including: Retrieve historical inspection data, real-time equipment operation data, and environmental monitoring data of the abnormal area, and perform multi-source cross-validation with the inspection report; If the cross-validation consistency exceeds the preset threshold, the inspection report is confirmed to be valid; if the consistency does not meet the standard, the corresponding area multispectral equipment is activated for secondary acquisition, and the inspection report is regenerated. The information of the verified inspection reports is supplemented and the format is standardized to obtain the final inspection results.
7. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 6, characterized in that, The method also includes an adaptive scheduling strategy for the multispectral acquisition network, specifically including: Based on the three-dimensional geographic information model of the wind power station, core and general inspection areas are divided, and a redundant data acquisition network is constructed. Real-time data collection of device operating status, battery level, and environmental information; and the construction of a device scheduling model using deep reinforcement learning algorithms. Based on the scheduling strategy output by the equipment scheduling model, the working mode, acquisition parameters and angle of the acquisition equipment are adjusted, and redundant backup and protection modes are activated in the event of equipment failure or severe weather.
8. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 7, characterized in that, Real-time data collection of device operating status, battery level, and environmental information; and the construction of a device scheduling model using deep reinforcement learning algorithms, specifically including: The device's operating status, battery level, and environmental information are converted into quantitative indicators. One-hot encoding is used to process the categorized data to form a standardized state vector. Define the state space, action space, and reward function of the deep reinforcement learning model. The state space includes standardized state vectors and inspection area priorities. The action space includes switching of device working modes, adjusting acquisition parameters, fine-tuning of shooting angle, and starting and stopping redundant devices. The reward function adopts a weighted summation method, and the weight coefficients are optimized through grid search. A deep reinforcement learning model is constructed using a proximal policy optimization algorithm. The input layer receives a standardized state vector, and the output layer outputs the probability distribution of each action. A training dataset is constructed based on historical scheduling data, equipment operation logs, and simulated extreme environment data. It is divided into a training set and a validation set in an 8:2 ratio. The training set is input into the model for training, and the model reward value is monitored in real time through the validation set. After training, the model outputs the optimal scheduling action based on the real-time state vector.
9. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 8, characterized in that, After obtaining the precise spatial coordinates of the anomalous features, the method further includes anomaly development trend prediction, specifically including: Based on the digital twin model of the equipment, historical operation, inspection and maintenance data of the abnormal location are retrieved to construct an anomaly development time series dataset; A prediction model is constructed using LSTM combined with an attention mechanism. The model is input into a time series dataset to predict the future development and impact of anomalies. The prediction results are then incorporated into the final inspection results to provide a priority basis for operation and maintenance.
10. The method for intelligent video inspection of wind power stations based on multispectral fusion according to claim 9, characterized in that, The specific steps of constructing a prediction model using LSTM combined with an attention mechanism, and inputting a time-series dataset to predict the future development and impact range of anomalies, include: A prediction model combining LSTM and self-attention mechanism is constructed. The model includes an input layer, an LSTM feature extraction layer, a self-attention enhancement layer, a fully connected prediction layer, and an output layer. The LSTM feature extraction layer has 3 hidden layers, and the self-attention layer uses a multi-head attention mechanism to enhance key temporal features. Set the model training parameters, adopt the adaptive moment estimation optimizer, use the mean squared error as the loss function, input the time series dataset into the model for training, and monitor the prediction accuracy by verifying the sample. Input the time series data of the anomaly to be predicted into the trained model, output the quantitative value of the anomaly's development degree and the boundary parameters of its impact range within a preset future period, and obtain the anomaly development trend prediction result.