Unmanned aerial vehicle inspection system multi-modal data fusion and intelligent analysis platform and method for wind power plant

Abstract

The invention discloses a multi-modal data fusion and intelligent analysis platform and method for an unmanned aerial vehicle inspection system for a wind power plant. The platform comprises a multi-modal data acquisition module, a feature extraction and standardization module, a multi-modal information fusion module, a joint learning and optimization module, a domain knowledge injection module and an intelligent decision and application module. The system processes multi-source heterogeneous data through an integrated learning and deep learning fusion strategy, projects features to a shared semantic space by using joint training and comparative learning to enhance the anomaly discrimination ability, and performs verification and semantic enhancement on a supervised retrieval result in combination with a knowledge base in the wind power field. And finally, outputting a high-reliability diagnosis report and a maintenance suggestion. According to the invention, accurate identification and positioning of the fan fault are realized, and the inspection efficiency and the system decision reliability are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for wind power equipment, and in particular to a multimodal data fusion and intelligent analysis platform and method for unmanned aerial vehicle (UAV) inspection systems for wind farms. Background Technology

[0002] With the rapid development of the wind power industry and the continuous expansion of wind turbine scale, the operational safety and maintenance efficiency of wind turbines are receiving increasing attention. Traditional manual inspection methods suffer from low efficiency, high risk, and strong subjectivity, making it difficult to meet the operation and maintenance needs of large-scale wind farms. In recent years, drone inspection technology has been gradually applied to the field of wind farm equipment monitoring due to its flexibility and high efficiency.

[0003] Currently, most drone-based inspection systems use a single type of sensor for data acquisition, such as visible light cameras or infrared thermal imagers, which suffers from limited data dimensionality and incomplete feature extraction. At the data analysis level, existing methods largely rely on single machine learning models, which lack the ability to identify complex fault modes and effectively mine the deep correlations between multimodal data. Furthermore, most systems remain at the data-driven level, failing to integrate domain expert knowledge and historical maintenance experience into the diagnostic decision-making process, resulting in a high misjudgment rate for early faults and complex abnormal states.

[0004] While existing research has attempted to employ multi-sensor fusion technology, the spatiotemporal alignment of different modal data is often overlooked during feature-level fusion, and quantitative assessment of model uncertainty is lacking during decision-level fusion. In particular, the recognition accuracy and reliability of existing systems still need improvement when facing typical faults such as wind turbine blade cracks, loose bolts, and bearing overheating.

[0005] Therefore, there is an urgent need for an intelligent inspection and analysis platform that can effectively integrate multimodal data, has continuous learning and optimization capabilities, and can integrate domain knowledge. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an intelligent inspection and analysis platform that can effectively fuse multimodal data, has continuous learning and optimization capabilities, and can integrate domain knowledge.

[0007] In a first aspect, embodiments of this application provide a multimodal data fusion and intelligent analysis platform for a drone inspection system for wind farms, comprising: The multimodal data acquisition module is used to simultaneously acquire multi-source heterogeneous data generated by the UAV during the wind farm inspection process. The data includes visible light images, infrared thermal imaging, laser point clouds, audio signals, and UAV flight status data. The feature extraction and standardization module is used to clean and align the multi-source heterogeneous data and extract standardized feature vectors, which include visual features, three-dimensional structural features, audio temporal features, and motion features. The multimodal information fusion module is used to process the standardized feature vectors using a fusion strategy that combines ensemble learning and deep learning to generate preliminary assessment results of the wind turbine equipment status and preliminary positioning information of the UAV. The joint learning and optimization module is used to project multimodal features into a shared semantic space through a joint training strategy, and introduces a contrastive learning mechanism to enhance the model's ability to discriminate abnormal wind turbine states. The domain knowledge injection module integrates a wind power domain knowledge base, which is used to call relevant historical fault cases and equipment parameters through a supervised retrieval mechanism to verify and semantically enhance the preliminary assessment results, and output highly reliable diagnostic and location results. The intelligent decision-making and application module is used to automatically generate inspection reports and maintenance suggestions based on the high-reliability diagnostic and positioning results, and provides a visual and intelligent question-and-answer interactive interface.

[0008] Secondly, embodiments of this application provide a method for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power, applied to the multimodal data fusion and intelligent analysis platform for unmanned aerial vehicle (UAV) inspection systems for wind farms as described in the first aspect, including: Receive multi-source heterogeneous data synchronously collected by the UAV, including visible light images, infrared thermal imaging, laser point clouds, audio signals, and UAV flight status data; Feature extraction and standardization are performed on the multi-source heterogeneous data to obtain standardized multimodal feature vectors; The standardized multimodal feature vectors are processed using a fusion strategy that combines ensemble learning and deep learning to generate preliminary evaluation results and preliminary localization information. By combining training and contrastive learning, multimodal features are projected onto a shared semantic space for optimization, thereby enhancing the ability to detect abnormal states. Based on the optimized evaluation results, relevant historical fault cases and equipment parameters are retrieved from the wind power knowledge base. The preliminary evaluation results are then verified and semantically enhanced to obtain high reliability diagnosis and location results. Based on the highly reliable diagnostic and location results, inspection reports and maintenance suggestions are automatically generated, and users are provided with visual information and intelligent Q&A services through an interactive interface.

[0009] Thirdly, embodiments of this application provide an electronic device, including: processor; Memory used to store processor-executable instructions; The processor is configured to implement, when executing the instructions, the multimodal data fusion and intelligent analysis platform for the unmanned aerial vehicle inspection system for wind farms as described in the first aspect and the method described in the second aspect.

[0010] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program that instructs a device to execute the multimodal data fusion and intelligent analysis platform for a wind farm-oriented UAV inspection system as described in the first aspect and the method described in the second aspect.

[0011] The beneficial effects of this invention are as follows: 1. Improve fault identification accuracy and reliability: By combining ensemble learning and deep learning with joint learning and contrastive learning mechanisms, the ability to identify early faults and complex abnormal states of wind turbines is significantly enhanced.

[0012] 2. Achieve effective fusion of multi-source data: By adopting a multi-modal data acquisition and feature alignment method with a unified spatiotemporal benchmark, the problem of fusion of multi-source heterogeneous data is solved, and the complementary advantages of different modal data are fully utilized.

[0013] 3. Enhance the intelligence of decision-making: By injecting domain knowledge, expert knowledge such as historical failure cases and equipment parameters is integrated into the diagnostic process, which improves the interpretability of diagnostic results and the reliability of decision-making.

[0014] 4. Improve inspection and maintenance efficiency: Automatically generate inspection reports and maintenance suggestions, combined with intelligent Q&A and visual interactive functions, which greatly reduces the workload of manual analysis and improves the efficiency of wind farm operation and maintenance.

[0015] 5. Strong environmental adaptability: Through multimodal data complementarity and knowledge enhancement mechanisms, the limitations of a single data source in complex environments are effectively overcome, and the robustness of the system in real-world scenarios is improved. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the architecture of a multimodal data fusion and intelligent analysis platform for a wind farm drone inspection system provided in an embodiment of this application.

[0017] Figure 2 A schematic diagram illustrating the method for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power provided in this application.

[0018] Figure 3 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0020] It should be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0021] Based on the embodiments described in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] Example 1 Figure 1 This is a schematic diagram of the multimodal data fusion and intelligent analysis platform architecture for a wind farm drone inspection system provided in one embodiment of this application. Figure 1 As shown, a multimodal data fusion and intelligent analysis platform for unmanned aerial vehicle (UAV) inspection systems in wind farms includes: a multimodal data acquisition module 110, a feature extraction and standardization module 120, a multimodal information fusion module 130, a joint learning and optimization module 140, a domain knowledge injection module 150, and an intelligent decision-making and application module 160. The functions of each module are as follows: The multimodal data acquisition module 110 is used to simultaneously acquire multi-source heterogeneous data generated by the UAV during wind farm inspections. This data includes visible light images, infrared thermal imaging, laser point clouds, audio signals, and UAV flight status data. This module simultaneously acquires multi-source heterogeneous raw data and controls various sensors on the UAV (visible light camera, infrared thermal imager, lidar, microphone, IMU / GNSS, etc.) to simultaneously obtain multi-dimensional raw data such as images, temperature, 3D structure, sound, and the UAV's own flight status, including those of the wind turbine and its environment. This is the fundamental data source for the entire system.

[0023] Specifically, in this embodiment, the multimodal data acquisition module 110 further includes a visible light and infrared imaging unit, a three-dimensional environment perception unit, an acoustic signal acquisition unit, and a flight status perception unit that work together.

[0024] The visible light and infrared imaging unit includes a high-resolution visible light camera and an infrared thermal imager, which are used to collect texture detail images and temperature distribution data of the surface of wind turbine components, respectively. The high-resolution visible light camera is used to capture visible defects such as texture, color, cracks, and corrosion of wind turbine blades, towers, and other components. At the same time, the infrared thermal imager is used to capture temperature distribution data of the same area to detect invisible problems such as overheating of electrical joints and abnormal heating caused by internal defects.

[0025] The three-dimensional environment perception unit includes a lidar system (LiDAR) for acquiring high-precision three-dimensional point cloud data of the wind turbine blades, tower, and surrounding environment; and for acquiring high-precision three-dimensional spatial information of the inspection target and its surrounding environment. The lidar actively emits laser beams and receives echoes to generate dense three-dimensional point cloud data. This data accurately describes the three-dimensional geometric structure and spatial relationships, such as wind turbine blade deformation, tower tilt, and distances to other objects.

[0026] The acoustic signal acquisition unit includes a directional microphone array for acquiring audio signals during wind turbine operation and possessing environmental wind noise suppression capabilities; it acquires audio signals emitted during wind turbine operation. The directional microphone array is used to directionally acquire the sound generated by wind turbine operation (such as gearbox, generator, and blade rotation). Its environmental wind noise suppression function is crucial, effectively filtering out wind noise interference to extract pure audio characteristics that reflect the mechanical condition of the equipment (such as looseness, wear, and imbalance).

[0027] The flight status perception unit is integrated into the UAV flight control system. It is used to acquire the UAV's three-dimensional position, attitude, and motion status data in real time via the inertial measurement unit (IMU) and a Global Navigation Satellite System (GNSS) receiver; it also senses and records the UAV's spatial position and motion attitude in real time. Integrated into the UAV flight control system, it measures the UAV's angular velocity and acceleration (attitude) through the IMU and obtains its absolute three-dimensional coordinates through the GNSS receiver. This data is not only used for UAV navigation but, more importantly, provides a precise spatial positioning reference for all other sensor data.

[0028] The multimodal data acquisition module also includes a data synchronization control unit, used to generate unified timestamps and spatial positioning tags for all sensor data streams to establish a synchronized acquisition sequence. This establishes a unified spatiotemporal reference for all sensor data streams, a core prerequisite for effective multimodal data fusion. This unit assigns a unified timestamp and spatial positioning tag to each frame of data (such as images, point clouds, and audio clips) from the four sensing units mentioned above. This ensures that during subsequent processing, a visible light image, an infrared thermal image, a laser point cloud, and an audio signal are acquired at the same time and from the same spatial location, thus solving the most critical spatiotemporal synchronization problem in multi-sensor data fusion.

[0029] Furthermore, to address the data mismatch issue caused by multiple sensors under complex operating conditions such as strong winds and drone vibrations, a dynamic optimization sampling strategy is introduced into the multimodal data acquisition module. A sensor scheduling cost function is introduced, with the following formula: , in, : The set of sensor control strategies at time t. : The trace of the covariance matrix of the i-th sensor prediction, representing the uncertainty measure. Penalty for the spatial distance between the current location of the drone and the target inspection component. Energy consumption estimation for sensor arrays. Weighting coefficients enable multi-objective trade-off optimization of sensing accuracy, positioning accuracy, and battery life.

[0030] Four specialized sensing units acquire data on appearance, temperature, 3D structure, sound, and self-positioning, respectively. A core synchronous control unit ensures that all these data are aligned in time and space, laying a solid and reliable data foundation for subsequent feature extraction and multimodal fusion.

[0031] The feature extraction and standardization module 120 is used to clean and align the multi-source heterogeneous data and extract standardized feature vectors, including visual features, three-dimensional structural features, audio temporal features, and motion features. This module transforms the raw data into unified and comparable standardized features. It refines the messy raw data collected upstream, including data cleaning (denoising and outlier removal) and spatiotemporal alignment. Then, it extracts features that represent the essential information of each modality (such as visual texture, three-dimensional geometry, audio spectrum, etc.). Finally, through normalization and dimensionality reduction, all features are unified to the same scale and dimension, preparing for subsequent fusion.

[0032] Specifically, in this embodiment, the feature extraction and standardization module includes the following collaborative processing units: The data cleaning and alignment unit is used to remove outliers and impute missing values ​​in the multimodal raw data, and to perform spatiotemporal alignment of sensor data with different sampling frequencies based on a unified timestamp; it preprocesses the raw data to ensure data quality and spatiotemporal consistency. This is the data preparation step before feature extraction. It is responsible for removing outliers (such as sensor transient errors), imputes missing values, and aligning various sensor data with different sampling frequencies (such as 30Hz for images, 44.1kHz for audio, and 100Hz for IMU) onto the same time axis based on a unified timestamp provided by the data acquisition module, ensuring that the features extracted subsequently are corresponding in time and space.

[0033] The visual feature extraction unit employs a pre-trained convolutional neural network to extract texture features from visible light images and temperature distribution features from infrared thermal imaging data, fusing the two into a comprehensive visual feature vector. It also extracts visual features characterizing the device's status from both visible light and infrared images. This unit processes the two types of images in parallel. It extracts texture, edge, and shape features from visible light images to identify physical damage (such as cracks and spalling); and it extracts temperature distribution and hotspot areas from infrared thermal imaging data to identify overheating anomalies. Finally, it fuses visual information from these two different sources into a single comprehensive visual feature vector.

[0034] The 3D structural feature extraction unit uses a PointNet++-based graph neural network to process laser point cloud data, extracting 3D geometric features of wind turbine blade surface deformation and tower structural integrity; it also extracts features characterizing the 3D geometric shape of wind turbine components from the laser point cloud data. It processes the raw 3D point cloud acquired by lidar. Through an advanced point cloud processing network, it learns and extracts 3D spatial geometric features that reflect problems such as wind turbine blade surface deformation, tower structural integrity, and missing or loose bolt components.

[0035] The audio timing feature extraction unit converts audio signals into time-frequency features using Mel-frequency cepstral coefficients and short-time Fourier transform, and then extracts timing pattern features representing the operating state of equipment using a long short-time memory network. This unit first converts the sound signal into a time-spectrum graph (e.g., using Mel-frequency cepstral coefficients MFCC), and then uses models adept at processing sequential data (e.g., long short-time memory networks LSTM) to capture patterns from the dynamic changes of the sound. This is used to identify abnormal vibration or noise patterns caused by faults such as bearing wear and blade imbalance.

[0036] The motion feature extraction unit extracts kinematic features, including angular velocity variance and acceleration spectrum characteristics, from UAV flight status data to characterize flight stability and environmental disturbances. It also extracts features characterizing flight stability and environmental impact from the UAV flight status data. The unit analyzes IMU and GNSS data. By calculating the variance of angular velocity and the spectrum characteristics of acceleration, this unit can quantify the flight stability of the UAV and indirectly sense the intensity of environmental wind disturbances. These features help determine the environmental conditions during data acquisition and provide auxiliary information for positioning.

[0037] The feature standardization unit performs max-min normalization on each extracted feature type to eliminate the influence of dimensions, and then uses principal component analysis for dimensionality reduction, ultimately outputting a standardized feature vector with a unified dimension. This unifies the extracted heterogeneous features into a standard format with consistent scale and dimension. This is a crucial step to ensure the effectiveness of subsequent fusion. It normalizes the features output by all the above units, which have different dimensions and numerical ranges (e.g., scaling them to the [0,1] interval), to eliminate the influence of dimensions. Subsequently, methods such as principal component analysis (PCA) are used for dimensionality reduction, retaining most of the key information while reducing data redundancy and computational complexity, ultimately outputting a feature vector with a unified dimension and standardized scale for use by downstream fusion modules.

[0038] Furthermore, to address the challenge of directly fusing features with different physical dimensions (such as images, point clouds, and audio), the feature extraction and normalization module 120 introduces manifold alignment technology, wherein the manifold alignment loss function is: , in, , : The feature matrices of the m-th and n-th modes. The association weight between modes m and n is determined by the mutual information between modes. Frobenius norm: measures the consistency of feature distributions across different modalities. : Graph Laplace regularization term, preserving the original data structure.

[0039] The multimodal information fusion module 130 processes the standardized feature vectors using a fusion strategy combining ensemble learning and deep learning to generate preliminary assessment results of the wind turbine equipment status and preliminary location information of the UAV. This module performs preliminary fusion and inference on the standardized features to generate a preliminary judgment. It employs a hybrid strategy of ensemble learning and deep learning to extract information from the features from different perspectives. Ensemble learning utilizes the collective wisdom of multiple models to vote and arrive at robust preliminary conclusions; deep learning, on the other hand, uses attention mechanisms and other methods to uncover deep, cross-modal, complex correlations. Finally, this module outputs preliminary assessment results of the wind turbine status and the UAV's location.

[0040] Specifically, in this embodiment, the multimodal information fusion module adopts a hybrid fusion architecture combining ensemble learning and deep learning, specifically including: The ensemble learning fusion submodule constructs a heterogeneous set of base learners consisting of Support Vector Regression (SVR), Extreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and K-Nearest Neighbors (KNN). Each base learner processes the standardized feature vectors in parallel, generating preliminary predictions of the device state. A dynamic weighted fusion strategy based on the inverse of the mean squared error (MSE) is then used to integrate these preliminary predictions, outputting preliminary UAV positioning information and device state assessment. Robust preliminary predictions are achieved by leveraging the collective intelligence of multiple heterogeneous models. This submodule deploys multiple different types (heterogeneous) base learners (such as SVR, XGBoost, MLP, and KNN) in parallel. Each base learner independently learns and predicts the standardized feature vectors, generating preliminary predictions of the device state and UAV location. Subsequently, a dynamic weighted fusion strategy (e.g., assigning weights based on the inverse of the MSE of each model) is used to integrate these preliminary results. Its advantage lies in reducing the risk of overfitting through model diversity, resulting in a stable and reliable preliminary assessment.

[0041] The deep learning fusion submodule employs a Transformer encoder based on a multi-head attention mechanism to perform cross-modal interaction and deep semantic fusion on the standardized feature vectors. It captures dependencies within each modality through self-attention and establishes semantic associations between different modalities, such as visual features, 3D point cloud features, and audio features, through cross-attention, generating a fused feature representation rich in contextual information. This submodule also mines deep, non-linear semantic relationships between multimodal features. This submodule uses a deep neural network based on an attention mechanism to process all standardized features at a deeper level. It captures feature dependencies within a single modality through self-attention and actively explores and establishes semantic connections between features from different modalities (such as visual, point cloud, and audio) through cross-attention. Ultimately, it outputs a deeply fused feature representation rich in contextual information, containing complex patterns that are difficult for a single model to capture.

[0042] The hybrid decision submodule, connecting the ensemble learning fusion submodule and the deep learning fusion submodule, performs feature-level concatenation of the preliminary evaluation and localization information output by the ensemble learning submodule and the fusion feature representation generated by the deep learning submodule. This concatenation is then performed through a fully connected decision network for final inference, generating a preliminary evaluation result of the wind turbine equipment status and preliminary localization information of the UAV. Finally, the robust results of the ensemble learning and the deep information from the deep learning submodule are combined to make the final decision.

[0043] This submodule is the decision center of the hybrid architecture. It receives information from the two paths mentioned above: one is the preliminary evaluation and location information based on model voting output from the ensemble learning fusion submodule; the other is the fusion feature representation containing complex semantics generated by the deep learning fusion submodule. This submodule combines these two types of information (e.g., through feature concatenation) and performs inference through a final decision network (such as a fully connected network) to generate the final evaluation result of the wind turbine equipment status and the drone's location information.

[0044] These three sub-modules collectively implement a complementary fusion strategy: an ensemble learning path providing breadth and robustness; a deep learning path providing depth and semantic understanding; and a hybrid decision-making process that combines breadth and depth to achieve precise decisions where 1+1>2. This mechanism ensures that the system neither becomes too complex and instable, nor too simple and unable to handle complex failure modes.

[0045] The joint learning and optimization module 140 projects multimodal features into a shared semantic space through a joint training strategy and introduces a contrastive learning mechanism to enhance the model's ability to discriminate abnormal wind turbine states. This module enhances the model's ability to distinguish abnormal states within the shared semantic space. This is the model's optimization training stage. It receives initially fused features and, through a contrastive learning mechanism, deliberately brings the features of normal samples closer together while pushing away the features of abnormal samples within a low-dimensional shared semantic space. This makes the optimized model more capable of distinguishing between normal and abnormal states, especially adept at detecting subtle early faults.

[0046] Specifically, in this embodiment, the joint learning and optimization module includes: The shared semantic space mapping unit projects the fused feature vectors from the multimodal information fusion module into a unified low-dimensional shared semantic space using a jointly trained multilayer perceptron network. This unit receives the pre-fused feature vectors from the upstream multimodal information fusion module. It transforms these related features from different modalities into a new, common low-dimensional space using a learnable mapping function (such as a neural network). In this space, the distance between data points directly reflects their semantic similarity, laying the foundation for subsequent comparative optimization.

[0047] The contrastive learning optimization unit, connected to the shared semantic space mapping unit, implements a contrastive learning mechanism within the shared semantic space. This mechanism operates by constructing triplet samples—using a normal wind turbine state sample as the anchor point, similar normal state samples as positive samples, and known fault or abnormal state samples as negative samples—and utilizing a triplet loss function to optimize the network parameters of the shared semantic space mapping unit. This achieves the optimization goal of narrowing the feature distance between the anchor point and positive samples while simultaneously widening the feature distance between the anchor point and negative samples. The contrastive learning mechanism optimizes the structure of the shared semantic space to increase the feature distance between normal and abnormal states. This is the core optimization step of the module. It employs a triplet learning strategy: starting from an anchor sample (such as data from a normal wind turbine), it selects a sample belonging to the same normal state as a positive sample and a known abnormal state sample as a negative sample. By optimizing the loss function, the parameters of the shared semantic space mapping unit are continuously adjusted so that in the final semantic space, the feature distance between the anchor point and positive samples is minimized, while the feature distance between the anchor point and negative samples is maximized. After training with a large number of such triples, the unit enables normal state sample points to cluster tightly in the entire shared semantic space, while sample points of various abnormal states are pushed away from the normal cluster, and clear boundaries are formed between different abnormal states.

[0048] Specifically, the contrastive learning optimization unit optimizes the network parameters of the shared semantic space mapping unit by minimizing an improved triplet loss function, which is defined as follows: , in, The standard triplet loss is calculated using the following formula: , in, This represents an anchor point sample, i.e., a sample of a wind turbine in normal condition. Indicates a positive sample, and They belong to the same normal state category; This indicates a negative sample, representing a known wind turbine fault or abnormal state sample. The function mapping represents the shared semantic space mapping unit; It is a preset boundary hyperparameter used to control the minimum interval between positive and negative sample pairs; Let L be the L2 norm of the vector. Let N be the number of samples.

[0049] at the same time, It is an introduced class center loss, and its calculation formula is: , in, Represents anchor point sample The feature center vector belonging to the normal state category is dynamically calculated and updated during training. λ is a hyperparameter that balances the contributions of the two losses. For the sample Category tags.

[0050] By jointly optimizing the total loss function The module can learn a more discriminative shared semantic space in which the features of normal state samples of the same type are highly clustered, while the features of different abnormal state samples are effectively separated, thereby significantly enhancing the ability to detect and distinguish early faults and rare abnormal states of wind turbines.

[0051] The anomaly detection enhancement unit, connected to the contrastive learning optimization unit, is used to process real-time inspection data using the optimized shared semantic space mapping unit. This results in a significantly increased inter-class distance and a significantly reduced intra-class distance between different categories of wind turbine anomaly state feature clusters in the semantic space, thereby significantly enhancing the ability to detect early wind turbine faults and rare anomaly states and improving model robustness.

[0052] Furthermore, the triplet loss in the original scheme is limited in effectiveness when there is extreme class imbalance (few outlier samples). The joint learning and optimization module 140 can introduce hyperspherical embedding and adaptive boundary, specifically a hyperspherical prototype contrastive loss, with the following formula: , Here, the features are mapped to a hyperspherical space, and s is a fixed value of the feature magnitude. It is the cosine similarity between sample i and its true class prototype. Temperature coefficient, used to adjust the concentration distribution. C Total number of categories. Dynamic margin: The margin α between normal and abnormal categories is no longer a fixed value, but is dynamically adjusted based on the intra-category clustering. N is the batch size. Let be the angle between the feature vector of the i-th sample and the prototype vector of the j-th class. This is the core of the hyperspherical model.

[0053] The domain knowledge injection module 150 integrates a wind power domain knowledge base. It uses a supervised retrieval mechanism to call relevant historical fault cases and equipment parameters to verify and semantically enhance the preliminary assessment results, outputting highly reliable diagnostic and location results. This module uses industry knowledge and historical experience to verify and enhance the model's judgments, incorporating expert experience. The module has a built-in wind power domain knowledge base (such as historical fault cases and equipment parameter manuals). Based on the model's preliminary judgment results, it supervisedly retrieves relevant cases and parameters from the knowledge base to verify the model's judgment. If consistent, the confidence level is increased; if contradictory, questions or alternative solutions are proposed. This greatly improves the reliability and interpretability of the diagnostic results.

[0054] Specifically, in this embodiment, the domain knowledge injection module 150 includes: The wind power knowledge base stores structured domain knowledge, including wind turbine equipment specifications, historical failure case databases, maintenance knowledge graphs, and typical environmental condition datasets. Each case in the historical failure case database includes a multimodal description of the failure phenomenon, root cause analysis, and the final maintenance solution. This dedicated database stores structured domain expertise and historical experience, including wind turbine equipment specifications (such as rated speed and normal temperature range), historical failure case databases (containing failure phenomena, causes, and solutions), maintenance knowledge graphs (relationships between failures), and typical environmental condition data. It provides an authoritative and searchable source of knowledge for the entire module.

[0055] The supervised retrieval mechanism unit, connected to the wind power knowledge base, receives the post-optimization evaluation results from the joint learning and optimization module. Using these results as a query request, it dynamically retrieves the most relevant historical fault cases and equipment technical parameters from the knowledge base using an attention-weighted multi-dimensional similarity matching algorithm. Based on the model's preliminary diagnostic results, the supervised retrieval mechanism unit accurately locates relevant information from the knowledge base. This unit receives the post-optimization evaluation results (i.e., the model's initial assessment of the fault type and location) from the joint learning and optimization module. It uses these results as a query request and employs an intelligent retrieval algorithm to dynamically and selectively filter the most relevant historical fault cases and equipment technical parameters from the vast knowledge base, rather than indiscriminately retrieving all information.

[0056] The knowledge verification and semantic enhancement unit, connected to the supervised retrieval mechanism unit, is used to fuse and reason with the retrieved relevant domain knowledge and the current evaluation result to achieve decision enhancement. This is the brain of the module. Based on the retrieved relevant knowledge, it executes at least one of the following three core logics: Consistency verification and confidence enhancement: If the current assessment result is consistent with the diagnostic conclusions of multiple relevant historical cases retrieved, the credibility of the diagnostic result is enhanced. This is equivalent to multiple experts endorsing the model's judgment, making the conclusion more reliable.

[0057] Conflict Detection and Review Suggestions: If the current assessment results conflict with typical failure modes in the knowledge base, a review result containing alternative diagnostic suggestions is generated. This is equivalent to an expert identifying discrepancies in the model's judgment and proposing other possibilities, effectively preventing misjudgments.

[0058] Physical interpretive correlation: This involves associating retrieved equipment parameters with currently detected abnormal features to provide engineering basis and physical explanation for fault diagnosis. For example, associating detected vibrations at a specific frequency with the model parameters of a wind turbine gearbox can explain the root cause of the fault.

[0059] The high-reliability diagnostic output unit integrates diagnostic information that has undergone knowledge verification and semantic enhancement to generate a final output that includes diagnostic conclusions, relevant case support, confidence assessment, and targeted maintenance suggestions, thus completing the final verification and semantic enhancement of the preliminary assessment results.

[0060] Furthermore, to enhance the physical interpretability and causality of diagnostic results, this module introduces a causal graph model for reasoning. Specifically, the causal verification based on Bayesian networks is formulated as follows: , Where E: the set of observed multimodal evidence (such as local overheating of the blade, abnormal noise in a specific frequency band). Prior probability of failure obtained from statistics from the knowledge base. The likelihood probability is determined by the fault-symptom association graph in the knowledge base. This model can quantify the confidence of different fault hypotheses and perform online Bayesian updates when new evidence is discovered, thereby continuously optimizing the diagnostic conclusions.

[0061] The above solution transforms the system from a pure data black box into an intelligent system capable of using industry-accumulated experience and principles to verify and explain its own judgments, ultimately significantly improving the reliability of diagnostic results and the level of intelligent decision-making.

[0062] Specifically, the supervised retrieval mechanism unit employs a knowledge retrieval strategy based on multi-objective optimization, achieving more accurate and intelligent relevant case retrieval through a multi-factor comprehensive scoring function. This unit no longer performs simple keyword or single similarity matching, but instead uses a comprehensive scoring function. This function acts like an intelligent filter, simultaneously considering multiple dimensions of factors during retrieval and weightedly evaluating the results. Its retrieval relevance scoring function is defined as follows: , in, Q The query vector represents the current optimized evaluation result. This represents the feature representation of the i-th historical case in the knowledge base. These are the weighting coefficients, and .

[0063] The specific definitions of the scoring functions for each item are as follows: Semantic similarity score This is used to measure the essential similarity between the current assessment results and historical cases in terms of fault characteristics and descriptions. The formula is as follows: .

[0064] Where σ is the Gaussian kernel width parameter, which controls the similarity decay rate.

[0065] Time decay score This is used to prioritize cases that are more recent in time, as the environment and equipment wear and tear status of recent cases are more valuable for reference. The formula is as follows: .

[0066] in, To adjust the parameters, and Representing the current time and case respectively Record time, Indicates the confidence level of the case. The variance represents the characteristics of the case. Time interval (e.g., number of days). e : Natural constant (approximately 2.718).

[0067] Confidence propagation score This is used to prioritize high-confidence cases with clear diagnoses and proven repair effectiveness in the past. The formula is: .

[0068] in, To adjust the parameters, Indicates the confidence level of the case. Example The characteristic variance. : The maximum value of the variance of features across all cases.

[0069] Through this multi-objective optimized retrieval, the system can more accurately and comprehensively find precedents that best match the current situation, significantly reducing the interference of noisy cases and providing higher-quality knowledge input for subsequent decision-making.

[0070] The knowledge verification and semantic enhancement unit employs an evidence theory fusion method to synthesize the current evaluation results with retrieved knowledge. It uses an uncertainty reasoning method to quantify the conflict and fusion between the preliminary diagnosis and domain knowledge. When the preliminary diagnosis results and retrieved knowledge need to be fused, this unit uses uncertainty reasoning methods (such as evidence theory, Bayesian methods, etc.). This method can: quantify uncertainty: treat both the preliminary diagnosis given by the data model and the retrieved historical knowledge as evidence with uncertainty, and assign them probabilities or confidence levels. Intelligently fuse conflicting evidence: even if the preliminary diagnosis conflicts or contradicts some historical case conclusions, this method can fuse these uncertain pieces of evidence through mathematical rules (such as the Dempster-Shafer synthesis rule) to arrive at a comprehensive, quantified final confidence level, rather than a simple binary choice. This gives the system a stronger processing capability when facing fuzzy, complex, or even contradictory diagnostic information. Its final decision is no longer black and white, but an optimal and reliable conclusion calculated based on all available information (including uncertain information), greatly enhancing the system's robustness in complex real-world scenarios. The formula is: , in This represents the basic probability allocation of the current evaluation results. This represents the basic probability allocation of retrieved knowledge. Through this synthesis rule, the uncertainty is quantitatively fused, and a highly reliable diagnostic result optimized by knowledge is output. A , B , C : Propositions or assumptions (such as fault types). The summation in the denominator represents the normalization process for all conflicting propositions.

[0071] The above steps improve the quality and efficiency of the domain knowledge injection process: In the retrieval stage, it upgrades from extensive matching to multi-factor intelligent screening. In the fusion stage, it upgrades from simple rule-based judgment to uncertainty-based quantitative reasoning. These two optimizations together ensure that the domain knowledge injection process is more scientific, accurate, and reliable, providing key technical support for improving the overall system's intelligent diagnostic capabilities.

[0072] The intelligent decision-making and application module 160 is used to automatically generate inspection reports and maintenance suggestions based on the high-reliability diagnostic and location results, and provides a visual and intelligent Q&A interactive interface. This module transforms the high-reliability diagnostic results into actual operation and maintenance actions. This is the terminal for system-user interaction. Based on the high-reliability diagnostic results produced by the aforementioned modules, it automatically generates structured inspection reports and actionable maintenance suggestions, and provides operation and maintenance personnel with intuitive data display and decision support through a visual interface and intelligent Q&A system, ultimately completing the closed loop from data to action.

[0073] Specifically, in this embodiment, the intelligent decision-making and application module includes: The report and decision generation unit is used to automatically generate inspection reports and output maintenance suggestions based on the high-reliability diagnosis and location results. Specifically, it includes: a dynamic report generation unit, used to automatically generate structured inspection reports based on the high-reliability diagnosis and location results; this unit integrates diagnostic conclusions, fault location, severity, relevant historical case evidence, and maintenance priorities through predefined report templates and natural language generation technology, and outputs a comprehensive report containing text descriptions, data tables, and key images; and a maintenance decision support unit, connected to the dynamic report generation unit, used to generate specific maintenance suggestions based on fault type, severity, and on-site conditions through a multi-criteria decision analysis model; this unit comprehensively considers fault urgency, maintenance resource availability, weather windows, and power generation loss costs, and outputs an optimized maintenance plan including maintenance steps, required spare parts, estimated working hours, and risk warnings.

[0074] An interactive visualization unit is used to display the high-reliability diagnostic and location results, wind turbine health status, and historical maintenance records, and provides a user interface. Specifically, it includes an intelligent question-and-answer interaction unit, which integrates a dialogue engine based on a pre-trained language model. This unit receives natural language queries from maintenance personnel and performs real-time reasoning based on the high-reliability diagnostic results and a domain knowledge base, providing targeted fault explanations, maintenance guidance, and decision-making support.

[0075] The intelligent question-and-answer unit is used to respond to queries from maintenance personnel and provide explanations and guidance related to inspection results. Specifically, it includes a multi-dimensional visualization unit, which integrates and displays real-time drone positioning, wind turbine health status heatmaps, 3D annotations of fault locations, and historical maintenance records within a unified geographic information platform; it supports users in performing data drill-down, status backtracking, and solution simulation through an interactive interface, achieving panoramic visualization monitoring and analysis of inspection data.

[0076] Example 2

[0077] like Figure 2 As shown in the diagram, this application provides a schematic diagram of a method for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power. This method is applied to the system described in Embodiment 1 for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power, and includes: S210, receiving multi-source heterogeneous data synchronously collected by the UAV, the data including visible light images, infrared thermal imaging, laser point clouds, audio signals and UAV flight status data; S220, Perform feature extraction and standardization on the multi-source heterogeneous data to obtain a standardized multimodal feature vector; S230, The standardized multimodal feature vector is processed using a fusion strategy that combines ensemble learning and deep learning to generate preliminary evaluation results and preliminary localization information; S240, through joint training and contrastive learning, projects multimodal features onto a shared semantic space for optimization, thereby enhancing the ability to discriminate abnormal states; S250, based on the optimized evaluation results, retrieve relevant historical fault cases and equipment parameters from the wind power knowledge base, verify and semantically enhance the preliminary evaluation results, and obtain high reliability diagnosis and location results; S260, based on the high-reliability diagnostic and positioning results, automatically generates inspection reports and maintenance suggestions, and provides users with visual information and intelligent Q&A services through an interactive interface.

[0078] Figure 3 This is an electronic device provided in one embodiment of this application. For example... Figure 3 As shown, the electronic device includes at least the following components: processor 301 and memory 300, communication interface 303, and bus 302.

[0079] In this embodiment of the application, memory 300 is used to store executable instructions of processor 301, which, when configured to execute instructions, implements the method as described in the first aspect.

[0080] In embodiments of this application, a computer-readable storage medium includes instructions that instruct a device to perform the method as described in the first aspect. For example, the instructions instruct the device to perform... Figure 1 The method is shown in the process steps.

[0081] In one embodiment of this application, the program operating in the electronic device may be a program that controls a central processing unit (CPU) or similar device to achieve the functions of the above-described embodiments of the present invention (a program that enables the computer to function). Information processed by these systems is then temporarily stored in random access memory (RAM) during processing, and subsequently stored in various ROMs such as read-only memory (FlashROM) and hard disk drives (HDDs), and read, corrected, and written by the CPU as needed.

[0082] It should be noted that a portion of the electronic device described in the above embodiments can also be implemented using a computer. In this case, the program for implementing the control function can be recorded on a computer-readable recording medium, and the program recorded on the recording medium can be read into the computer and executed.

[0083] It should be noted that the computer mentioned here refers to a computer built into an electronic device, employing hardware including an operating system and peripheral devices. Furthermore, computer-readable recording media refers to removable media such as floppy disks, magneto-optical disks, ROMs, and CD-ROMs, as well as storage systems such as hard drives built into the computer.

[0084] Furthermore, computer-readable recording media can include: media that dynamically stores programs for short periods of time, such as communication lines used when transmitting programs via networks like the Internet or communication lines like telephone lines; and media that store programs for fixed periods of time, such as volatile memory inside a computer that serves as a server or client in this case. In addition, the aforementioned program can be a program used to implement the above-mentioned functions, or it can be a program that can implement the above-mentioned functions by combining them with programs already recorded in the computer.

[0085] Furthermore, the electronic device in the above embodiments can also be implemented as an assembly (system group) composed of multiple systems. Each system constituting the system group can possess some or all of the functions or functional blocks of the electronic device in the above embodiments. As a system group, it is sufficient to have all the functions or functional blocks of the electronic device.

[0086] Those skilled in the art should recognize that the above embodiments are only used to illustrate this application and are not intended to limit this application. Any appropriate changes and variations made to the above embodiments within the essential spirit and scope of this application fall within the scope of protection claimed in this application.

Claims

1. A multimodal data fusion and intelligent analysis platform for unmanned aerial vehicle (UAV) inspection systems for wind farms, characterized in that, It includes the following six modules connected in sequence: The multimodal data acquisition module is used to simultaneously acquire multi-source heterogeneous data generated by the UAV during the wind farm inspection process. The data includes visible light images, infrared thermal imaging, laser point clouds, audio signals, and UAV flight status data. The feature extraction and standardization module is used to clean and align the multi-source heterogeneous data and extract standardized feature vectors, which include visual features, three-dimensional structural features, audio temporal features, and motion features. The multimodal information fusion module is used to process the standardized feature vectors using a fusion strategy that combines ensemble learning and deep learning to generate preliminary assessment results of the wind turbine equipment status and preliminary positioning information of the UAV. The joint learning and optimization module is used to project multimodal features into a shared semantic space through a joint training strategy, and introduces a contrastive learning mechanism to enhance the model's ability to discriminate abnormal wind turbine states. The domain knowledge injection module integrates a wind power domain knowledge base, which is used to call relevant historical fault cases and equipment parameters through a supervised retrieval mechanism to verify and semantically enhance the preliminary assessment results, and output highly reliable diagnostic and location results. The intelligent decision-making and application module is used to automatically generate inspection reports and maintenance suggestions based on the high-reliability diagnostic and positioning results, and provides a visual and intelligent question-and-answer interactive interface.

2. The multimodal data fusion and intelligent analysis platform for unmanned aerial vehicle (UAV) inspection systems for wind farms according to claim 1, characterized in that, The multimodal data acquisition module includes: The visible light and infrared imaging unit includes a high-resolution visible light camera and an infrared thermal imager, which are used to acquire texture detail images and temperature distribution data of the surface of the wind turbine components, respectively. The three-dimensional environment perception unit, including lidar, is used to acquire high-precision three-dimensional point cloud data of wind turbine blades, tower and surrounding environment; The acoustic signal acquisition unit includes a directional microphone array for acquiring audio signals during wind turbine operation and has an environmental wind noise suppression function. The flight status perception unit, integrated into the UAV flight control system, is used to collect the UAV's three-dimensional position, attitude, and motion status data in real time through the inertial measurement unit and the global navigation satellite system receiver. And a data synchronization control unit, used to generate unified timestamps and spatial positioning tags for the data streams collected by the visible light and infrared imaging unit, the three-dimensional environment perception unit, the acoustic signal acquisition unit and the flight status perception unit.

3. The multimodal data fusion and intelligent analysis platform for a UAV inspection system for wind farms according to claim 2, characterized in that, The feature extraction and standardization module includes: The data cleaning and alignment unit is used to remove outliers and fill in missing values ​​in the multimodal raw data, and to perform spatiotemporal alignment of sensor data with different sampling frequencies based on a unified timestamp; The visual feature extraction unit is used to extract texture features from visible light images and temperature distribution features from infrared thermal imaging data to form a visual feature vector. The three-dimensional structural feature extraction unit is used to extract the three-dimensional geometric features of wind turbine blades and tower from laser point cloud data. The audio timing feature extraction unit is used to extract timing pattern features that characterize the device's operating state from the audio signal; The motion feature extraction unit is used to extract kinematic features representing flight stability and environmental disturbances from UAV flight status data; It also includes a feature standardization unit, which is used to normalize and reduce the dimensionality of the extracted features and output a standardized feature vector with a unified dimension.

4. The multimodal data fusion and intelligent analysis platform for a UAV inspection system for wind farms according to claim 1, characterized in that, The multimodal information fusion module includes: The integrated learning fusion submodule is used to process the standardized feature vector in parallel through multiple heterogeneous base learners, and to integrate the initial prediction results of each base learner using a dynamic weighted fusion strategy to output preliminary UAV positioning information and equipment status assessment. The deep learning fusion submodule is used to perform cross-modal interaction and semantic fusion on the standardized feature vector through a deep neural network based on an attention mechanism, so as to generate a fusion feature representation rich in contextual information. And a hybrid decision submodule, used to generate an assessment result of the wind turbine equipment status and the drone's location information based on the preliminary assessment and positioning information output by the integrated learning fusion submodule and the fusion feature representation generated by the deep learning fusion submodule.

5. The multimodal data fusion and intelligent analysis platform for a UAV inspection system for wind farms according to claim 4, characterized in that, The joint learning and optimization module includes a shared semantic space mapping unit, used to project the fused feature vectors from the multimodal information fusion module onto a unified low-dimensional shared semantic space; The contrastive learning optimization unit is used to increase the feature distance between abnormal samples and normal samples in the semantic space by constructing triplets with normal state samples as anchors, normal state samples of the same type as positive samples, and abnormal state samples as negative samples, and optimizing the parameters of the shared semantic space mapping unit.

6. The multimodal data fusion and intelligent analysis platform for a UAV inspection system for wind farms according to claim 5, characterized in that, The contrastive learning optimization unit optimizes the parameters of the shared semantic space mapping unit by optimizing a joint loss function, which includes a first loss component for bringing positive sample pairs closer and pushing negative sample pairs further apart, and a second loss component for making features of similar samples more clustered in the shared semantic space.

7. A multimodal data fusion and intelligent analysis platform for a UAV inspection system for wind farms according to claim 5 or 6, characterized in that, The domain knowledge injection module includes a wind power domain knowledge base and a supervised retrieval mechanism unit; The supervised retrieval mechanism unit is used to retrieve relevant historical fault cases and equipment parameters from the knowledge base based on the post-optimization evaluation results output by the joint learning and optimization module. The domain knowledge injection module is configured to perform at least one of the following operations based on the retrieved relevant knowledge: If the current assessment results are consistent with the diagnostic conclusions of multiple historical cases retrieved, the credibility of the diagnostic results is enhanced. If the current assessment results conflict with the retrieved typical failure modes, a review result containing alternative diagnostic suggestions will be generated. The retrieved device parameters are correlated with currently detected anomalies to enhance the physical interpretability of the diagnostic results.

8. The multimodal data fusion and intelligent analysis platform for a UAV inspection system for wind farms according to claim 7, characterized in that, The supervised retrieval mechanism unit is configured to retrieve relevant cases through a comprehensive scoring function, which integrates at least semantic similarity, temporal relevance, and case confidence factors. Furthermore, the knowledge verification and semantic enhancement unit is configured to use an uncertainty reasoning method to fuse the current evaluation result with the retrieved knowledge.

9. A multimodal data fusion and intelligent analysis platform for a UAV inspection system for wind farms according to claim 1, characterized in that, The intelligent decision-making and application module includes: The report and decision generation unit is used to automatically generate inspection reports and output maintenance suggestions based on the high-reliability diagnosis and location results. An interactive visualization unit is used to display the high-reliability diagnostic and positioning results, wind turbine health status and historical maintenance records, and provides a user interface; The intelligent question-and-answer unit is used to respond to queries from operations and maintenance personnel and provide explanations and guidance related to the inspection results.

10. A method for aligning and optimizing the training accuracy of multimodal large models with autonomous and controllable computing power, applied to the multimodal data fusion and intelligent analysis platform of the UAV inspection system for wind farms as described in any one of claims 1 to 9, characterized in that, The method includes: Receive multi-source heterogeneous data synchronously collected by the UAV, including visible light images, infrared thermal imaging, laser point clouds, audio signals, and UAV flight status data; Feature extraction and standardization are performed on the multi-source heterogeneous data to obtain standardized multimodal feature vectors; The standardized multimodal feature vectors are processed using a fusion strategy that combines ensemble learning and deep learning to generate preliminary evaluation results and preliminary localization information. By combining training and contrastive learning, multimodal features are projected onto a shared semantic space for optimization, thereby enhancing the ability to detect abnormal states. Based on the optimized evaluation results, relevant historical fault cases and equipment parameters are retrieved from the wind power knowledge base. The preliminary evaluation results are then verified and semantically enhanced to obtain high reliability diagnosis and location results. Based on the highly reliable diagnostic and location results, inspection reports and maintenance suggestions are automatically generated, and users are provided with visual information and intelligent Q&A services through an interactive interface.

Images