Multi-modal fusion wind power plant fire hazard multi-source data space-time synchronization evaluation method and system

Abstract

The invention belongs to the technical field of wind power plant fire assessment, and discloses a multi-modal fusion wind power plant fire hazard multi-source data space-time synchronous assessment method and system, and the method is characterized in that a reference calibration module builds a digital twin simulation, double closed-loop dynamic calibration and credibility quantification three-in-one mechanism; the time synchronization adopts a triple strategy of GPS time service, local clock compensation and transmission delay prediction, and the transmission delay is compensated in advance in combination with an LSTM network; the space calibration depends on a three-dimensional digital twin model of the fan, and mounting deviation and vibration drift are corrected through visual identification and coordinate matching; the quality grading module is used for constructing a three-dimensional quality model, distributing weights according to data quality grading, and reducing evaluation deviation caused by data heterogeneity; the feature fusion module adopts a spatial-temporal feature, modal feature and quality weight cross fusion mechanism; and capturing data time sequence association through an overlapped time window, and mining spatial association in combination with a digital twin spatial topological relation.

Description

Technical Field

[0001] This invention belongs to the field of wind farm fire assessment technology, specifically a multimodal fusion method and system for spatiotemporal synchronous assessment of multi-source data on wind farm fire hazards. Background Technology

[0002] With the continuous expansion of wind farm scale and the increasing operational lifespan of wind turbines, fire hazards are exhibiting complex characteristics of multi-source initiation, cross-regional spread, and latent evolution. Multimodal data fusion assessment has become a core technological direction for fire prevention and control. Current related technologies face the following technical challenges: The sources of multimodal data in wind farms are scattered, including robot inspection images, temperature and humidity data collected by fixed sensors, and distributed partial discharge signals. There are differences in the deployment location and type of equipment, and timestamps are prone to significant time deviations due to equipment clock drift. Spatial coordinates are affected by wind turbine vibration and installation deviations, making it difficult to achieve precise alignment. Existing systems generally use static calibration methods, which are difficult to adapt to the dynamic operating environment of wind farms, resulting in spatiotemporal misalignment of multimodal data and affecting the accuracy of hazard correlation analysis.

[0003] Existing technologies often have limitations. They either focus solely on spatiotemporal synchronization, only ensuring data time sequence consistency, or they simply perform modal fusion, ignoring the spatiotemporal correlation of data and failing to establish a linkage mechanism between spatiotemporal synchronization accuracy, modal feature reliability, and hazard assessment weight. For example, when the same high-temperature hazard occurs in different locations in the cabin, its risk level should be differentiated and assessed based on differences in spatiotemporal attributes. However, existing systems only use modal features for uniform scoring, resulting in a mismatch between the assessment results and the actual risks.

[0004] In addition, existing technologies have three core shortcomings: First, they lack dynamic spatiotemporal calibration schemes based on digital twin technology, making it difficult to cope with spatiotemporal drift caused by wind turbine vibration and environmental disturbances; second, the multimodal fusion methods are relatively simple, mostly adopting a basic weighted superposition mode, without forming a deep fusion architecture that crosses spatiotemporal and modal relationships; and third, the evaluation model has insufficient adaptive capabilities and lacks an optimization mechanism based on operation and maintenance feedback, making it difficult to adapt to dynamic scenarios such as equipment aging and changes in operating conditions during the long-term operation of wind farms. Summary of the Invention

[0005] The purpose of this invention is to provide a multimodal fusion method and system for spatiotemporal synchronous assessment of multi-source data on fire hazards in wind farms, in order to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a multimodal fusion multi-source data spatiotemporal synchronous assessment system for fire hazards in wind farms, comprising a data preprocessing module, a benchmark calibration module, a quality grading module, a feature fusion module, a hazard assessment module, a decision and early warning module, and an adaptive optimization module; Preferably, the data preprocessing module establishes a two-layer architecture combining full-source access and edge preprocessing. The core data accessed includes four categories: pyrolysis nanoparticle monitoring data, robot motion perception data, fixed sensor monitoring data, and static background data. It is compatible with industrial Ethernet, LoRa, 4G / 5G and other communication protocols, and supports the parsing of multiple industrial standard protocols such as Modbus-RTU, MQTT and OPCUA. By deploying edge computing nodes locally in the wind turbine nacelle, it performs real-time preprocessing on the collected raw data, including image dehazing and noise reduction, audio signal filtering and preliminary removal of sensor outliers. The protocol adaptation gateway enables the unified format conversion of heterogeneous data, and the output is a combination structure of data body, device identifier, original spatiotemporal label and edge preprocessing quality mark, providing standardized initial data for the benchmark calibration module.

[0007] Preferably, the reference calibration module establishes a three-in-one mechanism of digital twin simulation, dual-loop dynamic calibration, and reliable quantification; the dual-loop dynamic calibration includes a real-time calibration inner loop and a periodic optimization outer loop, with the inner loop adjusting calibration parameters in real time and the outer loop periodically optimizing the calibration strategy; the time dynamic synchronization adopts a triple mechanism of GPS time synchronization combined with local clock dynamic compensation and transmission delay prediction, calibrating the sensor clock every second through the NTP server, correcting the data transmission delay by combining the sliding window algorithm, and building a transmission delay prediction model based on the LSTM network to predict the transmission delay value within the next 1 second based on historical transmission data and real-time network status, and compensating in advance; The specific structure of the LSTM transmission delay prediction model is as follows: The input layer has a dimension of 12 and includes 6 types of time-series features, such as the historical transmission delay value of the past 60 seconds, network bandwidth, signal strength, wind speed, and vibration intensity. Each type of feature takes data from 2 time steps. There are 3 hidden layers with 64, 32, and 16 neurons in each layer, respectively, and the activation function is ReLU. The output layer has 1 neuron and is used to predict the transmission delay value in the next 1 second. The activation function is Linear.

[0008] Regarding training parameters, the number of iterations was set to 500 rounds, the batch size was selected as 32, the loss function was the mean squared error, the optimizer was Adam, the initial learning rate was set to 0.001, and the learning rate was decayed to 0.8 of the previous round every 100 rounds.

[0009] The training dataset uses 12 months of historical transmission data from wind farms, with a sample size of 100,000. It is divided into training set, validation set, and test set in a 7:2:1 ratio. The model training stops when the validation set MSE ≤ 0.001.

[0010] The spatial adaptive calibration is based on the three-dimensional digital twin model of the wind turbine. A dynamic spatial coordinate library is established for each sensing device. First, the fixed marker points in the nacelle are identified by machine vision. Combined with the real-time simulation results of the digital twin model, the spatial drift caused by sensor installation deviation and vibration is corrected in real time. Then, a dual strategy of triangulation method combined with digital twin coordinate matching is adopted to achieve the unification of spatial coordinates between the mobile robot and the fixed sensor. Simultaneously, a three-dimensional reliability assessment model for calibration accuracy, environmental interference, and equipment status is established, and the spatiotemporal reliability scores of each data are output in real time. The environmental interference factor includes parameters such as wind speed, temperature, and vibration intensity, while the equipment status factor is determined based on historical fault data of sensors and real-time operating parameters.

[0011] Preferably, the quality grading module establishes a three-dimensional quality grading model of spatiotemporal reliability, modal integrity, and edge preprocessing quality based on the spatiotemporal reliability score output by the benchmark calibration module and the edge preprocessing quality mark in the data access preprocessing stage, and performs graded screening and dynamic weight allocation on the calibrated multi-source data. Spatiotemporal reliability is determined by the spatiotemporal reliability score output by the benchmark calibration module. Modal integrity is comprehensively evaluated by three indicators: data missing rate, noise intensity, and outlier ratio. Edge preprocessing quality is determined by the labels output by the edge computing nodes. Level A data is directly entered into the fusion feature extraction process and assigned the highest fusion weight. Level B data is used after secondary optimization through interpolation completion and noise filtering and assigned a medium fusion weight. Level C data is marked as invalid and triggers sensor status warning. At the same time, the warning information is fed back to the digital twin model. The weight allocation results are synchronized to the feature fusion module in real time.

[0012] Preferably, the feature fusion module receives the graded data and dynamic weight allocation results output by the quality grading module, and establishes a cross-fusion extraction mechanism for spatiotemporal features, modal features, and quality weights; based on the calibrated spatiotemporal labels, an overlapping time window segmentation strategy is used to capture the temporal correlation of the data, and the spatial correlation of the data is mined by combining the spatial topological relationship of the digital twin model; The sliding step of the overlapping time window is set to 2.5s, which, together with the 5s window size, achieves a 50% overlap rate, ensuring that the continuity and correlation of time series data are not interrupted. The window data processing adopts a smooth splicing strategy, which performs mean-weighted fusion on the data in the overlapping areas of adjacent windows, with the weight linearly transitioning from 1.0 at the beginning of the window to 0.5 at the end. For edge data that is less than a window's worth of data, zero-padding is used to fill in the window length, ensuring consistent feature extraction dimensions.

[0013] Dedicated modal feature extraction algorithms are set for different data types. For image data, the lightweight YOLOv9 algorithm is used to extract the visual features of potential hazards. For partial discharge data, wavelet packet transform combined with singular value decomposition is used to extract frequency domain features. For temperature data, time-series difference combined with threshold detection is used to extract abrupt change features. For pyrolysis nanoparticle data, a dedicated algorithm is used to extract particle size distribution and concentration change features. The lightweight YOLOv9 model is optimized for wind farm fire hazard scenarios. Input images are uniformly resized to 640×640 pixels, and RGB channel normalization is used, with a mean of [0.485, 0.456, 0.406] and a variance of [0.229, 0.224, 0.225]. The anchor frame size is configured according to the actual dimensions of typical wind farm hazards such as flames, smoke, and high-temperature areas. [(10,13),(16,30),(33,23),(30,61),(62,45),(59,119),(116,90),(156,198),(373,326)]; Regarding model training parameters, the number of iterations is 300 rounds, the batch size is 16, the initial learning rate is set to 0.01, the cosine annealing learning rate scheduling strategy is adopted, and the weight decay is set to 0.0005. The training dataset consists of hazard images collected from wind farms in real time and generated by simulation. The sample size includes 20,000 images of flames, 15,000 images of smoke, and 15,000 images of high-temperature areas. The annotation format is VOC. The dataset is divided into training, validation, and test sets in an 8:1:1 ratio. The pre-trained weights are loaded and fine-tuned through transfer learning. TensorRT quantization is used to accelerate the model deployment.

[0014] By constructing a spatiotemporal and modal cross-attention network, the spatiotemporal correlation within a single modality is strengthened by the intramodal attention mechanism, and the feature interaction between different modalities is realized through the cross-modal attention mechanism. The weights output by the three-dimensional quality grading model are embedded into the fusion process. Finally, the spatiotemporal features and modal features are transformed to the same dimension and the fused feature vector is output through the cross-domain feature fusion network.

[0015] The cross-domain feature fusion network adopts a convolutional and fully connected architecture. The input is a 1024-dimensional vector concatenated from spatiotemporal features (512-dimensional) and modal features (512-dimensional). After feature mapping, the output is a 1024-dimensional fused feature vector that retains more than 99% of the core information. Two convolutional layers are set up. The first layer has 256 kernels, a kernel size of 1×1, a stride of 1, and uses ReLU as the activation function. The second layer has 128 kernels, a kernel size of 1×1, a stride of 1, and uses ReLU as the activation function. The fully connected layer consists of two layers. The first layer has 256 neurons, uses ReLU as the activation function, and has a dropout rate of 0.2. The second layer has 1024 neurons, uses Linear as the activation function.

[0016] Choosing 1024 dimensions as the output dimension, we used principal component analysis to perform dimensionality reduction verification on the fused features. 1024 dimensions can retain more than 99% of the information of the original features, while also meeting the input dimension requirements of the subsequent three-level evaluation model. The network training and feature fusion module are trained together as a whole. The training dataset is the same as that of the feature fusion module. Training stops when the feature reconstruction error on the validation set is ≤0.005.

[0017] Spatiotemporal and modal cross-attention network structure: ① A single-modal intra-attention mechanism is used, employing multi-head attention with 8 heads, input spatiotemporal feature dimension of 512, attention weight calculation using a scaled dot product model, dropout rate set to 0.1, and output dimension maintained at 512; ② Cross-modal attention mechanism: Four modal branches are set up: image, partial discharge, temperature, and nanoparticles. Each branch outputs features with a dimension of 256. The features of different modalities are projected to the same space with a dimension of 256 through a modal mapping matrix. There are 4 cross-modal attention heads. The attention weight is obtained by weighting the modal correlation matrix and the quality weight, with the quality weight accounting for 0.3. ③ The quality weight embedding method multiplies the weights output by the three-dimensional quality grading model with the attention output features element by element, thereby enhancing the feature contribution of high-quality data.

[0018] The network training parameters are as follows: 300 iterations, batch size of 32, contrastive loss plus cross-entropy loss, weight ratio of 0.4:0.6, AdamW optimizer, and learning rate of 0.0005.

[0019] Preferably, the hazard assessment module takes the fused feature vector output by the feature fusion module as the core input and adopts a three-level assessment architecture of station, unit, and core component. The core component level assessment establishes a dedicated assessment index system for key components such as bearings, cables, gearboxes, and oil-immersed equipment. It uses a fusion algorithm of support vector machine combined with gradient boosting tree to output the component hazard level and corrects the assessment results by combining the component operating status of the digital twin model. The SVM+GBT fusion algorithm adopts a serial fusion architecture. The first step is to perform preliminary classification through the GBT model, outputting the probability of component hazard level, and dividing it into probability distributions of no hazard, minor hazard, and serious hazard. The second step is to concatenate the probability distribution output by GBT with the original fusion features and input it into the SVM model for final classification.

[0020] Regarding specific parameters, the GBT model uses 300 decision trees, each with a depth of ≤8, a learning rate of 0.05, and a minimum number of sample splits of 20. The SVM model uses the RBF kernel function, with the kernel function parameter γ of 0.1, the penalty coefficient C of 10, and the probability estimation parameter probability set to True.

[0021] For the adaptation logic of different core components, the bearing components are additionally weighted by the grease state feature in the SVM with a weight value of 0.3, the cable components are weighted by the nanoparticle concentration feature with a weight value of 0.25, and the oil-immersed equipment is weighted by the temperature change rate feature with a weight value of 0.2.

[0022] The model training dataset consists of two years of component operation data from 50 wind turbine units in a wind farm, with a sample size of 80,000, including 40,000 with no hidden dangers, 30,000 with minor hidden dangers, and 10,000 with serious hidden dangers. The SMOTE algorithm is used to balance the sample distribution.

[0023] Unit-level assessment focuses on the collaborative analysis of multimodal data of a single unit to identify cross-component hazard correlations, and sets up a hazard diffusion path prediction submodule. Based on a digital twin model, it simulates the diffusion speed and range of hazards under different operating conditions to correct the unit risk level. The site-level assessment is based on the spatiotemporal and modal fusion characteristics of all units in the region, combined with meteorological data and historical hazard distribution, and uses the fuzzy comprehensive evaluation method to calculate the regional fire risk index. At the same time, it sets up a regional correlation analysis submodule to consider factors such as distance between units and wind direction to assess the hazard transmission risk of adjacent units. The core input parameters for diffusion path prediction include the initial intensity of the hazard, the fire resistance rating of the component material, the wind speed inside the cabin, the equipment spacing, and the ventilation volume. The fire resistance rating of the material is quantified as a fire resistance coefficient of 0.1 to 1.0. A coupled simulation model based on heat conduction and flue gas flow was adopted, with the simulation time step set to 1s. The diffusion range was divided into "dangerous area (within 3m of the core hidden danger point), warning area (3~8m), and attention area (8~15m)". The risk level correction coefficients for different areas were 1.5, 1.2, and 1.0, respectively.

[0024] The cross-level linkage mechanism enables positive transmission from component-level hidden dangers to unit-level review, unit-level high risk, and station-level early warning. Furthermore, the station-level early warning information guides the parameter adjustment of the unit-level and component-level assessment models, achieving closed-loop optimization of the assessment. The assessment results are directly pushed to the decision-making and early warning module.

[0025] Preferably, the decision-making and early warning module receives the full-level assessment results and related data output by the hazard assessment module, establishes an interactive interface integrating digital twin visualization, hierarchical early warning, and intelligent operation and maintenance recommendation, and realizes three-dimensional visualization of data and assessment results based on the wind farm digital twin model, including three-dimensional annotation of hazard location, spatiotemporal dynamic trajectory display, and hazard diffusion simulation animation; It also supports spatiotemporal retrospection of multi-source data and dynamic curve display of evaluation indicators. The early warning mechanism is linked with the evaluation level. Minor hidden dangers are pushed through the platform, while serious hidden dangers trigger multiple channels of early warning, including sound and light alarms, SMS push, APP push, and local alarms at edge nodes. The warning levels are divided into four levels based on the risk index: no risk (risk index < 0.3) corresponds to no warning; low risk (risk index 0.3~0.6) corresponds to a general warning; medium risk (risk index 0.6~0.8) corresponds to an important warning; and high risk (risk index ≥ 0.8) corresponds to an emergency warning. General warnings are only sent via text notifications through the platform. Important warnings trigger both platform pop-ups and app push notifications. Emergency warnings are triggered in the following order: "local audible and visual alarm at edge nodes → SMS message from maintenance personnel → app push notification → emergency pop-up on the platform". The interval between warnings from each channel does not exceed 30 seconds to ensure timely maintenance response.

[0026] An intelligent operation and maintenance recommendation submodule is set up to recommend the optimal operation and maintenance solution for different levels of hidden dangers by combining historical operation and maintenance data and digital twin simulation results. At the same time, information such as the verification status of hidden dangers and the handling effect reported by operation and maintenance personnel is synchronized to the adaptive optimization module.

[0027] Preferably, the adaptive optimization module uses the operation and maintenance handling results fed back by the decision warning module, sensor status data, and evaluation results output by each module as the core data source to establish a closed-loop adaptive mechanism of data feedback, reinforcement learning training, parameter update, and model iteration. By collecting operation and maintenance handling results, sensor status data, and evaluation model output results in real time, an iterative optimization dataset is established. A feedback-based reinforcement learning framework is introduced, and the verification results of operation and maintenance personnel are used as reward signals. The evaluation model and feature fusion network are trained by combining the multi-agent DDPG algorithm to optimize the spatiotemporal calibration parameters, feature fusion weights, and evaluation index thresholds, respectively. The multi-agent DDPG algorithm is configured with three agents, which are respectively responsible for optimizing spatiotemporal calibration parameters, optimizing feature fusion weights, and optimizing evaluation index thresholds. The agents cooperate by sharing an experience pool.

[0028] Regarding the core parameters, the state space dimension is set to 20, including state features such as operation and maintenance feedback accuracy, sensor failure rate, evaluation false alarm rate, and missed alarm rate, with two time steps of data taken for each feature; the action space dimension is 15, including five spatiotemporal calibration parameters, six feature fusion weights, and four evaluation index thresholds. The reward function formula is designed as follows: Reward = 1.0 × number of accurate assessments - 0.5 × number of false alarms - 1.0 × number of missed alarms + 0.3 × sensor normal operating time / 1000; The Actor network structure is: input layer → 64-dimensional fully connected layer (ReLU) → 32-dimensional fully connected layer (ReLU) → output layer (Tanh activation). The Critic network structure is: input layer → 64-dimensional fully connected layer (ReLU) → 32-dimensional fully connected layer (ReLU) → output layer (Linear activation). Regarding training parameters, the experience replay pool capacity was set to 100,000 records, the target network update frequency was once every 100 steps, the exploration rate ε was initially 0.9, decayed by 0.1 every 500 steps, and finally stabilized at 0.1; the number of iterations was 1000 rounds, the batch size was 64, the optimizer was Adam, and the learning rate was set to 0.0001.

[0029] A dual mechanism of monthly full data iteration and real-time updates is set up. The model undergoes a major version iteration every month based on the full data. After the updated model is verified by digital twin simulation, it is deployed to core modules such as spatiotemporal benchmark calibration, fusion feature extraction, and hazard assessment linkage. Real-time parameter fine-tuning is also performed based on operation and maintenance feedback and unexpected working conditions.

[0030] The real-time fine-tuning trigger conditions include four scenarios: first, three consecutive false alarms or two consecutive missed alarms; second, a sensor failure rate of ≥3%; third, extreme operating conditions (wind speed ≥15m / s, temperature ≤-25℃ or ≥65℃) lasting for more than 10 minutes; and fourth, poor handling results reported by maintenance personnel (success rate <70%). After triggering, the parameter fine-tuning range is controlled within ±10%. Among them, the spatiotemporal calibration parameter is adjusted first to adjust the transmission delay prediction coefficient, the feature fusion weight is adjusted first to adjust the weight ratio of Class A data, and the evaluation index threshold is adjusted first to adjust the threshold for judging serious hidden dangers.

[0031] This invention also provides a multimodal fusion method for spatiotemporal synchronous assessment of multi-source data on fire hazards in wind farms. Based on the above system, the specific steps are as follows: Data access and preprocessing stage: Four types of data are accessed: pyrolysis nanoparticle monitoring data, robot motion perception data, fixed sensor monitoring data and static background data. The data is adapted to multiple industrial communication protocols and preprocessed locally in the wind turbine nacelle through edge computing nodes, converting it into a standardized format. Spatiotemporal reference calibration stage: Time synchronization is achieved by using a triple mechanism of GPS time synchronization combined with local clock dynamic compensation and transmission delay prediction; based on the three-dimensional digital twin model of the wind turbine, spatial calibration is completed through visual recognition and coordinate matching, and a data spatiotemporal credibility score is generated synchronously. Quality grading and fusion stage: Establish a three-dimensional grading model of spatiotemporal credibility, modal integrity, and edge preprocessing quality, grade the data and assign weights; mine spatiotemporal correlation features through overlapping time windows, extract multimodal features in combination with dedicated algorithms, and fuse them into a unified dimension feature vector through a cross-attention network; Hazard assessment and early warning phase: A three-tier architecture of site, unit, and core components is adopted, and the assessment results are corrected by digital twin simulation. Hazard verification and early warning are realized through cross-level linkage. Based on the assessment results, an interactive interface integrating digital twin 3D visualization, hierarchical early warning, and intelligent operation and maintenance recommendation is established to push early warning information and operation and maintenance solutions. Adaptive optimization phase: Collect operation and maintenance feedback data, use feedback-based reinforcement learning algorithms and multi-agent DDPG algorithms to optimize parameters of each stage, and set up a dual mechanism of monthly full data iteration and real-time update.

[0032] The beneficial effects of this invention are as follows: 1. This invention establishes a three-in-one mechanism of digital twin simulation, dual closed-loop dynamic calibration, and reliable quantification through a benchmark calibration module; time synchronization adopts a triple strategy of GPS timing, local clock compensation, and transmission delay prediction, combined with LSTM network to compensate for transmission delay in advance; spatial calibration relies on the three-dimensional digital twin model of the wind turbine, and corrects installation deviations and vibration drift through visual recognition and coordinate matching; the quality grading module constructs a three-dimensional quality model, allocates weights according to data quality grade, A-grade data directly participates in fusion, and C-grade data triggers sensor warnings, effectively filtering low-quality data and reducing evaluation bias caused by data heterogeneity.

[0033] 2. The feature fusion module of this invention adopts a cross-fusion mechanism of spatiotemporal features, modal features, and quality weights; it captures the temporal correlation of data through overlapping time windows and mines spatial correlations by combining digital twin spatial topological relationships; it designs dedicated feature extraction algorithms for different types of data such as images, partial discharge, temperature, and nanoparticles, such as using the lightweight YOLOv9 algorithm for image data and wavelet packet transform combined with singular value decomposition for partial discharge data; it uses a cross-attention network to strengthen the spatiotemporal correlation of single modality and the interaction of cross-modal features, and outputs a unified dimension feature vector after incorporating quality weights, so that the fused features more comprehensively reflect the multidimensional attributes of fire hazards.

[0034] 3. This invention constructs a three-level assessment architecture for wind farms, turbines, and core components, coupled with a cross-level linkage mechanism. This enables component hazards to trigger turbine verification, and high-risk turbines to trigger site early warnings. Simultaneously, digital twin simulation is used to correct assessment results, fully considering the risks of hazard propagation and transmission, making the assessment more aligned with real-world scenarios. The adaptive optimization module uses operation and maintenance feedback data as its core, employing reinforcement learning and the DDPG algorithm. Through a dual mechanism of monthly full-data iteration and real-time parameter fine-tuning, it continuously optimizes calibration parameters, fusion weights, and assessment thresholds. This not only improves the accuracy of hazard identification and the timeliness of early warnings but also adapts to dynamic scenarios such as wind farm equipment aging and changing operating conditions, ensuring long-term fire prevention effectiveness. Attached Figure Description

[0035] Figure 1 This is a flowchart of the overall system of the present invention; Figure 2 This is a flowchart of the multi-source data processing and feature fusion process of the present invention; Figure 3 This is a flowchart of the three-level hazard assessment and early warning process of this invention; Figure 4 This is a flowchart of the adaptive optimization closed-loop process of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] like Figures 1 to 4 As shown in the figure, this embodiment of the invention provides a multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms, including a data preprocessing module, a benchmark calibration module, a quality grading module, a feature fusion module, a hazard assessment module, a decision-making and early warning module, and an adaptive optimization module. The specific implementation of each module is as follows: The data preprocessing module establishes a two-layer architecture that combines full-source access with edge preprocessing to achieve standardized access and preliminary quality improvement of wind farm multi-scenario data. The core data accessed includes four categories: pyrolysis nanoparticle monitoring data, robot mobile perception data, fixed sensor monitoring data, and static background data. The monitoring data of pyrolysis nanoparticles is based on nanoparticle sensors to capture characteristic nanoparticles generated by the pyrolysis of cables and oil immersion equipment, supporting the detection of particle sizes from 10nm to 1000nm. The robot's motion perception data consists of dual-light images and acoustic signals for the rail-guided inspection robot, supporting 2688×1520 resolution images and 30Hz to 16kHz audio input. Fixed sensor monitoring data includes wireless temperature measurement data, partial discharge monitoring data, and temperature and humidity sensor data, and is compatible with both passive wireless and active sensors; Static background data includes wind farm GIS maps, 3D digital twin models of wind turbine units, and equipment ledger information; It is compatible with industrial Ethernet, LoRa, 4G / 5G and other communication protocols, and supports the parsing of industrial standard protocols such as Modbus-RTU, MQTT, OPCUA. By deploying edge computing nodes locally in the wind turbine nacelle, it performs real-time preprocessing on the collected raw data, including image dehazing and noise reduction, audio signal filtering, and preliminary removal of sensor outliers, thereby reducing the transmission pressure on the cloud. Image dehazing employs a dark channel prior algorithm, while noise reduction uses a hybrid algorithm combining median filtering and Gaussian filtering. The median filtering window size is set to 3×3, and the Gaussian filtering standard deviation is set to 0.8. Audio signal filtering uses Butterworth low-pass filtering with a cutoff frequency of 8kHz to effectively filter high-frequency noise generated by the fan operation. Sensor outlier removal is initially performed using the 3σ criterion, which calculates the mean and standard deviation based on the historical data of the past 5 minutes. Data exceeding the mean ± 3 times the standard deviation are marked as outliers and removed.

[0038] The protocol adapter gateway has a built-in multi-protocol parsing engine that supports Modbus-RTU RTU mode, MQTT QoS level 2, and OPCUA UA-TCP protocol transmission. Through standardized data frame format conversion, it maps heterogeneous data into JSON format, with fields including five core items: data type, acquisition time, device number, value, and preprocessing tags.

[0039] The protocol adaptation gateway enables the unified format conversion of heterogeneous data, and the output is a combination structure of data body, device identifier, original spatiotemporal label and edge preprocessing quality mark, providing standardized and high-quality initial data for the benchmark calibration module.

[0040] To address the spatiotemporal heterogeneity of the multi-source heterogeneous data output by the data preprocessing module, the benchmark calibration module establishes a three-in-one mechanism of digital twin simulation, dual closed-loop dynamic calibration, and reliable quantification. The dynamic time synchronization is based on GPS timekeeping and combined with a triple time synchronization mechanism of local clock dynamic compensation (calibrating the sensor clock every second through the NTP server) and transmission delay prediction (sliding window algorithm correction + LSTM network prediction of the 1-second delay) to compensate in advance, so that the timestamp error of all data is ≤10ms.

[0041] The dual-loop dynamic calibration consists of an inner loop and an outer loop. The inner loop is a real-time calibration loop, which adjusts the sensor's spatiotemporal calibration parameters every 50ms based on the deviation between the real-time simulation results of the digital twin model and the actual sensor measurement data to ensure real-time alignment under dynamic operating conditions. The outer loop is a periodic optimization loop, which statistically analyzes the calibration error distribution of the inner loop every hour and optimizes the step size and threshold of calibration parameter adjustment based on the changing trend of environmental interference factors. When the average error of the inner loop exceeds 2ms / 2cm for three consecutive times, the outer loop is triggered for emergency optimization to quickly correct the calibration strategy.

[0042] The spatial adaptive calibration is based on a three-dimensional digital twin model of the wind turbine. The digital twin model integrates the OpenFAST multiphysics simulation engine to establish a dynamic spatial coordinate library for each sensing device. By recognizing fixed markers such as equipment nameplates and positioning targets in the nacelle through machine vision, and combining the real-time simulation results of the wind turbine vibration and deformation state output by the digital twin model, the spatial drift caused by sensor installation deviation and vibration is corrected in real time. A dual strategy of triangulation and digital twin coordinate matching is adopted to achieve the unification of spatial coordinates between the mobile robot and the fixed sensors, with an alignment error of ≤5cm.

[0043] Simultaneously, a three-dimensional reliability assessment model for calibration accuracy, environmental interference, and equipment status is established, and the spatiotemporal reliability score of each data is output in real time. The score range is from 0 to 1. The environmental interference factor includes parameters such as wind speed, temperature, and vibration intensity, and the equipment status factor is determined based on historical fault data of sensors and real-time operating parameters.

[0044] The three-dimensional reliability score is calculated using a weighted summation method, with calibration accuracy accounting for 40%, environmental interference accounting for 30%, and equipment status accounting for 30%. Calibration accuracy is based on spatiotemporal alignment error. A perfect score is achieved when the time error is ≤10ms and the spatial error is ≤5cm. For each 1ms / 1cm error exceeding the limit, 5% of the score is deducted. Environmental interference is judged comprehensively based on wind speed, temperature, and vibration intensity. Wind speed ≤8m / s is normal. For each 2m / s increase beyond the threshold, 10% of the score is deducted. Temperature is -20℃ to 60℃ is normal. For each 10℃ increase beyond the range, 10% of the score is deducted. Vibration intensity ≤0.1g is normal. For each 0.05g increase beyond the threshold, 10% of the score is deducted. The equipment status is based on the failure rate of the sensors in the past 3 months. A failure rate of ≤1% will get full marks. For every 1% increase, 20% of the score will be deducted. The spatiotemporal reliability score is the weighted average of the three scores, with a value range of 0 to 1.

[0045] The quality grading module establishes a three-dimensional quality grading model of spatiotemporal reliability, modal integrity, and edge preprocessing quality based on the spatiotemporal reliability score output by the benchmark calibration module and the edge preprocessing quality mark in the data access preprocessing stage. It then performs graded screening and dynamic weight allocation on the calibrated multi-source data to enhance the contribution of high-quality data. Spatiotemporal reliability is determined by the spatiotemporal reliability score output by the benchmark calibration module: ≥0.8 is grade A, 0.6 to 0.8 is grade B, and <0.6 is grade C. Modal integrity is comprehensively evaluated using three indicators: missing data rate, noise intensity, and outlier percentage. A grade of A is defined as missing data rate ≤1%, noise intensity ≤5%, and outlier percentage ≤2%; a grade of B is defined as missing data rate 1%-5%, noise intensity 5%-10%, and outlier percentage 2%-5%; and a grade of C is defined as missing data rate >5%, noise intensity >10%, and outlier percentage >5%. The quality of edge preprocessing is determined by the labels output by the edge computing nodes: no distortion is grade A, slight distortion is grade B, and severe distortion is grade C. Grade A data directly enters the fusion feature extraction process and is assigned the highest fusion weight, ranging from 0.7 to 1.0; Grade B data is used after secondary optimization through interpolation completion and noise filtering and is assigned a medium fusion weight, ranging from 0.3 to 0.7; Grade C data is marked as invalid and triggers sensor status warnings, while the warning information is fed back to the digital twin model. The weight allocation results are synchronized to the feature fusion module in real time, realizing deep linkage between quality grading and feature extraction.

[0046] The interpolation completion method uses linear interpolation. For missing time series data, the missing value is calculated by fitting the data based on three valid data points before and after the missing point. For spatially distributed missing data, the method uses weighted interpolation based on neighboring sensor data, with the weights inversely proportional to the sensor spacing. Noise filtering uses the Kalman filter algorithm, the process noise covariance is set to 0.01, and the observation noise covariance is dynamically adjusted based on the data noise intensity. The filtered data noise intensity must be ≤3% before it can enter the feature fusion process.

[0047] The dynamic weight allocation follows the principle of adaptive adjustment based on quality fluctuations. The initial weight range for Class A data is 0.7 to 1.0. When the modal integrity of this type of data remains at a missing rate of ≤0.5% and a noise intensity of ≤3% for 5 consecutive minutes, the weight is increased to 0.9 to 1.0. If the missing rate is ≥0.8% and the noise intensity is ≥4%, the weight is decreased to 0.7 to 0.8.

[0048] The initial weight range for Level B data is 0.3 to 0.7. After secondary optimization, if the proportion of outliers is ≤1%, the weight is increased to 0.5 to 0.7; if there are still ≥3% outliers after optimization, the weight is decreased to 0.3 to 0.5.

[0049] When the proportion of A-level data in the same modality is ≥80%, the weight of B-level data in that modality is reduced by 10% overall; when the proportion of A-level data in the same modality is <50%, the weight of B-level data in that modality is increased by 10% overall.

[0050] The feature fusion module receives the graded data and dynamic weight allocation results output by the quality grading module, and establishes a cross-fusion extraction mechanism for spatiotemporal features, modal features, and quality weights. Based on the calibrated spatiotemporal labels, an overlapping time window segmentation strategy is adopted, with a window size of 5s and an overlap rate of 50%, to capture the temporal correlation of the data, such as the temperature change trend over 30 seconds and the partial discharge signal pulse sequence. The spatial correlation of the data is also mined by combining the spatial topological relationship of the digital twin model, such as the data synergy of different sensors of the same component and the correlation of the spread of hidden dangers of adjacent components.

[0051] Dedicated modal feature extraction algorithms are set for different data types. For image data, the lightweight YOLOv9 algorithm is used to extract visual features of potential hazards such as flames, smoke, and high-temperature areas. For partial discharge data, wavelet packet transform combined with singular value decomposition is used to extract frequency domain features. For temperature data, time-series difference combined with threshold detection is used to extract abrupt change features. For pyrolysis nanoparticle data, a dedicated algorithm is used to extract particle size distribution and concentration change features. The wavelet packet transform uses the db4 wavelet basis function, the decomposition level is set to 4 levels, the corresponding frequency band is divided to 1kHz~16kHz, covering the main frequency range of wind farm partial discharge signal, and the high frequency coefficients of the 2nd to 4th levels after decomposition are reconstructed. Singular value decomposition constructs a 128×128-dimensional Hankel matrix for the reconstructed signal, extracts the top 10 largest singular values ​​as frequency domain feature vectors, and removes redundant data with amplitudes less than 10% of the largest singular values ​​to ensure that the feature vector dimensions are consistent and that core information is not lost.

[0052] The dedicated extraction algorithm adopts a combination of particle size spectrum analysis and concentration time series modeling. First, it calculates the proportion of particles in each interval through particle size classification statistics (three intervals: 10nm~100nm, 100nm~500nm, and 500nm~1000nm) and extracts two core features: peak particle size and particle size distribution standard deviation. Then, it calculates the concentration change rate and concentration mutation amplitude based on the sliding time window, and judges the effectiveness of the features by combining the concentration trend of three consecutive windows. Only feature data with a concentration change rate ≥5% / s or peak particle size falling in the 100nm~500nm interval are retained.

[0053] By constructing a spatiotemporal and modal cross-attention network, the spatiotemporal correlation within a single modality is strengthened by the intramodal attention mechanism, and the feature interaction between different modalities is realized through the cross-modal attention mechanism. The weights output by the three-dimensional quality grading model are embedded into the fusion process to give higher attention to the features of high-quality data. Finally, the spatiotemporal features and modal features are transformed to the same dimension (1024-dimensional feature vector) through the cross-domain feature fusion network and the fused feature vector is output.

[0054] The hazard assessment module uses the 1024-dimensional fused feature vector output by the feature fusion module as its core input. It adopts a three-level assessment architecture of station, unit, and core component, and combines cross-level linkage mechanism and scenario-based assessment strategy to achieve full-dimensional hazard assessment from macro to micro, improving the matching degree between assessment results and actual risks. The core component level assessment establishes a dedicated assessment index system for key components such as bearings, cables, gearboxes, and oil-immersed equipment. For example, the three-dimensional indexes of bearing temperature, partial discharge intensity, and grease state, and the three-dimensional indexes of cable temperature, partial discharge, and nanoparticle concentration are used. The fusion algorithm of support vector machine combined with gradient boosting tree is used to output the hazard level of the component, including three levels: no hazard, minor hazard, and serious hazard. The assessment results are corrected by combining the component operating status (wear degree, aging state) of the digital twin model. The threshold values ​​for core component evaluation indicators are set based on the equipment's rated parameters and industry standards. The bearing temperature threshold is 85℃, exceeding which is considered abnormal; the partial discharge intensity threshold is 50pC, exceeding which is considered abnormal; the grease condition is considered normal if the dielectric constant is ≥2.5; the cable temperature threshold is 90℃, the partial discharge intensity threshold is 30pC, and the nanoparticle concentration threshold is 1000 particles / cm³. Exceeding a single indicator for a duration ≥30s is considered a minor hazard, while exceeding two or more indicators is directly considered a serious hazard.

[0055] Unit-level assessment focuses on the collaborative analysis of multimodal data of a single unit to identify cross-component hazard correlations, such as the correlation between high gearbox temperature and lubricating oil leakage, and the correlation between cable overheating and abnormal partial discharge signals. It also sets up a hazard propagation path prediction submodule, which uses a digital twin model to simulate the propagation speed and range of hazards under different operating conditions to correct the unit risk level. The site-level assessment is based on the spatiotemporal and modal fusion characteristics of all units in the region, combined with meteorological data such as wind speed, precipitation, and temperature and historical hazard distribution. It uses the fuzzy comprehensive evaluation method to calculate the regional fire risk index, and sets up a regional correlation analysis submodule to consider factors such as distance between units and wind direction to assess the hazard transmission risk of adjacent units. The evaluation factor set of the fuzzy comprehensive evaluation method includes four core factors: the average hidden danger level of the unit, the degree of meteorological influence, the density of historical hidden dangers, and the distribution of the distance between units. The weights of each factor are 40%, 25%, 20%, and 15%, respectively. The evaluation level is divided into four levels: no risk, low risk, medium risk, and high risk, corresponding to risk index ranges of 0~0.3, 0.3~0.6, 0.6~0.8, and 0.8~1.0. The degree of meteorological impact is determined by a combination of wind speed, precipitation, and temperature. The historical hazard density is calculated as the ratio of the number of hazard occurrences in the area to the number of generating units in the past year. The distribution of the distance between generating units is divided into three categories: close correlation (≤500m), moderate correlation (500~1000m), and weak correlation (>1000m), each assigned a different impact coefficient.

[0056] The cross-level linkage mechanism enables component-level hidden dangers to trigger unit-level review to verify whether the hidden dangers have spread; high risks at the unit level trigger station-level early warnings, and station-level early warning information guides the adjustment of parameters in the unit-level and component-level assessment models. For example, the assessment threshold of components in high-risk areas is reduced, achieving closed-loop optimization of the assessment, and the assessment results are directly pushed to the decision-making and early warning module.

[0057] When the core component level assessment result is a serious hidden danger or three consecutive minor hidden dangers, the unit level review is automatically triggered; when the unit level risk level reaches high risk (risk index ≥ 0.8) or two consecutive adjacent units have high risk, the site level early warning is automatically triggered. After a station-level early warning is activated, the threshold values ​​for component-level assessment indicators of units within the area are lowered by 20%, and the time coefficient for predicting the spread path of unit-level hidden dangers is increased by 30%, until the default parameters are restored after the early warning is lifted. For unconfirmed component-level hidden dangers, the feature weights of the corresponding components are temporarily increased by 15% in the next assessment to improve identification sensitivity.

[0058] The decision-making and early warning module receives the full-level assessment results and related data output by the hidden danger assessment module, and establishes an interactive interface that integrates digital twin 3D visualization, hierarchical early warning, and intelligent operation and maintenance recommendation to improve the practicality of decision-making, including 3D annotation of hidden danger location, spatiotemporal dynamic trajectory display, and hidden danger diffusion simulation animation. It also supports spatiotemporal backtracking of multi-source data and dynamic curve display of evaluation indicators. The early warning mechanism is linked with the evaluation level. Minor hidden dangers are pushed through the platform, while serious hidden dangers trigger multi-channel early warnings through audible and visual alarms, SMS push, APP push and local alarms at edge nodes. An intelligent operation and maintenance recommendation submodule is set up to recommend the optimal operation and maintenance solution for different levels of hidden dangers by combining historical operation and maintenance data and digital twin simulation results, such as operation and maintenance time, required tools, and safety precautions; at the same time, information such as the verification status of hidden dangers and the handling effect reported by operation and maintenance personnel is synchronized to the adaptive optimization module.

[0059] Intelligent operation and maintenance recommends building a three-level solution library, corresponding to minor hidden dangers, serious hidden dangers, and cross-unit hidden dangers respectively; The minor hazard solution library matches component type with hazard characteristics. For example, abnormal bearing temperature corresponds to the "shutdown and cooling + grease detection" solution, and abnormal cable partial discharge corresponds to the "local insulation detection + tightening" solution. The serious hazard solution library is associated with emergency response procedures, including personnel allocation, tool list, and response time limit. The cross-unit hazard solution library combines wind direction and distance to formulate the zoning response order, prioritizing the hazard source unit and adjacent downwind units. The priority for matching solutions is: historical success rate > solution cost > downtime. Solutions with a success rate of ≥90% are given priority. At the same time, the solution time limit is dynamically adjusted according to the speed of hazard spread in digital twin simulation.

[0060] The adaptive optimization module uses the operation and maintenance handling results fed back by the decision-making and early warning module, sensor status data, and evaluation results output by each module as core data sources. It establishes a closed-loop adaptive mechanism for data feedback, reinforcement learning training, parameter updates, and model iteration. By collecting operation and maintenance handling results (hazard verification status, handling effect, false alarm or missed alarm records), sensor status data, and evaluation model output results in real time, it establishes an iterative optimization dataset and introduces a feedback-based reinforcement learning framework. The verification results of operation and maintenance personnel are mapped into numerical reward signals: +1 reward for accurate evaluation, and -0.5 reward for false alarm and missed alarm respectively. The evaluation model and feature fusion network are trained by combining the multi-agent DDPG algorithm to optimize the spatiotemporal calibration parameters, feature fusion weights, and evaluation index thresholds, respectively, to achieve multi-objective collaborative optimization that reduces the false alarm rate, improves the missed alarm recognition rate, and enhances the evaluation accuracy. A dual mechanism of monthly full data iteration and real-time updates is set up. The model undergoes a major version iteration every month based on the full data. After the updated model is verified by digital twin simulation, it is deployed to core modules such as spatiotemporal benchmark calibration, fusion feature extraction, and hazard assessment linkage. Real-time parameter fine-tuning is performed based on operation and maintenance feedback and sudden conditions such as extreme weather and equipment failure to ensure the adaptability and accuracy of the system in long-term operation.

[0061] The simulation verification selected three typical scenarios, including normal working conditions, minor hidden danger working conditions, and serious hidden danger working conditions. The scenario input was reconstructed based on historical measured data, and the evaluation accuracy, early warning response time, and parameter stability of the model before and after optimization were compared. The validation criteria are an improvement of ≥3% in evaluation accuracy, a reduction of ≥100ms in warning response time, and a parameter fluctuation of ≤5%. Models that do not meet the criteria are returned to the reinforcement learning training stage, and the training rounds and reward function weights are adjusted and optimized again until the validation criteria are met.

[0062] This invention also provides a multimodal fusion method for spatiotemporal synchronous assessment of multi-source data on fire hazards in wind farms. Based on the above system, the specific steps are as follows: Data access and preprocessing stage: Four types of data are accessed: pyrolysis nanoparticle monitoring data, robot motion perception data, fixed sensor monitoring data and static background data. The data is adapted to multiple industrial communication protocols and preprocessed locally in the wind turbine nacelle through edge computing nodes, converting it into a standardized format. Spatiotemporal reference calibration stage: Time synchronization is achieved by using a triple mechanism of GPS time synchronization combined with local clock dynamic compensation and transmission delay prediction; based on the three-dimensional digital twin model of the wind turbine, spatial calibration is completed through visual recognition and coordinate matching, and a data spatiotemporal credibility score is generated synchronously to provide a basis for subsequent quality grading.

[0063] Quality grading and fusion stage: Establish a three-dimensional grading model of spatiotemporal credibility, modal integrity, and edge preprocessing quality, grade the data and assign weights; mine spatiotemporal correlation features through overlapping time windows, extract multimodal features in combination with dedicated algorithms, and fuse them into a unified dimension feature vector through a cross-attention network; Hazard assessment and early warning phase: A three-tier architecture of site, unit, and core components is adopted, and the assessment results are corrected by digital twin simulation. Hazard verification and early warning are realized through cross-level linkage. Based on the assessment results, an interactive interface integrating digital twin visualization, hierarchical early warning, and intelligent operation and maintenance recommendation is established to push early warning information and operation and maintenance solutions. Adaptive optimization phase: Collect operation and maintenance feedback data, use feedback-based reinforcement learning algorithms and multi-agent DDPG algorithms to optimize parameters of each stage, and set up a dual mechanism of monthly full data iteration and real-time update.

[0064] The core evaluation metrics of the artificial intelligence model involved in this invention are as follows: ①LSTM transmission delay prediction model, mean absolute error ≤0.5ms, root mean square error ≤0.8ms on the test set; ② Lightweight YOLOv9 model, with mAP ≥ 0.92 for vulnerability detection, recall ≥ 0.95, and inference speed ≥ 30 FPS; ③ The SVM+GBT fusion algorithm achieves a component hazard classification accuracy of ≥0.95, a serious hazard recall rate of ≥0.98, and a minor hazard precision rate of ≥0.90. ④ The multi-agent DDPG algorithm, after optimization, has a false alarm rate of ≤3%, a false negative rate of ≤1%, and a parameter iteration convergence speed of ≤10 rounds / month. ⑤ A spatiotemporal and modal cross-attention network + cross-domain feature fusion network improves the accuracy of hazard identification by ≥20% after feature fusion compared to a single modality, and the cross-modal feature interaction efficiency is ≥90%.

[0065] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal fusion system for spatiotemporal synchronous assessment of multi-source data on fire hazards in wind farms, characterized in that, It includes a data preprocessing module, a benchmark calibration module, a quality grading module, a feature fusion module, a hazard assessment module, a decision and early warning module, and an adaptive optimization module; Data preprocessing module: Establishes a full-source access and edge preprocessing architecture, accesses multiple types of wind farm data and completes format unification and preliminary quality improvement, and outputs standardized combined data including data ontology, device identifier, original spatiotemporal label, and edge preprocessing quality label; The benchmark calibration module achieves spatiotemporal unification of multi-source data based on a digital twin model, generating a spatiotemporal reliability score that reflects the reliability of the data. Quality grading module: Combines spatiotemporal credibility scoring with preprocessed quality labels to establish a three-dimensional quality grading model, and performs data grading and filtering and dynamically allocates weights; Feature fusion module: Receives hierarchical data and weight allocation results, mines the temporal and spatial correlations of the data, fuses features from multiple types of data and converts them into feature vectors of a unified dimension; Hazard assessment module: It establishes a three-level assessment architecture of site, unit and core components with the input of fusion features, and realizes full-dimensional hazard assessment by combining cross-level linkage mechanism, corrects the risk level of unit and forms assessment closed loop; Decision-making and early warning module: Receives evaluation results, establishes an interactive interface, realizes the visualization of evaluation data, provides tiered early warning and operation and maintenance solution recommendations, and synchronizes operation and maintenance feedback data to the adaptive optimization module; Adaptive optimization module: Based on operation and maintenance feedback data, train, evaluate, and optimize the model and feature fusion network, and iteratively update the parameters of each core module.

2. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 1, characterized in that, The data preprocessing module's full-source access architecture is compatible with various industrial communication protocols and industrial standard protocol parsing. The core data accessed includes four categories: pyrolysis nanoparticle monitoring data, robot motion perception data, fixed sensor monitoring data, and static background data. Edge preprocessing is completed by deploying edge computing nodes locally in the wind turbine nacelle, including image dehazing and noise reduction, audio signal filtering, and preliminary removal of sensor outliers; A unified format conversion of heterogeneous data is achieved through a protocol adaptation gateway, and the standardized combined data is output.

3. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 2, characterized in that, The reference calibration module establishes a three-in-one mechanism of digital twin simulation, dual closed-loop dynamic calibration, and reliable quantification; time synchronization adopts a triple mechanism of GPS time synchronization combined with local clock dynamic compensation and transmission delay prediction. Spatial adaptive calibration is based on the three-dimensional digital twin model of the wind turbine. A dynamic spatial coordinate library of sensing equipment is established. Fixed marker points inside the nacelle are identified by machine vision. The spatial drift caused by sensor installation deviation and vibration is corrected by combining the real-time simulation results of the digital twin model. A dual strategy of triangulation method combined with machine vision recognition and digital twin coordinate matching is adopted to achieve the unification of spatial coordinates between the mobile robot and the fixed sensor. The three-dimensional credibility assessment model combines calibration accuracy, environmental interference, and equipment status parameters to generate a spatiotemporal credibility score.

4. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 3, characterized in that, The three-dimensional quality grading model of the quality grading module specifically includes three dimensions: spatiotemporal reliability, modal integrity, and edge preprocessing quality. Spatiotemporal reliability is determined by the spatiotemporal reliability score output by the benchmark calibration module. Modal integrity is comprehensively evaluated by three indicators: data missing rate, noise intensity, and outlier ratio. Edge preprocessing quality is determined by the labels output by the edge computing nodes. After classification, Grade A data directly participates in the fusion feature extraction process and is assigned the highest fusion weight. Grade B data is used after secondary optimization through interpolation completion and noise filtering and is assigned a medium fusion weight. Grade C data is marked as invalid and triggers sensor status warning. The weight allocation results are synchronized to the feature fusion module.

5. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 4, characterized in that, The feature fusion module employs a cross-fusion extraction mechanism of spatiotemporal features, modal features, and quality weights; and mines the temporal and spatial correlations of the data based on the calibrated spatiotemporal labels. Dedicated algorithms are set up to extract modal features for different data types. A spatiotemporal and modal cross-attention network is used to enhance the spatiotemporal association of a single modality and the interaction of cross-modal features. After incorporating quality weights, a cross-domain feature fusion network is used to output a feature vector with a unified dimension.

6. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 5, characterized in that, In the three-level assessment architecture of the hazard assessment module, the core component level assessment establishes a dedicated assessment index system for key components, outputs the component hazard level, and corrects the assessment results by combining the component's operating status with the digital twin model. The unit-level assessment includes a sub-module for predicting the spread path of potential hazards, which uses a digital twin model to simulate the spread speed and scope of potential hazards and correct the unit's risk level. The site-level assessment uses the fuzzy comprehensive evaluation method to calculate the regional fire risk index and sets up a regional correlation analysis submodule to assess the risk of hazard transmission between adjacent units. The cross-level linkage mechanism enables component-level hidden dangers to trigger unit-level review, unit-level high risks to trigger station-level early warnings, and station-level early warning information to guide the parameter adjustment of unit-level and component-level assessment models.

7. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 6, characterized in that, The interactive interface of the decision-making and early warning module is based on the digital twin model of the wind farm to realize the three-dimensional visualization of data and evaluation results; It supports spatiotemporal backtracking of data and dynamic curve display of evaluation indicators. The early warning mechanism is linked with the evaluation level. Minor hidden dangers are pushed to the platform, while serious hidden dangers trigger multiple channels of early warning, including audible and visual alarms, SMS push, APP push, and local alarms at edge nodes. It includes an intelligent operation and maintenance recommendation submodule, which combines historical operation and maintenance data with digital twin simulation results to recommend the optimal operation and maintenance solution, while synchronizing operation and maintenance feedback information to the adaptive optimization module.

8. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 7, characterized in that, The adaptive optimization module establishes a closed-loop adaptive mechanism encompassing data feedback, reinforcement learning training, parameter updates, and model iteration. An iterative optimization dataset was established using operation and maintenance results, sensor status data, and evaluation model output results as core data sources. A reinforcement learning algorithm based on multi-agent DDPG was adopted to optimize spatiotemporal calibration parameters, feature fusion weights, and evaluation index thresholds. A dual update mechanism is set up, which includes monthly full data iteration and real-time parameter fine-tuning. After the updated model is verified by digital twin simulation, it is then deployed.

9. The multimodal fusion-based spatiotemporal synchronous assessment system for multi-source data on fire hazards in wind farms according to claim 8, characterized in that, The interactive interface supports spatiotemporal backtracking queries of multi-source data and real-time display of dynamic curves of evaluation indicators.

10. A multimodal fusion method for spatiotemporal synchronous assessment of multi-source data on fire hazards in wind farms, based on the system described in claim 9, characterized in that, The specific steps are as follows: Data access and preprocessing stage: four types of core data are accessed, industrial communication protocols are adapted, and preprocessing is completed through the local edge computing node in the cabin and converted into a standardized format; Spatiotemporal reference calibration phase: A triple time synchronization mechanism is adopted to achieve time unification, spatial calibration is completed based on a three-dimensional digital twin model, and a spatiotemporal credibility score is generated synchronously; Quality grading and fusion stage: Data grading and weight allocation are completed through a three-dimensional quality grading model, spatiotemporal correlation features are mined and multimodal features are extracted, and then fused into a unified dimension feature vector through a spatiotemporal and modal cross-attention network; Hazard assessment and early warning stage: Based on the three-level assessment architecture and combined with digital twin simulation to correct the assessment results, cross-level linkage is used to realize hazard review and early warning, and a visual interactive interface is built to push early warning information and operation and maintenance solutions; Adaptive optimization phase: Collect operation and maintenance feedback data, use reinforcement learning and multi-agent algorithms to optimize parameters of each stage, and implement a dual mechanism of monthly iteration and real-time update.

Images