Multi-source heterogeneous data access method and system for photovoltaic power generation digital twin platform

Abstract

The invention discloses a multi-source heterogeneous data access method and system for a photovoltaic power generation digital twin platform. The method comprises the steps of collecting original data from meteorological, environmental, electrical, image and other multi-source sensors, performing data cleaning, format standardization and exception elimination, then performing sampling, synchronization, feature extraction and structured preprocessing on the data, and respectively storing the data in a time sequence database, a relational database or an object storage according to data types. The system further pushes the processed data to a digital twin platform in real time through a standardized interface, so that visual presentation and model-driven simulation of the data are realized. According to the method, the problems of heterogeneous multi-source data, large precision difference, high real-time requirement and the like of a photovoltaic field station are solved, and the method has good system expansibility and engineering adaptability and is widely applied to distributed and centralized photovoltaic power generation systems.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing of photovoltaic power generation digital twin systems, and relates to a method and system for fusing multi-source heterogeneous, multi-acquisition frequency, and multi-precision data, and specifically to a method and system for accessing multi-source heterogeneous data to a photovoltaic power generation digital twin platform. Background Art

[0002] Currently, with the increasing proportion of photovoltaic power generation in the energy structure, digital twin technology is widely used in the operation and maintenance management, fault prediction, and system simulation of photovoltaic stations. Digital twin platforms typically require real-time access to multiple types of data from multiple sensors and systems, such as ambient temperature and humidity, irradiance, wind speed, dust concentration, voltage and current, infrared images, etc. However, in existing technologies, the non-uniform format of multi-source data, inconsistent collection frequency, and non-standard storage structure seriously restrict the twin platform's ability to efficiently utilize and analyze data. Existing data access solutions generally lack systematic cleaning, preprocessing, and dynamic sampling mechanisms for heterogeneous data streams, making it difficult to meet the unified requirements of digital twins for timeliness, accuracy, and structuring, limiting the improvement of the platform's intelligence and visualization.

[0003] In the existing technology, for example, Chinese patent CN105678901A discloses a data acquisition and monitoring system for photovoltaic power stations, which adopts a centralized data collection and display architecture, but does not involve the processing and structured access of image data, nor does it provide unified standards and cleaning strategies for multi-source heterogeneous data.

[0004] Patent CN109832145A proposes a data fusion method for photovoltaic systems, which can perform time-series alignment processing on some heterogeneous data. However, it still has shortcomings in real-time performance, data loss compensation, and structured processing of image data.

[0005] Existing methods generally have problems such as inconsistent structure, uneven data quality, inconsistent collection frequency, and scattered storage methods. They are difficult to support unified, standard, and efficient data-driven models and need to be improved. Summary of the Invention

[0006] To solve the above problems, the present invention discloses a multi-source heterogeneous data access method and system for a photovoltaic power generation digital twin platform, which has the capabilities of cleaning, sampling, structuring and efficient warehousing, providing high-quality, real-time underlying data support for the digital twin platform.

[0007] To achieve the above object, the technical solution of the present invention is as follows: The multi-source heterogeneous data access method of the photovoltaic power generation digital twin platform includes the following steps: Step 1: Data Acquisition: This module aims to achieve real-time, stable, and high-precision acquisition of multiple types of data, including meteorological, environmental, electrical, and image data. The data acquisition module ensures the effective integration and synchronization of various heterogeneous source data through a unified scheduling and access mechanism, forming the basis for subsequent system cleaning, preprocessing, and push. This module specifically includes the following technical content: Step 1.1 Data source and collection object configuration The types of data collected include meteorological data (such as total irradiance, reflected irradiance, ultraviolet irradiance, wind speed, wind direction, temperature, humidity, rainfall, and atmospheric pressure); environmental data (such as PM2.5 and dust concentration); electrical data (such as voltage, current, frequency, power factor, generated power, and power output); and image data (such as surface contamination, obstruction, cracks, hot spots, visible light images, and thermal infrared images of photovoltaic modules). Collection equipment includes weather stations, industrial sensors, electricity meters, cameras, drones, and inspection robots.

[0008] Step 1.2 Communication protocol and data format adaptation mechanism Taking into account the heterogeneity of multi-source devices, this invention supports multiple communication protocols such as Modbus RTU / TCP, OPC-UA, MQTT, HTTP / HTTPS, and FTP, and is compatible with multiple data formats such as CSV, JSON, XML, and image binary. Devices can be accessed through edge gateways or remotely collect data through secure channels such as VPNs and dedicated lines.

[0009] Step 1.3 Time synchronization and labeling mechanism To ensure data timing consistency, the acquisition system uniformly uses NTP for device time calibration and embeds UTC timestamps in all sampled data. The system supports time-window-based resampling and alignment algorithms to adapt to the platform's unified simulation rhythm.

[0010] Step 1.4 Edge collection and breakpoint resume mechanism The data collection terminal features local buffering and data packaging. If the uplink is interrupted, it automatically caches the data to be uploaded and resumes transmission by block number after communication is restored. The system supports decoupling the data sampling cycle from the upload cycle to improve bandwidth utilization and reliability.

[0011] Step 1.5 Image acquisition control strategy Image data collection takes into account both fixed time intervals and edge AI models that trigger collection based on the detection of events such as occlusion, cracks, or hot spots. Furthermore, the system supports the dispatching of drones or intelligent robots to specific locations to capture images on demand, enabling the automated collection of high-value image data.

[0012] Step 1.6: Acquisition of data quality control mechanisms Each data entry is accompanied by a quality indicator (Q-flag) to record sampling status, device status, and data validity. The system marks statuses such as device disconnection, abnormal values, and missing data, ensuring that the platform access end can accurately judge and process the raw data quality.

[0013] Step 2: Data cleaning: standardize the formats of data from different sources, remove outliers, and repair missing values; Step 2.1 Data format standardization mechanism Different data sources use different data formats. The data cleaning module first unifies the structure and maps the fields of the original data, and converts it into an intermediate format that can be uniformly recognized by the platform.

[0014] The system uses the following format (taking meteorological data as an example) as the intermediate format: { "device_id": "WS-1001", "data_type": "weather", "timestamp_utc": "2025-04-10T12:00:00Z", "location": { "lat": 34.2501, "lon": 108.9518, "alt": 410 }, "data": { "temperature": { "value": 25.1, "unit": "°C", "quality": "good" }, "humidity": { "value": 62.5, "unit": "%", "quality": "good" }, "irradiance": { "value": 850, "unit": "W / m²", "quality": "good" } }, "q_flag": "000" / / Quality control flag } The image formats are as follows: { "device_id": "CAM-01A", "data_type": "image", "timestamp_utc": "2025-04-10T12:01:00Z", "location": { "lat": 34.2501, "lon": 108.9518 }, "image_metadata": { "resolution": "1920x1080", "format": "JPEG", "angle": "Top-Down", "illumination": "Natural", "ai_flags": ["shadow", "crack"] }, "image_url": "https: / / www.fze.com / images / CAM-01A / 20250410_120100.jpg" } Perform field matching, renaming, and type conversion on structured data such as CSV, JSON, and XML, including: converting strings to values, converting timestamps to standard UTC format, and converting photovoltaic panel model codes; For image data, the JPEG format is uniformly used, EXIF ​​information is retained and metadata is encapsulated: acquisition device number, time, coordinates, etc. Step 2.2 Outlier detection and removal algorithm The system introduces a variety of rules and statistical methods to achieve anomaly detection, including: Threshold method: Preset a reasonable physical range for meteorological and electrical data (such as temperature between -50℃ and 85℃), and automatically mark values ​​outside the range as abnormal values; Trend shift detection: Calculate the mean and standard deviation based on a sliding window to detect data mutation points; Image quality detection: If the image resolution is too low, the blur is too high, or there is severe occlusion, the image will be marked as abnormal and will not be included in subsequent processing.

[0015] Step 2.3 Missing Value Identification and Repair Strategy Some data points may be missing during the data collection process. The system adopts the following repair strategies: Short-term missing data completion: Use linear interpolation, sliding average, etc. to fill in data for continuous time periods; Long-term missing data warning: If data is missing for more than a preset period of time (e.g., 5 minutes), the system will record an abnormal log and trigger an alarm mechanism; Image missing replacement: When image data is missing, the system can call images from adjacent time periods or generate prompt images as placeholders to ensure a stable image data flow; Step 2.4 Cleaning rules and process automation configuration Provides cleaning rule templates for different device types and data sources, and supports dynamic loading through configuration files. Environmental sensors and electrical instruments use different cleaning logic. The system automatically identifies the data type and loads the corresponding rule chain. Step 2.5 Data Quality Scoring and Logging Mechanism Each cleaned data item is assigned a quality score (Q-score) to assess its validity and usability. The system also records a complete cleaning log, including the original value, processing method, processing results, and processing time, for easy tracking and tracing. Step 3: Data preprocessing: including data sampling synchronization, feature extraction, standardization, image summarization and structured packaging; Step 3.1 Data sampling synchronization mechanism Because different data sources, such as meteorological, environmental, electrical, and image data, have different sampling periods, the platform uses the following synchronization technologies to ensure timing alignment and data consistency: Alignment based on the main time axis: For electrical data, 1s sampling is used as the main time axis, and the remaining data are aligned; Nearest neighbor interpolation method: For low-frequency sampling data (such as an image with one frame every 5 minutes), the nearest valid data copy is inserted according to the timestamp; Window averaging and resampling strategy: In environmental data, which is collected every 10 seconds, a sliding window can be used to statistically match 5-second alignment points to ensure uniform spatial and temporal resolution. Step 3.2 Feature extraction and module construction Extract key statistical features of meteorological and electrical data: mean, extreme value, volatility, slope, etc. Introducing energy characteristics: photovoltaic output per unit area, module efficiency; equipment health indicators: harmonic content, insulation value trend; Aggregation windows are used to construct sequence features, and a 5-minute sliding window is used to extract the maximum / minimum / average values ​​for predictive modeling input; Step 3.3 Standardization and normalization Perform a unified standardization operation on all numerical features, using Z-score standardization to eliminate dimensional differences; Perform one-hot encoding or label encoding on categorical / discrete fields in the system, such as device status and alarm type, to ensure they can be used as input for machine learning models. Step 3.4 Image summary and feature structured encapsulation For image data, image summarization technology is used to compress its main information and perform structured packaging. The methods include: Edge detection and morphological feature extraction: Use Sobel and Canny operators to identify edge changes and calculate crack length, occlusion area ratio, etc. Color and lighting analysis: Extracts the average brightness, color temperature offset, and saturation distribution of the image to determine whether there is haze, pollution, or other issues. AI model-assisted extraction: Use the object detection model YOLO to identify bird damage, dust accumulation, cracks, obstructions, etc., and encapsulate the results into structured label information; Image summary encapsulation format: After extracting the structured image summary, the image ID, time, device location, key tags (such as "hot spot", "crack", and "good lighting") are stored in JSON format and bound to the original image URL. Step 3.5 Packaging and data distribution preparation All pre-processed data items are uniformly encapsulated, including field identifiers, timestamps, source devices, spatial location information, processing status, etc., to generate a standardized intermediate data format (enhanced JSON format) for subsequent database storage and twin platform push. Step 4: Data storage: Based on the data characteristics, store it in a time series database, relational database, or object storage, and establish a unified index. Step 4.1 Classify storage mechanisms by data characteristics This method divides multi-source data into three categories based on differences in data structure, access frequency, and capacity scale, and stores them in the most suitable data systems, specifically: Time series data: This includes indicators that continuously change over time, such as photovoltaic module voltage and current, inverter output power, irradiance, ambient temperature and humidity, and wind speed. This data is stored using the high-performance time series database TDengine, which supports nanosecond timestamps, automatic compression, and aggregate queries. Structured data: This includes static device information (serial number, location, specifications), status alarm information, image summary structured tags, etc., stored in the relational database MySQL, facilitating multi-table associations, transaction support, and standard SQL queries; Unstructured large object data, including surveillance images, infrared images, and inspection videos, is stored in the object storage system MinIO and referenced and managed in the form of paths and metadata. Step 4.2 Unified index construction mechanism To achieve fast retrieval and data retrieval across systems and data types, the platform builds a unified data index, which mainly includes: Primary key structure: All data units have a unified primary key structure, using {source_id}-{timestamp}, to ensure that data in different systems can be linked; Metadata index table: Establish a "global data index table" with timestamp, spatial location, data type, data level and other fields as the main index to achieve rapid screening of any data object; Index synchronization mechanism: Every time data is stored, an update notification is sent through the queue service Kafka, and the index management module performs dynamic maintenance to keep the index real-time; Step 4.3 Multi-database collaboration mechanism The platform has an internal cross-database access module designed to support access to data from multiple databases through a single interface, enabling seamless joint calls of heterogeneous data. Specifically, it includes: Metadata distributor: parses the data type according to the query request and automatically routes it to the corresponding data system; Cache optimization mechanism: For frequently accessed data items, including the latest light power curve and fault alarms in the past 24 hours, the Redis cache system is used to accelerate responses. Permission and security control: External access to different data systems is achieved through a unified authentication and authorization interface, supporting access control lists (ACLs) and role-based permission configuration (RBAC). Step 4.4 Data persistence and backup disaster recovery mechanism To ensure the security and reliability of key operating data, the platform adopts the following disaster recovery technologies: Regular snapshot and incremental backup mechanisms ensure that the specified historical state can be restored in the event of hardware failure; Multi-node mirroring and master-slave redundant architecture improves system fault tolerance; Remote disaster recovery deployment to prevent regional failures from affecting data; Step 5: Data push: Push the processed data to the core engine of the digital twin platform through a data interface that supports multiple protocols.

[0016] Step 5.1 Multi-protocol data interface adapter module To meet the needs of receiving multi-source data on the digital twin platform, the system supports multiple communication protocols, including but not limited to: MQTT protocol: Suitable for lightweight, real-time sensor data push, with advantages such as low bandwidth usage and support for QoS mechanisms; WebSocket protocol: used to achieve real-time two-way communication with the twin front-end platform, for graphics rendering synchronization, alarm status push, etc. HTTP / HTTPS protocol: Suitable for batch data transmission and interface calls, mainly used for periodic reporting of data such as logs and image summaries; gRPC / RESTful API: used for high-performance communication between microservices, supporting data request response and data streaming scenarios; OPC-UA protocol: used to interconnect with PLCs in industrial control systems to meet edge computing or field device data backhaul requirements; Step 5.2 Data push scheduling strategy To improve the efficiency and fault tolerance of data push, the system introduces a data push scheduling mechanism, including: Data classification strategy: Based on data type and application requirements, data is divided into high-frequency real-time data (such as power and voltage), low-frequency status data (such as device status), image data, etc., and different push cycles and priorities are configured for each data type. Task queue buffering mechanism: All data to be pushed is buffered through the Kafka message queue to prevent abnormalities such as interface blocking and system jitter from affecting the overall operation; Batch and real-time fusion mechanism: The system supports real-time data push or batch aggregation and push based on time windows to improve communication efficiency; Step 5.3 Data format unification and structure encapsulation Before pushing, the data must be uniformly packaged into an intermediate exchange format recognized by the platform to ensure compatibility of different data types. The intermediate format adopts the following structure: { "source_id": "sensor_001", "data_type": "electric", "timestamp": "2025-04-10T12:00:00Z", "payload": { "voltage": 380.5, "current": 12.3, "temperature": 27.1 } } All pushed data are timestamped and identified with the source; Image data encapsulates metadata and image summaries through links to avoid transmitting large files; Supports flexible and extensible serialization formats such as JSON, Protobuf, and Avro; Step 5.4 Exception handling and retry mechanism To ensure the robustness of data transmission, the platform has built a robust error detection and compensation mechanism: Interface response monitoring: automatically monitor the push interface response status code and abnormal content, and record logs in real time; Push failure retry mechanism: When an exception such as interface response failure or network interruption occurs, the system will retry the push according to the exponential backoff strategy; Offline caching mechanism: During a disconnection or system downtime, data is temporarily cached locally or in a cache database and automatically re-uploaded after the connection is restored; Step 5.5 Security and permission control mechanism A multi-layered security strategy is used during data push to prevent data leakage and illegal access: TLS encrypted transmission ensures the security of communication links; Token / APIKey verification mechanism to ensure the authenticity of the pushed data source; Transmission log audit and blacklist and whitelist mechanisms facilitate data tracking and security protection; The present invention also provides a corresponding system architecture, which includes a data acquisition module, a cleaning module, a preprocessing module, a storage module and a push module, supports dynamic sampling strategies, version control and image intelligent analysis, and has highly flexible configuration capabilities and platform compatibility.

[0017] The beneficial effects of the present invention are: The multi-source heterogeneous data access method and system for the photovoltaic power generation digital twin platform described in the present invention cleans the original data, standardizes the format and eliminates anomalies, and then performs sampling, synchronization, feature extraction and structured preprocessing, and stores them in a time series database, relational database or object storage according to the data type; the digital twin platform provides high-quality, real-time underlying data support, and the system also pushes the processed data to the digital twin platform in real time through a standardized interface, realizing data visualization and model-driven simulation. It has good system scalability and engineering adaptability, and is widely applicable to distributed and centralized photovoltaic power generation systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1This is a system architecture diagram of the multi-source data access method of the present invention; Figure 2 This is a data cleaning and preprocessing flowchart: it shows the entire process from raw data input to structured output, including outlier removal, missing value filling, synchronization and feature extraction; Figure 3 This is a diagram of the data storage and push structure: it shows how data is distributed and stored in different database systems according to data types, and pushed to the digital twin platform through interfaces; Figure 4 This is a diagram of feature extraction from image data: it shows the entire process of extracting feature vectors through the image recognition model and then storing them in the database; Figure 5 The actual power on sunny days and the forecast results of the upcoming photovoltaic power generation; Figure 6 The actual power and upcoming photovoltaic power generation prediction results on cloudy days; Figure 7 The actual power on rainy days and the forecast results of the upcoming photovoltaic power generation. DETAILED DESCRIPTION

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0020] Deploy a variety of data collection devices: environmental sensors (temperature, wind speed, PM2.5), weather stations, power data collectors (RTU / PLC), and cameras (visible light, infrared). The data collection end uses the MQTT protocol to send data to the data processing module. All data is timestamped in UTC+0 format, with time accuracy accurate to the second.

[0021] During the data cleaning phase, the following strategies are used: Missing meteorological and environmental data are filled using a 5-minute sliding average; Electrical outliers were identified and removed using three times the standard deviation; The image is processed using OpenCV and the YOLO model to remove background noise and enhance clarity.

[0022] Example 2: Preprocessing and storage structure The preprocessing is done in the following ways: All data are sampled at a unified period (1 minute by default), which is dynamically adjusted by the scheduling module based on load and network status; The feature extraction module calculates features such as average value and rate of change of electrical data; The image processing module uses the ResNet network to extract key feature vectors such as component hot spots and cracks; Final structure: {timestamp, device ID, data type, feature value, source label, quality score}.

[0023] Data storage part: Time series data (temperature, irradiation, wind speed, etc.) are stored in TDengine; Electrical data is stored in MySQL; Image data is stored in the NoSQL database MinIO object storage database, linked to the URL, and associated with the image feature vector.

[0024] The push interface supports both RESTful API and WebSocket. The platform dynamically obtains real-time data streams based on subscribed topics for: 3D modeling display; Real-time monitoring of equipment status; Predictive maintenance and energy efficiency analysis; Generate twin simulation scene data.

[0025] The system can label the processing flow with a version (such as V1.0_V1001) to facilitate data rollback and offline reconstruction.

[0026] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A multi-source heterogeneous data access method for a photovoltaic power generation digital twin platform, characterized in that: The steps include: Step 1, data acquisition: acquiring data from multiple heterogeneous data sources, including meteorological sensors, environmental monitoring equipment, power collection devices, and image acquisition terminals; Step 2, data cleaning: normalize the format of the collected data, handle missing values, and detect anomalies. Image data is further subjected to noise removal and image enhancement. Step 3: Data preprocessing: This includes sampling synchronization, feature extraction, standardization, and encoding conversion, and constructing a unified data structure for storage. Step 4, data storage: store the preprocessed structured data in a relational database, time series database, or image object storage, and generate data indexes and metadata; Step 5, data push: Push the data to the core module of the digital twin platform in real time through a standardized communication interface to achieve data visualization, modeling and analysis.

2. The multi-source heterogeneous data access method for the photovoltaic power generation digital twin platform according to claim 1 is characterized in that: The data cleaning in step 2 includes: For electrical data, use sliding average and median filtering to remove outliers; Linear interpolation is used to fill in missing time period data for meteorological and environmental data.

3. The multi-source heterogeneous data access method for the photovoltaic power generation digital twin platform according to claim 1 is characterized in that: The cleaning of the image data in step 2 also includes enhancing the image quality using an edge detection algorithm and color histogram equalization.

4. The multi-source data access method supporting the photovoltaic power generation digital twin platform according to claim 1 is characterized in that: The data preprocessing described in step 3 uses a unified time base for sampling synchronization processing, and the sampling granularity is adjustable from 1 second to 5 minutes and is dynamically set by the scheduling module.

5. The multi-source heterogeneous data access method for the photovoltaic power generation digital twin platform according to claim 1 is characterized in that: The image data preprocessing described in step 3 also includes using a deep convolutional neural network to extract features from the component image. The extracted information includes: component occlusion, hot spot location, and crack identification.

6. The multi-source heterogeneous data access method for the photovoltaic power generation digital twin platform according to claim 1 is characterized in that: In the data structuring process described in step 3, the following fields are established: data source identification, acquisition time, data type, data content, confidence label and spatial location coordinates.

7. The multi-source heterogeneous data access method for the photovoltaic power generation digital twin platform according to claim 1 is characterized in that: The database described in step 4 adopts a hierarchical design. Meteorological and environmental data are stored in a time series database, electrical data is stored in a relational database, and image data is stored in a NoSQL database in the form of URL links and associated with metadata.

8. The multi-source heterogeneous data access method for the photovoltaic power generation digital twin platform according to claim 1 is characterized in that: The data push module described in step 5 supports multiple data interfaces, including RESTful API, MQTT protocol, and WebSocket protocol, to adapt to different receiving modes of the digital twin platform.

9. The multi-source heterogeneous data access method for photovoltaic power generation digital twin platform according to claim 1 is characterized in that: The data processing process supports version control and backtracking mechanisms, generating a unique version number after each data processing to facilitate historical data reconstruction and comparative analysis.

10. The multi-source heterogeneous data access system of the photovoltaic power generation digital twin platform is characterized by: It includes data acquisition module, cleaning module, preprocessing module, storage module and push module, supports dynamic sampling strategy, version control and image intelligent analysis, and has highly flexible configuration capabilities and platform compatibility.

Images