An open-source method, system, and storage medium for dynamic acquisition of solar irradiance data.

Abstract

This invention discloses a method for dynamically acquiring solar irradiance data, comprising the following steps: S1, screening open-source data and analyzing acquisition conditions; S2, data acquisition: S2-1, setting acquisition rules for general open-source data source websites and acquiring data; S2-2, setting acquisition rules for special open-source data source websites and acquiring data; S2-3, cleaning and storing the acquired data. This method provides data support and application support for future research in fields such as solar energy resource assessment and prediction, and large-scale development and application of photovoltaic power generation.

Description

Technical Field

[0001] This invention belongs to the field of solar irradiance data calculation and processing technology, and specifically relates to a method, system and storage medium for dynamic acquisition of solar irradiance data in this field. Background Technology

[0002] According to data from the National Energy Administration, in 2021, my country's photovoltaic (PV) power generation accounted for only 3.9% of total electricity consumption and 13.1% of renewable energy power generation, indicating that PV power generation needs further development. The assessment and prediction of solar energy resources rely heavily on historical solar irradiance data. Historical solar irradiance values ​​can accurately assess a region's solar power generation potential and predict future solar irradiance, which is crucial for the large-scale development and application of PV power generation. However, historical solar irradiance data is complex to obtain, difficult to acquire, and of varying quality, requiring complex preliminary processes before it can be used for solar energy resource assessment or prediction.

[0003] Historical solar irradiance data is divided into two types: one obtained through ground-based observations and the other derived from satellite data through reanalysis. The former consists of point observations with temporal resolutions ranging from 1 minute to 10 minutes to 1 hour. Examples of such data come from observation networks operated by institutions such as the Baseline Surface Radiation Network, the Bureau of Meteorology (Australia), the National Renewable Energy Laboratory (USA), the National Oceanic and Atmospheric Administration (NOAA), and the South African Universities Radiation Network. The latter consists of gridded data with spatial resolutions such as 0.5° × 0.625° (latitude × longitude) and temporal resolutions ranging from 0.5 hours to 1 hour. Examples of such data include MERRA-2, CAMS-RAD, SARAH-2, SARAH-E, CERES-SYN1deg, Solcast, ERA5, and ECMWF.

[0004] With the joint efforts of scientists, research organizations, and relevant departments in various countries around the world, an increasing number of ground-based remote sensing stations and geostationary meteorological satellites are being put into use, providing researchers with a massive amount of new remote sensing data every moment. However, not all remote sensing data sources are easily accessible; some are not publicly available, some require registration, and some require writing custom programs to download them. Dazhi Yang developed an R package for collecting data from a solar radiation observation network at five locations and upgraded the package to include BSRN data retrieval. JamieBright et al. developed a Python package for downloading, extracting, and using the MERRA-2 grid reanalysis database. Xixi Sun developed a Python package for downloading, extracting, and using Himawari-8 / 9 satellite data. However, research on large-scale solar irradiance data acquisition remains scarce, making solar irradiance data a high-barrier-to-entry point. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by proposing an open-source method, system, and storage medium for dynamic acquisition of solar irradiance data, providing data support and application support for future research in fields such as solar energy resource assessment, prediction, and large-scale development and application of photovoltaic power generation.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0007] An open-source method for dynamically acquiring solar irradiance data includes the following steps:

[0008] S1. Analysis of open-source data selection and collection conditions

[0009] The open-source data screening method is as follows: multiple global solar irradiance data are screened to select data sources that can be used for solar irradiance prediction and solar energy resource assessment. Meta-information of the selected data is obtained, and data with long running time, high reliability, and high spatiotemporal resolution are selected as open-source solar irradiance data dynamic acquisition objects based on the meta-information.

[0010] The analysis of open-source data collection conditions involves analyzing the patterns in obtaining data addresses from open-source solar irradiance data source websites. This includes statistical analysis of factors such as the format of the data to be collected, whether a login account and password are required, whether FTP connection is required, and whether data format transformation is necessary. Based on this analysis, the open-source data website collection conditions are categorized into general and special types, and collection rules for open-source solar irradiance data sources are formulated accordingly.

[0011] S2, Data Acquisition

[0012] S2-1. Set the collection rules for general open source data source websites and collect data;

[0013] Specifically, the general open-source data source website collection rules are as follows: based on the Uniform Resource Locator (URL) of the data address, the data generation time, the time interval, and the site, generate a URL for the user-specified site, time, and time interval, locate the data location using the URL, and execute the data collection task.

[0014] S2-2, Set specific open-source data source website collection rules and perform data collection;

[0015] Specifically, the data collection rules for the special open-source data source website are as follows: log in to the FTP server of the special website by entering an account and password, and execute the data collection task through the server interface.

[0016] S2-3. Collect, clean, and store the data.

[0017] Preferably, the metadata includes type, acquisition device, acquisition frequency, acquisition time, and maintenance frequency.

[0018] Preferably, in step S2-1, the method for collecting data according to the general open-source data source website collection rules is as follows:

[0019] S2-1-1. Create a new task, select general data source X, select a site under data source X, and give the start time and end time of the data to be collected in the form of year, month and day.

[0020] S2-1-2. Obtain the general format of the Uniform Resource Locator (URL) of the data source X by its name, map the selected site to the site number, standardize the data generation time, and finally associate the general format of the URL, the site number, and the standardized time to form the specific URL of the data. Generate all data collection tasks with start time and end time in a loop.

[0021] S2-1-3. Execute the data acquisition task and export the data. Run the configured acquisition task. After the data acquisition is completed, convert the original zip or dat format of the data and export it as csv.

[0022] Preferably, in step S2-2, the method for collecting data according to the collection rules of the specific open-source data source website is as follows:

[0023] S2-2-1. Create a new task, select a special open source data source Y, enter the login account and password of the data source website, and log in to the FTP server of the website.

[0024] S2-2-2: Obtain the general format of the data source path prefix through the name of data source Y, standardize the data generation time, associate the general format of the path prefix, the site name, and the standardized time to form the specific data collection path, and generate all data collection tasks with start time and end time in a loop.

[0025] S2-2-3. Execute the data acquisition task and export the data. Run the configured acquisition task. After the data acquisition is completed, convert the original dat.gz format of the data and export it as CSV.

[0026] Preferably, the method for data cleaning and storage in steps S2-3 is as follows:

[0027] S2-3-1. Read the total solar irradiance G, direct solar irradiance B, acquisition time T, location longitude Lon, and latitude Lat from the CSV file, and convert the acquisition time T into UTC.

[0028] S2-3-2. Set the physical upper and lower limits for G and B, and calculate the extraterrestrial solar irradiance as the upper limit.

[0029] S2-3-3, Handling missing values;

[0030] S2-3-4. Filter out G and B records within the time period when the solar zenith angle θ ≥ 85°, mark the corresponding G and B as missing values, and calculate θ using T and latitude and longitude.

[0031] S2-3-5. Import the cleaned data directly into the MySQL database.

[0032] This invention also provides an open-source dynamic solar irradiance data acquisition system, including a backend main service program, an acquisition rule configuration frontend, and a data storage cluster.

[0033] The data collection rule configuration front-end is used to configure data collection rules and send the configured collection rules to the back-end main service program.

[0034] The backend main service program is used to collect data according to the configured collection rules sent by the frontend, clean the collected data, and finally import it into the data storage cluster.

[0035] Preferably, the backend main service program, the collection rule configuration frontend, and the data storage cluster are all deployed on a cloud server.

[0036] As a preferred approach, the open-source solar irradiance data dynamic acquisition system is integrated into the solar irradiance data application system. The solar irradiance data application system deployed on the cloud server is connected to the open-source solar irradiance data dynamic acquisition system. The open-source solar irradiance data dynamic acquisition system regularly updates the open-source solar irradiance data and historical solar irradiance data. The solar irradiance data application system then uses the exported data for its applications.

[0037] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0038] This invention has the following characteristics and beneficial effects:

[0039] The method disclosed in this invention selects data sources with long operating times, high reliability, and high quality as the targets for dynamic acquisition of open-source solar irradiance data based on information such as the display method, acquisition technology, acquisition time, and acquisition frequency of each data source. The selected open-source irradiance data websites are analyzed sequentially to customize the data acquisition strategy for each website. Following the generation rules of Uniform Resource Locators (URLs) pointing to data addresses, the corresponding data addresses (URLs) for the data acquisition rules are generated from existing data to complete the data acquisition. Special rules are applied to websites requiring account and password login to FTP servers. To address the issue of heterogeneous and low-quality data from different data source websites, quality control is performed on the downloaded data, including but not limited to the selection of data features, data type conversion, and handling of noise, outliers, and missing values.

[0040] The method disclosed in this invention provides researchers and related photovoltaic companies with convenient data access by utilizing an open-source solar irradiance data dynamic acquisition system, reducing data barriers and providing data support and application guarantee for solar energy resource assessment and prediction, thereby improving the feasibility of large-scale development and application of photovoltaic power generation and contributing to my country's energy structure reform. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the method disclosed in Embodiment 1 of the present invention;

[0043] Figure 2This is a technical roadmap of the method disclosed in Embodiment 1 of the present invention;

[0044] Figure 3 This is a schematic diagram illustrating the configuration of a general data source 1 for data collection in the method disclosed in Embodiment 1 of the present invention;

[0045] Figure 4 This is a schematic diagram illustrating the configuration of a general data source 2 for data collection in the method disclosed in Embodiment 1 of the present invention;

[0046] Figure 5 This is a schematic diagram illustrating the configuration of a general data source 3 for data collection in the method disclosed in Embodiment 1 of the present invention;

[0047] Figure 6 This is a schematic diagram illustrating the configuration of a general data source 4 for data acquisition in the method disclosed in Embodiment 1 of the present invention;

[0048] Figure 7 This is a schematic diagram illustrating the configuration of the special data source 5 for data collection in the method disclosed in Embodiment 1 of the present invention;

[0049] Figure 8 This is a schematic diagram of the system architecture for the integrated application of the irradiance data acquisition system and the solar radiation system in the method disclosed in Embodiment 1 of the present invention. Detailed Implementation

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0051] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0052] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0053] This invention discloses an open-source method for dynamically acquiring solar irradiance data. Figure 1 Sections 2 and 3 respectively illustrate the system structure and method flow of the present invention. The present invention will now be described in detail with reference to Embodiment 1, specifically including the following steps:

[0054] Step 1: Filtering and analyzing the collection criteria for open-source data:

[0055] Step 11: Conduct research and analysis on open source solar irradiance data. Research and screen multiple global solar irradiance data sources, select data sources that can be used for solar irradiance prediction and solar energy resource assessment, and analyze data metadata such as data type, acquisition equipment, acquisition frequency, acquisition time, and maintenance frequency. Data with long running time, high reliability, and high spatiotemporal resolution will be selected as the objects for dynamic acquisition of open source solar irradiance data.

[0056] Step 12: Study the collection characteristics of open-source solar irradiance data. Analyze the surveyed open-source solar irradiance data source websites, focusing on statistical analysis of aspects such as the format of the data to be collected, whether a login account and password are required, whether FTP connection is required, and whether data format transformation is required. Analyzing the format of the data to be collected determines whether the raw data needs to be filtered. Analyzing the Uniform Resource Locators (URLs) of the open-source solar irradiance data source websites identifies the patterns between these URLs and the data. Analyzing whether a login account and password are required determines whether account registration is necessary. Analyzing whether FTP connection is required determines whether to log in to the other party's FTP server to collect data. Analyzing the specific data format determines whether the data format needs to be changed to achieve format uniformity. Based on the above analysis, the collection conditions of open-source data websites are divided into two categories: general and special, and collection rules for open-source solar irradiance data sources are formulated for each category.

[0057] Step 2, Design of data acquisition method:

[0058] Step 21: Define custom collection rules for general open source data source websites. Based on the Uniform Resource Locator (URI) of the data address, data generation time, time interval, site, and other information, summarize the generation rules of the URI and generate a URI for the data with user-specified site, time, time interval, and other metadata. Use this URI to locate the data and execute the data collection task.

[0059] This embodiment provides a first example of a typical open-source data source website: such as Figure 3 As shown, the steps for collecting data from a general open-source data source website 1 (URL: https: / / reg.bom.gov.au / climate / reg / oneminsolar / , name: BoM) are as follows:

[0060] Create a new task, select data source 1, select a site under data source 1, and specify the start and end times of the data to be collected in the form of year, month and day.

[0061] The general format of the Uniform Resource Locator (URL) for the data source is obtained by using the name of data source 1. The selected site is mapped to the site number, the data generation time is standardized, and finally the general format of the URL, the site number, and the standardized time are associated to form the specific URL for the data. All data collection tasks with start and end times are generated in a loop.

[0062] The general format of the Uniform Resource Locator (URL) for data from data source 1 is: https: / / reg.bom.gov.au / cgi-bin / climate / oneminsolar / getFile.cgi?stn_num=016001&year=2013&month=01, where 016001 is the site number, corresponding one-to-one with the site, 2013 is the year, and 01 is the month. (Site number, year, month) uniquely identifies a piece of data.

[0063] Execute the data acquisition task and export the data. Run the configured acquisition task, and after the data acquisition is completed, convert the original zip or dat format of the data and export it as CSV.

[0064] This embodiment provides a second example of a typical open-source data source website: such as Figure 4 As shown, the steps for collecting data from a general open-source data source website 2 (URL: https: / / gml.noaa.gov / aftp / data / radiation / solrad / , name: NOAASOLRAD) are as follows:

[0065] Create a new task, select data source 2, select a site under data source 2, and specify the start and end times of the data to be collected in the form of year, month and day.

[0066] The general format of the Uniform Resource Locator (URL) for the data source is obtained by using the name of data source 2. The data generation time is standardized. Finally, the general format of the URL, the site name, and the standardized time are associated to form the specific URL for the data. All data collection tasks with start and end times are generated in a loop.

[0067] The general format of the Uniform Resource Locator (URL) for data from data source 2 is: https: / / gml.noaa.gov / aftp / data / radiation / solrad / abq / 2023 / abq23001.dat. Here, "abq" represents the site name, "23" represents the year, "001" represents the first day of the year 2023, and "dat" represents the data file format.

[0068] Execute the data acquisition task and export the data. Run the configured acquisition task, and after the data acquisition is completed, convert the original zip or dat format of the data and export it as CSV.

[0069] This embodiment provides a third example of a typical open-source data source website: such as Figure 5 As shown, the steps for collecting data from a typical open-source data source website 3 (URL: https: / / sauran.ac.za / , name: SAURAN) are as follows:

[0070] Create a new task, select data source 3, select a site under data source 3, and specify the start and end times of the data to be collected in the form of year, month and day.

[0071] The general format of the Uniform Resource Locator (URL) for the data source is obtained by using the name of data source 3. The data generation time is standardized, and the data time interval is set to every minute. Finally, the general format of the URL, the site name, the time interval, and the standardized time are associated to form the specific URL for the data. Finally, all data collection tasks with start and end times are generated in a loop.

[0072] The general format of the Uniform Resource Locator (URL) for data source 3 is: https: / / sauran.ac.za / api / DataDownload / KZH / Day / 2021-5-27 / 2021-7-28. Here, KZH is the site name, 2021-5-27 is the data start time, and 2021-7-28 is the data end time.

[0073] Execute the data acquisition task and export the data. Run the configured acquisition task, and after the data acquisition is completed, convert the original zip or dat format of the data and export it as CSV.

[0074] This embodiment provides a fourth example of a typical open-source data source website: such as Figure 6 As shown, the steps for collecting data from a typical open-source data source website 4 (URL: http: / / solardat.uoregon.edu / SelectArchivalUpdatedFormat.html, name: Uoregon) are as follows:

[0075] Create a new task, select data source 4, select a site under data source 4, and specify the start and end times of the data to be collected in the form of year, month and day.

[0076] The general format of the Uniform Resource Locator (URL) for the data source is obtained by using the name of data source 4, and the data generation time is standardized. Because the website's data history is very long, the data format has changed somewhat, so different strategies need to be developed for older and newer data. Finally, the general format of the URL, the site name, the standardized time, and the new strategy are associated to form the specific URL for the data. Finally, all data collection tasks with start and end times are generated in a loop.

[0077] The general format of the Uniform Resource Locator (URL) for Data Source 4 is: http: / / solardat.uoregon.edu / download / ArchiveUpdatedFormat / SIO_Updated / SIO_2002_ComprehensiveFormat / SIO_05_2002-12.csv. Here, SIO represents the site name, 2002 represents the year, 05 represents the sampling interval (sampling every 5 minutes), and 12 represents the month. Data from different time periods may have different sampling intervals; some data may have a sampling interval of 01, indicating sampling every 1 minute. Because this data source is old, the general format of the URL for some early data is: "http: / / solardat.uoregon.edu / download / ArchiveUpdatedFormat / DIM_Updated / Step4_DIM_2002 / DIM_05_2002-10.csv". Additionally, some data with excessively long sampling intervals cannot meet the needs of solar energy forecasting and are therefore not collected.

[0078] Execute the data acquisition task and export the data. Run the configured acquisition task, and after the data acquisition is completed, convert the original zip or dat format of the data and export it as CSV.

[0079] Step 22: Define custom data collection rules for special open-source data source websites. Log in to the FTP server of the special website by entering your username and password, and execute the data collection task through the server interface.

[0080] This embodiment provides the first example of a special open-source data source website: such as Figure 7 As shown, the steps for collecting data from a special open-source data source website (URL: https: / / bsrn.awi.de / project / background / , name: BSRN) are as follows:

[0081] Create a new task, select data source 5 corresponding to the special open source data source website, enter the login account and password of the data source website, and log in to the FTP server of the website;

[0082] The general format of the data source path prefix is ​​obtained by the name of the data source Y (data source 5). The data generation time is standardized. The general format of the path prefix, the site name, and the standardized time are associated to form the specific data collection path. All data collection tasks with start time and end time are generated in a loop.

[0083] The general format of the data address for data source 5 is: abs / 2019-01 / 2019-10, where abs represents the site name, 2019-01 represents the data start time, and 2019-10 represents the data end time.

[0084] Execute the data acquisition task and export the data. Run the configured acquisition task, and after the data acquisition is completed, convert the original dat.gz format of the data and export it as CSV.

[0085] Step 23, data cleaning and storage: Through data quality control methods, poor quality data is filtered out and missing data is filled to achieve data cleaning. The cleaned data is then periodically imported into the MySQL database.

[0086] In step 23, the data cleaning and data entry steps are as follows:

[0087] Read the total solar irradiance G, direct solar irradiance B, acquisition time T, location longitude Lon, and latitude Lat from the CSV file, and convert the acquisition time T into UTC.

[0088] Set physical upper and lower limits for G and B, and calculate the extraterrestrial solar irradiance E. extAs an upper limit, specifically, extraterrestrial solar irradiance E ext The calculation formula is as follows:

[0089]

[0090] Where d is the dth day of the year represented by the collection time T, and 0 is the physical lower limit;

[0091] Missing value handling: No processing is performed for 10 or more consecutive missing values, and Newton interpolation is used to fill in 9 or fewer missing values;

[0092] Filter out G and B records within the time period when the solar zenith angle θ ≥ 85°, and mark the corresponding G and B as missing values. θ is calculated using T and latitude and longitude.

[0093] The cleaned data is directly imported into the MySQL database for subsequent data export and application.

[0094] This embodiment also provides an open-source dynamic solar irradiance data acquisition system, including a backend main service program, an acquisition rule configuration frontend, and a data storage cluster.

[0095] The data collection rule configuration front-end is used to configure data collection rules and send the configured collection rules to the back-end main service program.

[0096] The backend main service program is used to collect data according to the configured collection rules sent by the frontend, clean the collected data, and finally import it into the data storage cluster.

[0097] Understandably, the entire system is deployed on a cloud server. In addition, the solar irradiance data application system is integrated with the open-source solar irradiance data dynamic acquisition system. The open-source solar irradiance data dynamic acquisition system regularly updates open-source solar irradiance data and historical solar irradiance data. The solar irradiance data application system develops applications based on the exported data.

[0098] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0099] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0100] It should be noted that, in the embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some service interfaces, devices, or units, and may be electrical or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0104] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0105] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. An open-source method for dynamically acquiring solar irradiance data, characterized in that, Includes the following steps: S1. Analysis of open-source data selection and collection conditions The open-source data screening method is as follows: multiple global solar irradiance data are screened to select data sources that can be used for solar irradiance prediction and solar energy resource assessment. Meta-information of the selected data is obtained, and data with long running time, high reliability, and high spatiotemporal resolution are selected as open-source solar irradiance data dynamic acquisition objects based on the meta-information. The analysis of open-source data collection conditions involves: sequentially analyzing the patterns of obtaining data addresses from open-source solar irradiance data source websites; statistically analyzing the format of the data to be collected, whether a login account and password are required, whether FTP connection is required, and whether data format transformation is required; based on the above analysis, the open-source data website collection conditions are divided into two categories: general and special, and collection rules for open-source solar irradiance data sources are formulated accordingly. S2, Data Acquisition S2-1. Set general open source data source website collection rules and perform data collection; the general open source data source website collection rules are as follows: based on the Uniform Resource Locator (URL) of the data address, the data generation time, the time interval, and the site, generate a URL for the user-specified site, time, and time interval, locate the data location through the URL, and execute the data collection task. S2-2. Set special open source data source website collection rules and perform data collection; the special open source data source website collection rules are as follows: log in to the FTP server of the special website by entering an account and password, and execute the data collection task through the server interface; S2-3. Collect, clean, and store the data. The method for cleaning and storing the data is as follows: S2-3-1. Read the total solar irradiance G, direct solar irradiance B, acquisition time T, location longitude Lon, and latitude Lat from the CSV file, and convert the acquisition time T into UTC. S2-3-2. Set the physical upper and lower limits for G and B, calculate the extraterrestrial solar irradiance and use it as the upper limit. S2-3-3, Handling missing values; S2-3-4, Filtering solar zenith angle For G and B values ​​recorded within a time period, the corresponding G and B values ​​are marked as missing values; S2-3-5. Import the cleaned data directly into the MySQL database.

2. The open-source solar irradiance data dynamic acquisition method according to claim 1, characterized in that, The metadata includes type, acquisition device, acquisition frequency, acquisition time, and maintenance frequency.

3. The open-source solar irradiance data dynamic acquisition method according to claim 1, characterized in that, In step S2-1, the method for collecting data according to the general open-source data source website collection rules is as follows: S2-1-1. Create a new task, select general data source X, select a site under data source X, and give the start time and end time of the data to be collected in the form of year, month and day. S2-1-2. Obtain the general format of the Uniform Resource Locator (URL) of the data source X by its name, map the selected site to the site number, standardize the data generation time, and finally associate the general format of the URL, the site number, and the standardized time to form the specific URL of the data. Generate all data collection tasks with start time and end time in a loop. S2-1-3. Execute the data acquisition task and export the data. Run the configured acquisition task. After the data acquisition is completed, convert the original zip or dat format of the data and export it as csv.

4. The open-source solar irradiance data dynamic acquisition method, system, and storage medium according to claim 1, characterized in that, In step S2-2, the method for collecting data according to the collection rules of the special open-source data source website is as follows: S2-2-1. Create a new task, select a special open source data source Y, enter the login account and password of the data source website, and log in to the FTP server of the website. S2-2-2: Obtain the general format of the data source path prefix through the name of data source Y, standardize the data generation time, associate the general format of the path prefix, the site name, and the standardized time to form the specific data collection path, and generate all data collection tasks with start time and end time in a loop. S2-2-3. Execute the data acquisition task and export the data. Run the configured acquisition task. After the data acquisition is completed, convert the original dat.gz format of the data and export it as CSV.

5. An open-source dynamic solar irradiance data acquisition system, implementing the steps of the method as described in any one of claims 1-4, characterized in that, This includes the backend main service program, the frontend for configuring data collection rules, and the data storage cluster. The data collection rule configuration front-end is used to configure data collection rules and send the configured collection rules to the back-end main service program. The backend main service program is used to collect data according to the configured collection rules sent by the frontend, clean the collected data, and finally import it into the data storage cluster.

6. The open-source solar irradiance data dynamic acquisition system according to claim 5, characterized in that, The backend main service program, the frontend for configuring collection rules, and the data storage cluster are all deployed on cloud servers.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-4.

Images