Method and system for dynamically acquiring open source solar irradiance data and storage medium

Abstract

A method for dynamically acquiring open source solar irradiance data. Global solar irradiance data is screened and a data source is selected that is capable of use for predicting solar irradiance and assessing a solar energy resource. Meta-information of the data source is used to select data for acquisition with a long running time, high reliability and high spatio-temporal resolution. Rules for obtaining an address of the data (e.g. URL) from a website are analysed. Statistical analysis determines the data format, account password requirements, and need for data reformatting. Based on the analysis, the website is categorised into a general condition or a special condition, and a rule for acquiring the data is formulated. Data acquisition is performed according to the formulated rules for general and special websites. Acquired data is cleaned and stored in a data warehouse. An associated system has a front-end for configuring a data acquisition rule and sending the rule to a back-end program for data acquisition, cleaning and storage.

Description

TECHNICAL FIELD

[0001] The present disclosure belongs to the technical field of calculation and processing of solar irradiance data, and in particular to a method and system for dynamically acquiring open source solar irradiance data and a storage medium. BACKGROUND

[0002] Based on accounting data of the National Energy Administration, in 2021, a photovoltaic power generating capacity accounted for only 3.9% of a total electricity consumption and 13.1% of a renewable energy generating capacity in China. Photovoltaic power generation needs further development. Assessment and prediction on solar energy resources are not separated from a support of historical solar irradiance data. The historical solar irradiance data is used to accurately assess solar power potential in a region, and predict future solar irradiance. The historical solar irradiance data is of great significance for large-scale development and application of the photovoltaic power generation. However, the historical solar irradiance data comes from complex sources, are not easy to obtain, have variable quality, and need to go through a complicated pre-process before being used to assess or predict solar resources.

[0003] The historical solar irradiance data is divided into two types. One type is obtained by ground observation, and the other type is obtained by satellite-derived reanalysis. The historical solar irradiance data obtained by ground observation is a value obtained through observation at a point, for example, the Baseline Surface Radiation Network (BSRN), The Bureau of Meteorology (BOM), The National Renewable Energy Laboratory (NREL), The National Oceanic Atmospheric Administration (NOAA), and the South African Universities Radiation Network (SAURN), with time accuracy of 1 minute, 10 minutes, and 1 hour. The historical solar irradiance data obtained by satellite-derived reanalysis is in a grid form, for example, MERRA-2, CAMS-RAD, SARAH-2, SARAH-E, CERES-SYNldeg, Solcast, ERA5, and ECMWF, with a spatial resolution of 0.5° x 0.6235° latitude x longitude and a temporal resolution of 0.5 hours to 1 hour.

[0004] With the joint efforts of scientists, research organizations, and national authorities around the world, more ground-based remote sensing observation stations and geosynchronous meteorological satellites are put into use. Therefore, a huge amount of new remote sensing data is available for researchers at all hours. However, not all source data of remote sensing data is easy to obtain. A part of the data is not open to the outside, a part of the data needs to be applied for registration, and a part of the data needs to be downloaded by self-programming. Dazhi Yang develops an R package for collecting data from solar radiation observation networks at five locations, and on this basis, upgrades the R package, that is, adds retrieval of BSRN data. Jamie Bright develops a Python package for downloading, extracting, and using a grid reanalysis database MERRA-2. Xixi Sun develops a Python package for downloading, extracting, and using Himawari-8 / 9 satellite data. However, there are still few researches on large-scale solar irradiance data acquisition. As a result, solar irradiance data thresholds are high. SUMMARY

[0005] In view of shortages in the prior art, the present disclosure provides a method and system for dynamically acquiring open source solar irradiance data and a storage medium, to provide a data support and an application support for researches in the fields of assessment and prediction on solar energy resources and large-scale development and application of photovoltaic power generation in the future.

[0006] In order to resolve the above technical problem, the technical solution used by the present disclosure is as follows:

[0007] A method for dynamically acquiring open source solar irradiance data includes:

[0008] SI: screening open source data, and analyzing an acquisition condition for the open source data, where

[0009] the screening open source data includes: screening a plurality pieces of global solar irradiance data, selecting a data source that is capable of being used to predict solar irradiance and assess a solar energy resource, obtaining meta-information of the data source that is capable of being used to predict solar irradiance and assess a solar energy resource, and determining, based on the meta-information, data with a long running time, high reliability, and a high temporal-spatial resolution as a dynamic acquisition object for the open source solar irradiance data; and

[0010] the analyzing the acquisition condition for the open source data includes: sequentially analyzing a law of obtaining an address of data from an open source solar irradiance data source website, and performing statistical analysis in terms of a form of data to be acquired, whether to enter an account and a password, whether to perform association in an ftp manner, and whether to change a data format; and dividing, based on the statistical analysis, an acquisition condition for the open source solar irradiance data source website into a general condition and a special condition, and formulating a rule of acquiring source data of the open source solar irradiance data; and

[0011] S2: acquiring the open source data, where the acquiring open source data specifically includes:

[0012] S2-1: formulating a rule of acquiring data from a general open source data source website, and performing data acquisition, where

[0013] the rule of acquiring data from a general open source data source website is as follows: generating, based on a uniform resource locator for a data address, a data generation time, a time interval, and a site, a uniform resource locator that is specified by a user for the site, the data generation time, and the time interval, positioning a data location based on the uniform resource locator, and performing a data acquisition task;

[0014] S2-2: formulating a rule of acquiring data from a special open source data source website, and performing data acquisition.

[0015] Specifically, the rule of acquiring data from a special open source data source website is as follows: entering the account and a password, logging in to an ftp server of the special open source data source website, and performing a data acquisition task through a server interface; and

[0016] S2-3: performing data cleaning and data warehousing.

[0017] Preferably, the meta-information includes a type, an acquisition device, an acquisition frequency, an acquisition time, and a maintenance frequency.

[0018] Preferably, in step S2-1, the formulating a rule of acquiring data from a general open source data source website, and performing data acquisition includes:

[0019] S2-1-1: creating a new task, selecting a general open source data source X, selecting a site under the data source X, and determining a start time and an end time of the data to be acquired in a form of month day, year;

[0020] S2-1-2: obtaining a general format of a uniform resource locator for a data source based on a name of the data source X, mapping the site into a site number, standardizing the data generation time, associating the general format of the uniform resource locator, the site number, and the data generation time to form a specific uniform resource locator for the data to be acquired, and generating, through circulation, all data acquisition tasks within the start time and the end time; and

[0021] S2-1-3: performing the data acquisition task and data exporting, specifically including: running the data acquisition task, after data exporting is completed, converting an original zip format or dat format of the data to be acquired, and uniformly exporting the data to be acquired in a csv format, where

[0022] Preferably, in step S2-2, the formulating a rule of acquiring data from a general open source data source website, and performing data acquisition includes:

[0023] S2-2-1: creating a new task, selecting a special open source data source Y, entering an account and a password of the special open source data source website, and logging in to an ftp server of the special open source data source website;

[0024] S2-2-2: obtaining a general format of a data source path prefix based on a name of the data source Y, standardizing the data generation time, associating the general format of the data source path prefix, the site name, and the data generation time to form a specific data acquisition, and generating, through calculation, all data acquisition tasks within the start time and the end time; and

[0025] S2-2-3: performing the data acquisition task and data exporting, specifically including: running the data acquisition task, after data exporting is completed, converting an original dat.gz format of the data to be acquired, and uniformly exporting the data to be acquired in a csv format.

[0026] Preferably, in step S2-3, the performing data cleaning and data storing includes:

[0027] S2-3-1: reading a total solar irradiance G, a direct solar irradiance B, an acquisition time T, a longitude (Lon) of a location, and a latitude (Lat) of the location in the csv file, and converting an acquisition time T into a coordinated universal time;

[0028] S2-3-2: setting physical upper limits and physical lower limits of G and B, calculating an extraterrestrial solar irradiance, and determining the extraterrestrial solar irradiance as the physical upper limit;

[0029] S2-3-3: processing a missing value;

[0030] S2-3-4: filtering G and B recorded during a period of time at which a solar zenith angle is 9 > 85°, and marking G and B as missing values, where 9 is obtained through calculation based on T, the longitude, and the latitude; and

[0031] S2-3-5: directly importing data cleaned into a mysql database.

[0032] The present disclosure further provides a system for dynamically acquiring open source solar irradiance data. The system for dynamically acquiring open source solar irradiance data includes a back-end main service program, an acquisition rule configuration front-end, and a data storage cluster.

[0033] The acquisition rule configuration front-end is used to configure a data acquisition rule, and send the data acquisition rule to the back-end main service program.

[0034] The back-end main service program is used to acquire data based on the data acquisition rule sent by the acquisition rule configuration front-end, clean the data, and finally import the data into the data storage cluster.

[0035] Preferably, the back-end main service program, the acquisition rule configuration front-end, and the data storage cluster are deployed on a cloud server.

[0036] Preferably, applications of the system for dynamically acquiring open source solar irradiance data are integrated in a solar irradiance data application system, deployed in a solar irradiance data application system on the cloud server, and connected to the system for dynamically acquiring open source solar irradiance data. The system for dynamically acquiring open source solar irradiance data regularly updates the open source solar irradiance data and historical solar irradiance data. The solar irradiance data application system carries out an application based on exported data.

[0037] The present disclosure further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the foregoing method.

[0038] The present disclosure has the following advantages and beneficial effect.

[0039] In the method provided in the present disclosure, a data source with a long running time, high reliability, and high quality is screened out based on information about a present manner, an acquisition technology, an acquisition time, and an acquisition frequency, and used as the dynamic acquisition object for the open source solar irradiance data. The open source irradiance data websites are analyzed in sequence, to complete customization of data acquisition strategies for each website. According to a law of generating the uniform resource locator for the data address, a data address corresponding to a data acquisition rule is reversely generated based on existing data, that is, data acquisition is completed based on the uniform resource locator. In addition, special rules are formulated for logging in to an ftp server of a website by entering an account and a password. Because data from different data source websites is heterogeneous data with poor quality, quality of downloaded data needs to be controlled, including, but is not limited to, selection of a data feature, conversion of a data type, and processing of noise, abnormal data, and a missing value.

[0040] In the method provided in the present invention, the system for dynamically acquiring open source solar irradiance data provides convenient data access for researchers and related photovoltaic enterprises. This reduces a data barrier, and provides a data support and an application guarantee for assessment and prediction on solar energy resources. Therefore, feasibility of large-scale development and application of photovoltaic power generation is improved, to support reform of an energy structure in China. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly describe a technical solution in the embodiments of the present disclosure or in the prior art, a brief introduction to the accompanying drawings required for the description of the embodiments or the prior art will be provided below. Obviously, the accompanying drawings in the following description are merely some embodiments of the present disclosure. Those of ordinary skill in the art would also to derive other accompanying drawings from these accompanying drawings without making inventive efforts.

[0042] FIG. 1 is a schematic flowchart of a method according to an embodiment 1 of the present disclosure;

[0043] FIG. 2 is a diagram of a technical route of a method according to an embodiment 1 of the present disclosure;

[0044] FIG. 3 is a schematic diagram of configuration of an acquisition task from a general open source data source 1 in a method according to an embodiment 1 of the present disclosure;

[0045] FIG. 4 is a schematic diagram of configuration of an acquisition task from a general open source data source 2 in a method according to an embodiment 1 of the present disclosure;

[0046] FIG. 5 is a schematic diagram of configuration of an acquisition task from a general open source data source 3 in a method according to an embodiment 1 of the present disclosure;

[0047] FIG. 6 is a schematic diagram of configuration of an acquisition task from a general open source data source 4 in a method according to an embodiment 1 of the present disclosure;

[0048] FIG. 7 is a schematic diagram of configuration of an acquisition task from a general open source data source 5 in a method according to an embodiment 1 of the present disclosure; and

[0049] FIG. 8 is a schematic diagram of a system architecture of application integration of an irradiance data acquisition system with a solar radiation system in a method according to an embodiment 1 of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] It should be noted that embodiments in the present disclosure or features in the embodiments may be combined with one another without conflict.

[0051] It should be understood that in the description of the present disclosure, terms such as "central", "longitudinal", "transverse" "upper", "lower", "front", "rear", "left", "right" "vertical", "horizontal", "top", "bottom", "inside" and "outside" indicate the orientations or positional relationships based on the drawings, and these terms are merely intended to facilitate and simplify the description of the present disclosure, rather than to indicate or imply that the mentioned apparatus or element must have a specific orientation or must be constructed and operated in a specific orientation, and thus cannot be construed as limitations to the present disclosure. Moreover, terms such as "first" and "second" are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features denoted. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present disclosure, unless otherwise specified, "a plurality of' means two or more.

[0052] In the description of the present disclosure, it should be noted that, unless otherwise clearly specified, meanings of terms "mount", "connected with", and "connected to" should be understood in a board sense. For example, the connection may be a fixed connection, a removable connection, or an integral connection; may be a mechanical connection or an electrical connection; may be a direct connection or an indirect connection by using an intermediate medium; or may be intercommunication between two elements. A person of ordinary skill in the art may understand specific meanings of the above terms in the present disclosure based on a specific situation.

[0053] The present disclosure provides a method for dynamically acquiring open source solar irradiance data. FIG. 1 and FIG. 2 respectively shows a system structure and a method flow in the present disclosure. The present disclosure is described in detail below with an embodiment 1. The method for dynamically acquiring open source solar irradiance data includes:

[0054] Step 1: Screen open source data, and analyze an acquisition condition for the open source data:

[0055] Step 11: Perform investigation and analysis on the open source solar irradiance data, screen a plurality of pieces of global solar irradiance data, select a data source that is capable of being used to predict solar irradiance and assess a solar energy resource, analyze meta-information of the data source that is capable of being used to predict solar irradiance and assess a solar energy resource, such as a type, an acquisition device, an acquisition frequency, an acquisition time, and a maintenance frequency, and determine data with a long running time, high reliability, and a high temporal-spatial resolution as a dynamic acquisition object for the open source solar irradiance data.

[0056] Step 12: Research an acquiring feature of the open source solar irradiance data, sequentially analyze an open source solar irradiance data source website, perform statistical analysis in terms of a form of data to be acquired, whether to enter an account and a password, whether to perform association in an ftp manner, and whether to change a data format, and determine, based on the form of the data to be acquired, whether to screen the raw data by analyzing the form of the data to be collected; obtain, by analyzing a uniform resource locator for the open source solar irradiance data source website, and identifying a law between the uniform resource locator and the open source solar irradiance data; determine, by analyzing whether to enter an account and a password, whether to register the account; determine, by analyzing whether to perform association in the ftp manner, whether to log in to an ftp server of another party and perform data acquisition; determine, by analyzing a specific format of the data, whether to change the format of the data, to implement a uniform format; and dividing, based on the statistical analysis, an acquisition condition for the open source solar irradiance data source website into a general condition and a special condition, and formulating a rule of acquiring source data of the open source solar irradiance data; and

[0057] Step 2: Design a data acquisition method:

[0058] Step 21: Customize a rule of acquiring data from a general open source data source website, summarize, based on a uniform resource locator for a data address, a data generation time, a time interval, and a site, a rule of generating a uniform resource locator that is specified by a user for meta-information such as the site, the data generation time, and the time interval, positioning a data location based on the uniform resource locator, and performing a data acquisition task.

[0059] A first case of the general open source data source website provided in this embodiment: As shown in FIG. 3, steps for acquiring data from a general open source data source website 1 (https: / / reg.bom.gov.au / climate / reg / oneminsolar / , name: BoM) include:

[0060] Create a new task, select a data source 1, select a site under the data source 1, and determine 7 a start time and an end time of data to be acquired in a form of month day, year.

[0061] Obtain a general format of a uniform resource locator for a data source based on a name of the data source 1, map the site into a site number, standardize a data generation time, associate the general format of the uniform resource locator, the site number, and the data generation time to form a specific uniform resource locator for the data to be acquired, and generate, through circulation, all data acquisition tasks within the start time and the end time of the data to be acquired.

[0062] A general format of a uniform resource locator for data from the data source 1 is as follows: https: / / reg.bom.gov.au / cgi-bin / climate / oneminsolar / getFile.cgi?stn_num= 016001&year=2013&month=01, 016001 represents the site number and is in a one-to-one correspondence to the site, 2013 represents the year, and 01 represents the month, and (the site number, the year, or the month) uniquely indicates one piece of data.

[0063] Perform the data acquisition task and data exporting, run the data acquisition task, after data exporting is completed, convert an original zip format or dat format of the data to be acquired, and uniformly export the data to be acquired in a csv format.

[0064] A second case of the general open source data source website provided in this embodiment: As shown in FIG. 4, steps for acquiring data from a general open source data source website 2 (https: / / gml.noaa.gov / aftp / data / radiation / solrad / , name: NOAA SOLRAD) include:

[0065] Create a new task, select a data source 2, select a site under the data source 2, and determine a start time and an end time of data to be acquired in a form of month day, year.

[0066] Obtain a general format of a uniform resource locator for a data source based on a name of the data source 2, standardize a data generation time, associate the general format of the uniform resource locator, the site name, and the data generation time to form a specific uniform resource locator for the data to be acquired, and generate, through circulation, all data acquisition tasks within the start time and the end time of the data to be acquired.

[0067] A general format of a uniform resource locator for data from the data source 2 is as follows: https: / / gml.noaa.gov / aftp / data / radiation / solrad / abq / 2023 / abq23001.dat. In the source, abq represents the site name, and 2023 represents the year. In abq23001.dat, abq represents the site name, 23 represents the year 2023, 001 represents the first day of year 23, and dat represents a file format of the data to be acquired.

[0068] Perform the data acquisition task and data exporting, run the data acquisition task, after data exporting is completed, convert an original zip format or dat format of the data to be acquired, and uniformly export the data to be acquired in a csv format.

[0069] A third case of the general open source data source website provided in this embodiment: As shown in FIG. 5, steps for acquiring data from a general open source data source website 3 (https: / / sauran.ac.za / , name: SAURAN) include:

[0070] Create a new task, select a data source 3, select a site under the data source 3, and determine a start time and an end time of data to be acquired in a form of month day, year.

[0071] Obtain a general format of a uniform resource locator for a data source based on a name of the data source 3, standardize a data generation time, set a time interval to per minute, associate the general format of the uniform resource locator, the site number, the time interval, and the data generation time to form a specific uniform resource locator for the data to be acquired, and generate, through circulation, all data acquisition tasks within the start time and the end time of the data to be acquired.

[0072] A general format of a uniform resource locator for data from the data source 3 is as follows: https: / / sauran.ac.za / api / DataDownload / KZH / Day / 2021-5-27 / 2021-7-28. KZH represents the site name, 2021-5-27 represents the start time of the data to be acquired, and 2021-7-28 represents the end time of the data to be acquired.

[0073] Perform the data acquisition task and data exporting, run the data acquisition task, after data exporting is completed, convert an original zip format or dat format of the data to be acquired, and uniformly export the data to be acquired in a csv format.

[0074] A fourth case of the general open source data source website provided in this embodiment: As shown in FIG. 6, steps for acquiring data from a general open source data source website 4 (http: / / solardat.uoregon.edu / SelectArchivalUpdatedFormat.html, name: Uoregon) include:

[0075] Create a new task, select a data source 4, select a site under the data source 4, and determine a start time and an end time of data to be acquired in a form of month day, year.

[0076] Obtain a general format of a uniform resource locator for a data source based on a name of the data source 4, standardize a data generation time. Because the data of the site has a long history and the data format has changed, different strategies need to be developed for the previous data and current data. Associate the general format of the uniform resource locator, the site name, the data generation time, and new strategies to form a specific uniform resource locator for the data, and generate, through circulation, all data acquisition tasks within the start time and the end time.

[0077] A general format of a uniform resource locator for data from the data source 4 is as follows: http: / / solardat.uoregon.edu / download / ArchiveUpdatedFormat / SIO_Updated / SIO_2002_Comprehen siveFormat / SIO_05_2002-12.csv. SIO represents the site name, 2002 represents the year, and 12 represents the month. 05 represents a sampling interval, indicating sampling per five minutes. Sampling intervals of data at different times may be different. A sampling interval for some data is 01, indicating sampling per one minute. Because the data source has a long history, a general format of a uniform resource locator for some earlier data is as follows: http: / / solardat.uoregon.edu / download / ArchiveUpdatedFormat / DIM_Updated / Step4_DIM_2002 / DI M_05_2002-10.csv. In addition, because an interval of time at which some data is sampled is large, solar prediction requirements are not met, and data acquisition is not performed.

[0078] Perform the data acquisition task and data exporting, run the data acquisition task, after data exporting is completed, convert an original zip format or dat format of the data to be acquired, and uniformly export the data to be acquired in a csv format.

[0079] Step 22: Customize a rule of acquiring data from a special open source data source website, entering an account and a password, logging in to an ftp server of the special open source data source website, and performing a data acquisition task through a server interface.

[0080] A first case of the special open source data source website provided in this embodiment: As shown in FIG. 7, steps for acquiring data from a special open source data source website (https: / / bsrn.awi.de / project / background, name: BSRN) include:

[0081] Create a new task, select a data source 5 corresponding to a special open source data source website, enter an account and a password of the special open source data source website, and log in to an ftp server of the special open source data source website.

[0082] Obtain a general format of a data source path prefix based on a name of the data source Y (data source 5), standardize a data generation time, associate the general format of the data source path prefix, the site name, and the data generation time to form a specific data acquisition, and generate, through calculation, all data acquisition tasks within the start time and the end time.

[0083] A general format of a data address of the data source 5 is as follows: abs / 2019-01 / 2019-10, where abs represents the site name, 2019-01 represents the start time of the data to be acquired, and 2019-10 represents the end time of the data to be acquired.

[0084] Perform the data acquisition task and data exporting, run the data acquisition task, after data exporting is completed, convert an original dat.gz format of the data, and uniformly export the data in a csv format.

[0085] Step 23: Perform data cleaning and data warehousing, screen out poor quality data through a data quality control method, fill in missing data to implement data cleaning, and regularly import data cleaned into a mysql database.

[0086] In step 23, the performing data cleaning and data storing includes:

[0087] reading a total solar irradiance G, a direct solar irradiance B, an acquisition time T, a longitude (Lon) of a location, and a latitude (Lat) of the location in the csv file, and converting an acquisition time T into a coordinated universal time;

[0088] setting physical upper limits and physical lower limits of G and B, calculating an extraterrestrial solar irradiance Eext, and determining the extraterrestrial solar irradianceEext as the physical upper limit, where a calculation formula of the extraterrestrial solar irradiance Eext, where a calculation formula is as follows:

[0089] Eext = 1361.1 X [1.00011 + 0.034221 cos 360(d l) + 0.00128 sin 360(d 1} + L                                 365                        365 0.000719 cos 720^"^ + 0.000077 sin 720^"1^ 365                          365 J

[0090] d represents an acquisition time T, indicating a dth day of the year, and 0 represents the physical lower limit;

[0091] processing a missing value, where ten or more consecutive missing values are not processed, and nine or fewer missing values are completed through a Newton interpolation method;

[0092] filtering G and B recorded during a period of time at which a solar zenith angle is 9 > 85°, and marking G and B as missing values, where 9 is obtained through calculation based on T, the longitude, and the latitude; and

[0093] directly importing the data cleaned into the mysql database for subsequently performing data exporting.

[0094] An embodiment provides a system for dynamically acquiring open source solar irradiance data. The system for dynamically acquiring open source solar irradiance data includes a back-end main service program, an acquisition rule configuration front-end, and a data storage cluster.

[0095] The acquisition rule configuration front-end is used to configure a data acquisition rule, and send the data acquisition rule to the back-end main service program.

[0096] The back-end main service program is used to acquire data based on the data acquisition rule sent by the acquisition rule configuration front-end, clean the data, and finally import the data into the data storage cluster.

[0097] It may be understood that, the system is deployed on the cloud server. In addition, applications are integrated in a solar irradiance data application system, deployed in a solar irradiance data application system on the cloud server, and connected to the system for dynamically acquiring open source solar irradiance data. The system for dynamically acquiring open source solar irradiance data regularly updates the open source solar irradiance data and historical solar irradiance data. The solar irradiance data application system carries out an application based on exported data.

[0098] The present disclosure further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the foregoing method.

[0099] The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disc, a DVD, a CD-ROM, a micro driver, a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash storage device, a magnetic card, an optical card, a nano-systems (including molecular memory integrated circuit (IC)), or any type of medium or device for storing instructions and / or data.

[0100] In several embodiments provided in the present disclosure, it should be understood that the disclosed system may be implemented in other manners. For example, the described system embodiments are merely examples. For example, division into the units is merely logical function division and may be another division in actual implementation. For example, a plurality of units or components may be combined or integrated into another system, or some features may be ignored or not performed. In other respects, the intercoupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some service interfaces, apparatuses, or units, or may be implemented in an electrical or other forms.

[0101] The units described as separate parts may be or may not be physically separate, and parts displayed as units may be or may not be physical units, may be located in one position, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of embodiments.

[0102] In addition, functional units in the embodiments of this application may be integrated into one processing unit, or each of the units may exist alone physically, or two or more units are integrated into one unit. The above integrated unit may be implemented either in a form of hardware or in a form of a software functional unit.

[0103] The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable memory. Based on such an understanding, the technical solutions in this application essentially, or the part contributing to the prior art, or all or some of the technical solutions may be implemented in a form of a software product. The computer software product is stored in a memory, and includes instructions for instructing a computer device (which may be a personal computer, a server, or a network device) to perform all or some of the operations of the methods described in the embodiments of this application. The memory includes any medium capable of storing program code, such as a USB flash disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, an optical disc, or other media capable of storing program code.

[0104] Those of ordinary skill in the art can understand that all or some of the steps in the method of the above embodiment may be implemented by instructing related hardware by using a program. The program may be stored in a computer-readable memory. The memory may be a flash memory, a read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0105] The implementations of the present disclosure are described in detail above in combination with the accompanying drawings, but the present disclosure is not limited thereto. For those skilled in the art, various changes, modifications, replacements, and variations made to the implementations including components without departing from the principle and spirit of the present disclosure still fall within the protection scope of the present disclosure.

Claims

1. A method for dynamically acquiring open source solar irradiance data, comprising:SI: screening open source data, and analyzing an acquisition condition for the open source data, whereinthe screening open source data comprises: screening a plurality pieces of global solar irradiance data, selecting a data source that is capable of being used to predict solar irradiance and assess a solar energy resource, obtaining meta-information of the data source that is capable of being used to predict solar irradiance and assess a solar energy resource, and determining, based on the meta-information, data with a long running time, high reliability, and a high temporal-spatial resolution as a dynamic acquisition object for the open source solar irradiance data; andthe analyzing the acquisition condition for the open source data comprises: sequentially analyzing a law of obtaining an address of data from an open source solar irradiance data source website, and performing statistical analysis in terms of a form of data to be acquired, whether to enter an account and a password, whether to perform association in an ftp manner, and whether to change a data format; and dividing, based on the statistical analysis, an acquisition condition for the open source solar irradiance data source website into a general condition and a special condition, and formulating a rule of acquiring source data of the open source solar irradiance data; andS2: acquiring the open source data, wherein the acquiring open source data specifically comprises:S2-1: formulating a rule of acquiring data from a general open source data source website, and performing data acquisition;S2-2: formulating a rule of acquiring data from a special open source data source website, and performing data acquisition; andS2-3: performing data cleaning and data warehousing.

2. The method for dynamically acquiring open source solar irradiance data according to claim 1, wherein the meta-information comprises a type, an acquisition device, an acquisition frequency, an acquisition time, and a maintenance frequency.

3. The method for dynamically acquiring open source solar irradiance data according to claim 1, wherein the rule of acquiring data from a general open source data source website is as follows: generating, based on a uniform resource locator for a data address, a data generation time, a time interval, and a site, a uniform resource locator that is specified by a user for the site, the data generation time, and the time interval, positioning a data location based on the uniform resourcelocator, and performing a data acquisition task.

4. The method for dynamically acquiring open source solar irradiance data according to claim 1, wherein the rule of acquiring data from a special open source data source website is as follows: entering the account and the password, logging in to an ftp server of the special open source data source website, and performing a data acquisition task through a server interface.

5. The method for dynamically acquiring open source solar irradiance data according to claim 3, wherein in step S2-1, the formulating a rule of acquiring data from a general open source data source website, and performing data acquisition comprises:S2-1-1: creating a new task, selecting a general open source data source X, selecting a site under the data source X, and determining a start time and an end time of data to be acquired in a form of month day, year;S2-1-2: obtaining a general format of a uniform resource locator for a data source based on a name of the data source X, mapping the site into a site number, standardizing the data generation time, associating the general format of the uniform resource locator, the site number, and the data generation time to form a specific uniform resource locator for the data to be acquired, and generating, through circulation, all data acquisition tasks within the start time and the end time; andS2-1-3: performing the data acquisition task and data exporting, specifically comprising: running the data acquisition task, after data exporting is completed, converting an original zip format or dat format of the data to be acquired, and uniformly exporting the data to be acquired in a csv format.

6. The method for dynamically acquiring open source solar irradiance data according to claim 4, wherein in step S2-2, the formulating a rule of acquiring data from a special open source data source website, and performing data acquisition comprises:S2-2-1: creating a new task, selecting a special open source data source Y, entering the account and the password of the special open source data source website, and logging in to an ftp server of the special open source data source website;S2-2-2: obtaining a general format of a data source path prefix based on a name of the data source Y, standardizing a data generation time, associating the general format of the data source path prefix, a site name, and a data generation time to form a specific data acquisition, and generating, through calculation, all data acquisition tasks within the start time and the end time; andS2-2-3: performing the data acquisition task and data exporting, specifically comprising: running the data acquisition task, after data exporting is completed, converting an original dat.gzformat of the data to be acquired, and uniformly exporting the data to be acquired in a csv format.

7. The method for dynamically acquiring open source solar irradiance data according to claim 5 or 6, wherein in step S2-3, the performing data cleaning and data storing comprises:S2-3-1: reading a total solar irradiance G, a direct solar irradiance B, an acquisition time T, a longitude (Lon) of a location, and a latitude (Lat) of the location in the csv file, and converting an acquisition time T into a coordinated universal time;S2-3-2: setting physical upper limits and physical lower limits of G and B, calculating an extraterrestrial solar irradiance, and determining the extraterrestrial solar irradiance as the physical upper limit;S2-3-3: processing a missing value;S2-3-4: filtering G and B recorded during a period of time at which a solar zenith angle is 9 > 85°, and marking G and B as missing values; andS2-3-5: directly importing data cleaned into a mysql database.

8. A system for dynamically acquiring open source solar irradiance data, comprising a back-end main service program, an acquisition rule configuration front-end, and a data storage cluster, whereinthe acquisition rule configuration front-end is used to configure a data acquisition rule, and send the data acquisition rule to the back-end main service program; andthe back-end main service program is used to acquire data based on the data acquisition rule sent by the acquisition rule configuration front-end, clean the data, and finally import the data into the data storage cluster.

9. The system for dynamically acquiring open source solar irradiance data according to claim 8, wherein the back-end main service program, the acquisition rule configuration front-end, and the data storage cluster are deployed on a cloud server.

10. A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the method according to any one of claims 1 to 7.

Images