GEOSS application analysis method and platform based on big data

Abstract

The invention provides a GEOSS application analysis method and platform based on big data, and relates to the technical field of data analysis, K multi-source global observation data sets are obtained, data standardization processing is carried out according to an extraction result, multi-source data fusion is carried out according to K data reliable factors, and a preset first-level index set is used as an index; retrieving the fused global observation data set to obtain a fused evaluation result and a preset first-level index weight set, performing evaluation model training in combination with a plurality of first-level index observation data sets to obtain a trained evaluation model, and evaluating a plurality of real-time first-level index observation data sets by using the evaluation model to obtain a plurality of real-time second-level index observation data sets. The technical problems of low accuracy and low efficiency of data fusion evaluation caused by various data sources of the global earth observation system in the prior art are solved. Efficient fusion and accurate evaluation of multi-source data are realized, and the technical effect of improving the accuracy of data evaluation of the global earth observation system is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to a GEOSS application analysis method and platform based on big data. Background Art

[0002] As a global observation network, the Global Earth Observation System of Systems (GEOSS) covers multiple data sources, observation equipment and regions, and can provide important information on climate change, environmental monitoring and disaster warning. However, due to the diversity of observation equipment and observation data, there are differences in data format, type and data quality, which leads to great complexity in data fusion. Traditional data integration methods often have difficulty in processing large-scale heterogeneous data, resulting in low accuracy and reliability of data evaluation.

[0003] The existing technology has the technical problem that the data sources of the global earth observation system are diverse, resulting in low accuracy and low efficiency of data fusion evaluation. Summary of the invention

[0004] This application provides a GEOSS application analysis method and platform based on big data, which is used to solve the technical problem in the prior art that the data sources of the global earth observation system are diverse, resulting in low accuracy and low efficiency of data fusion evaluation.

[0005] In view of the above problems, this application provides a GEOSS application analysis method and platform based on big data.

[0006] In a first aspect of the present application, a GEOSS application analysis method based on big data is provided, the method comprising: performing multi-source global observation data collection on K databases to obtain K multi-source global observation data sets, wherein K is an integer greater than or equal to 1; traversing the K multi-source global observation data sets to extract data format, data type and data missingness, and performing data standardization processing according to the extraction results to obtain a standardized multi-source global observation data set; performing multi-source data fusion on the standardized multi-source global observation data set according to the K data reliability factors of the K databases to obtain a fused global observation data set; using a preset first-level indicator set as an index, respectively retrieving the fused global observation data set to obtain multiple first-level indicator observation data sets, wherein each first-level indicator observation data set corresponds to a first-level indicator; obtaining a fusion evaluation result and a preset first-level indicator weight set corresponding to the fused global observation data set, and training an evaluation model in combination with the multiple first-level indicator observation data sets to obtain a trained evaluation model; collecting multiple real-time first-level indicator observation data sets under the current window, and using the evaluation model to evaluate the multiple real-time first-level indicator observation data sets to obtain real-time evaluation results.

[0007] The second aspect of the present application provides a GEOSS application analysis platform based on big data, the platform comprising: a data acquisition module, the data acquisition module is used to perform multi-source global observation data acquisition on K databases to obtain K multi-source global observation data sets, wherein K is an integer greater than or equal to 1; a data processing module, the data processing module is used to traverse the K multi-source global observation data sets to extract data format, data type and data missing degree, and perform data standardization processing according to the extraction results to obtain a standardized multi-source global observation data set; a data fusion module, the data fusion module is used to perform multi-source data fusion on the standardized multi-source global observation data set according to K data reliability factors of the K databases to obtain a fused global observation data set; A data retrieval module, the data retrieval module is used to retrieve the fused global observation data set respectively with a preset first-level indicator set as an index, and obtain multiple first-level indicator observation data sets, wherein each first-level indicator observation data set corresponds to a first-level indicator; an evaluation model acquisition module, the evaluation model acquisition module is used to obtain the fusion evaluation result and the preset first-level indicator weight set corresponding to the fused global observation data set, and train the evaluation model in combination with the multiple first-level indicator observation data sets to obtain a trained evaluation model; an evaluation result acquisition module, the evaluation result acquisition module is used to collect multiple real-time first-level indicator observation data sets under the current window, and use the evaluation model to evaluate the multiple real-time first-level indicator observation data sets to obtain real-time evaluation results.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] The method provided in the embodiment of the present application performs multi-source global observation data collection on K databases to obtain K multi-source global observation data sets, wherein K is an integer greater than or equal to 1; traverses the K multi-source global observation data sets to extract data format, data type and data missing degree, and performs data standardization processing according to the extraction results to obtain a standardized multi-source global observation data set; performs multi-source data fusion on the standardized multi-source global observation data set according to the K data reliability factors of the K databases to obtain a fused global observation data set; retrieves the fused global observation data set respectively with a preset first-level indicator set as an index to obtain multiple first-level indicator observation data sets, wherein each first-level indicator observation data set corresponds to a first-level indicator; obtains the fusion evaluation result and the preset first-level indicator weight set corresponding to the fused global observation data set, and performs evaluation model training in combination with the multiple first-level indicator observation data sets to obtain a trained evaluation model; collects multiple real-time first-level indicator observation data sets under the current window, and uses the evaluation model to evaluate the multiple real-time first-level indicator observation data sets to obtain real-time evaluation results. It has achieved efficient fusion and accurate evaluation of multi-source data, achieving the technical effect of improving the accuracy of data evaluation of the global earth observation system. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 Schematic diagram of the GEOSS application analysis method based on big data provided for this application;

[0012] Figure 2 Schematic diagram of the structure of the big data-based GEOSS application analysis platform provided for this application.

[0013] Explanation of the reference numerals: data acquisition module 11 , data processing module 12 , data fusion module 13 , data retrieval module 14 , evaluation model acquisition module 15 , evaluation result acquisition module 16 . DETAILED DESCRIPTION

[0014] This application provides a GEOSS application analysis method and platform based on big data, which is used to solve the technical problem in the prior art that the data sources of the global earth observation system are diverse, resulting in low accuracy and low efficiency of data fusion evaluation. It achieves efficient fusion and accurate evaluation of multi-source data, and improves the accuracy and efficiency of global earth observation system data evaluation.

[0015] Below, the technical solutions in the present invention will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments of the present invention. It should be understood that the present invention is not limited to the example embodiments described herein. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all of them.

[0016] Embodiment 1, as Figure 1 As shown, the present application provides a GEOSS application analysis method based on big data, the method comprising:

[0017] Multi-source global observation data collection is performed on K databases to obtain K multi-source global observation data sets, where K is an integer greater than or equal to 1.

[0018] Specifically, the Global Earth Observation System of Systems (GEOSS) is a global, integrated system that integrates multiple observation systems. By integrating observation data from different sources, it provides powerful data support for scientific research, environmental protection, disaster warning, etc. To implement GEOSS data analysis, we first need to determine the number of databases K to be accessed. K is an integer greater than or equal to 1, representing global observation databases from different sources and different themes. According to the research purpose and data requirements, global observation information is integrated from multiple data sources to form K multi-source global observation data sets, each of which contains observation data from a specific database.

[0019] In one embodiment, multi-source global observation data collection is performed on K databases to obtain K multi-source global observation data sets, including: step one: select K URLs from a preset starting URL list to form a download queue, wherein the URL is an entry to the database; step two: randomly extract a first URL from the download queue to download and parse a web page to obtain a plurality of parsed URLs; step three: filter the plurality of parsed URLs using a web page analysis algorithm to obtain a plurality of newly added URLs, and add the plurality of newly added URLs to the download queue; step four: repeat step two and step three until the download queue is empty or the K databases are all collected, stop data collection, and obtain the K multi-source global observation data sets.

[0020] Specifically, in step 1, K URLs are selected from a preset starting URL list as the initial entry of the download queue, and the selected K URLs are formed into a download queue for subsequent access and parsing. URL refers to the entry address of the database, ensuring that the root location of the global observation data can be accurately accessed. The starting URL is obtained from an authoritative source or a trusted database directory, ensuring the reliability and accuracy of the data source. Then, a URL is randomly selected from the download queue in step 2, called the first URL, and the web page content is downloaded from the Internet using the HTTP or HTTPS protocol. After the download is completed, the web page parsing tool, such as a regular expression, HTML parser, and other web page parsing algorithms are used to parse the web page content, and multiple parsed URLs are extracted from it, that is, more data entry links are obtained, and the web page parsing algorithm is used to ensure that useful observation data or paths pointing to more observation data are extracted from each entry link. After obtaining multiple parsed URLs, the multiple parsed URLs are filtered by the web page analysis algorithm in step 3, and URLs that meet specific rules or patterns are screened out to obtain new entries that may contain global observation data. The web page analysis algorithm can be screened according to set rules, such as URL format, content relevance, etc. The multiple URLs that are screened out are called new URLs. The multiple new URLs are dynamically added to the download queue. Through screening, invalid or duplicate data sources can be avoided, ensuring the continuous expansion of the collection scope and increasing the data source. Step 4, repeat the process of steps 2 and 3, continuously extract URLs from the download queue for web page download and parsing, obtain new parsed URLs, and add new URLs to the download queue through filtering until any of the following conditions are met: first, the download queue is empty, that is, all available URLs have been accessed and parsed; second, the data in all K databases have been completely collected, that is, the collection target has been achieved. Through the loop method, the comprehensiveness of data collection can be ensured to the greatest extent. When the stop condition is met, the data collection process is stopped, thereby forming a complete set of K multi-source global observation data. The K multi-source global observation data sets include observation data sets from different sources, ensuring the diversity and reliability of data sources.

[0021] The K multi-source global observation data sets are traversed to extract data formats, data types and data missingness, and data standardization is performed according to the extraction results to obtain a standardized multi-source global observation data set.

[0022] Furthermore, the K multi-source global observation data sets are traversed to extract data formats, data types and data missing degrees, and data standardization is performed according to the extraction results to obtain standardized multi-source global observation data sets, including: respectively counting the number of missing fields of the K multi-source global observation data sets, corresponding to the total number of fields of the K multi-source global observation data sets, to obtain K data missing degree sets; traversing the K multi-source global observation data sets to extract data formats and data types to obtain K data format sets and K data type sets; using Pandas to perform format standardization on the K multi-source global observation data sets according to the K data format sets and K data type sets to obtain K format-standardized multi-source global observation data sets; processing the K format-standardized multi-source global observation data sets according to the data missing degree of the K data missing degree sets to obtain the K multi-source global observation data sets.

[0023] Specifically, K multi-source global observation data sets may contain data of various formats and types, which need to be processed uniformly. First, the number of missing fields is counted. By traversing the K multi-source global observation data sets one by one, the number of missing fields in each set is counted. The number of missing fields refers to the number of missing fields in the data set, indicating that some data is incomplete or the number of records with empty field values. After counting the number of missing fields in each set, it is divided by the total number of fields in the set to calculate the data missingness of each data set and obtain K data missingness sets. The missingness set reflects the proportion of missing information in the data set and is used to reflect the integrity of each data set. Next, the data format and data type are extracted. Each data set of the K multi-source global observation data sets is traversed again to extract its data format and data type. The data format refers to the representation of the data, such as CSV, Excel, etc., while the data type refers to the type of the field, such as integer, floating point number, string, etc. Two sets are formed according to the extraction results, namely K data format sets and K data type sets, which are used to describe the characteristics of each data set. By extracting the format and type information, it can be ensured that the data from different data sources are formatted uniformly. Then, the Pandas library is used to standardize the format of K data format sets and K data type sets to ensure that each data set is consistent in structure. Pandas is a powerful Python data analysis tool library that provides a wealth of data operations and processing functions for converting data formats and types, so that the K data sets meet unified specifications after standardization, thereby obtaining K format-standardized multi-source global observation data sets, which is convenient for subsequent data analysis and processing. Finally, according to the size of the K data missing degree sets calculated previously, the K format-standardized multi-source global observation data sets after format standardization are screened or filled to obtain K multi-source global observation data sets that have been processed by data standardization. Through multiple steps such as calculating the data missing degree, extracting the data format and data type, and using Pandas for format standardization, data sets from different sources and with different formats and types are unified into a standard format, ensuring the integrity and consistency of the data and improving the accuracy of subsequent data analysis.

[0024] Furthermore, the method also includes: determining whether the data missing degree of the K data missing degree sets is greater than or equal to a preset data missing degree, and if so, eliminating the corresponding data; if not, filling the data according to the mean filling method to obtain the K multi-source global observation data sets.

[0025] Specifically, in the process of processing the K format standardized multi-source global observation data sets according to the data missingness of the K data missingness sets, different operations are performed according to the data missingness situation to judge the data missingness of each data set. The data missingness refers to the proportion of missing fields in the data, which is used to measure the integrity of the data. The missingness of each data set is compared with a preset data missingness. The preset data missingness is a threshold used to define the maximum missingness range allowed for the data, which can be set according to actual needs. If the missingness of a data set is greater than or equal to the preset missingness threshold, then the data set is considered to be seriously missing and unsuitable for further processing, and the data set is eliminated to prevent the incomplete data from affecting subsequent analysis. Eliminating the data set helps to maintain the reliability and accuracy of the data analysis results. If the missingness of the data set is less than the preset threshold, the data set is considered to be retained, and the missing fields are completed using the mean filling method. Mean filling is a common processing method that replaces the missing value with the average value of the field to ensure the consistency and integrity of the data. Mean filling not only helps to reduce the impact of missing data, but also maintains the overall trend and distribution characteristics of the data. Through the elimination and filling operations, a higher quality multi-source global observation data set with processed missing values ​​is obtained. After standardization and missing value processing, the integrity and consistency of the K multi-source global observation data sets can be ensured, providing a reliable data foundation for subsequent data analysis and fusion.

[0026] According to the K data reliability factors of the K databases, multi-source data fusion is performed on the standardized multi-source global observation data set to obtain a fused global observation data set.

[0027] Specifically, since the observation data of K databases are limited by the influence of multiple factors such as observation angle, observation equipment accuracy and data processing method, there are inevitably some deviations between these data. Therefore, in order to obtain more accurate and reliable global observation data, the data reliability factor is introduced, and the data is weighted and fused according to the reliability of each database in the K databases. First, the K data reliability factors of the K databases are obtained. The data reliability factor is a key evaluation indicator that reflects the accuracy, completeness and consistency of the data in each database and is used to measure the reliability of the data source. Then, based on the K data reliability factors, the standardized multi-source global observation data set is weighted and fused. Specifically, a weight value corresponding to its reliability factor is assigned to each standardized data set, so that the weight is proportional to the reliability. The K standardized data sets are weighted according to their respective weights, and then averaged to ensure that the contribution of each data source matches its reliability. In the fusion process, the data set with high reliability has a greater impact on the final fusion result, while the data with low reliability has a relatively small impact on the result. Through weighted fusion, the observation data in K databases are effectively utilized, and the reliability of each database is fully considered. Finally, a fused global observation data set is generated. This set integrates the observation data from different databases and can more accurately and comprehensively reflect global observation information. By considering the reliability of each data source, data deviation is minimized and the overall accuracy and reliability of the data are improved.

[0028] Furthermore, the method includes: obtaining K data update frequencies, K data error rates and K equipment failure coefficients of the K databases; constructing a database reliable identification network layer; using the database reliable identification network layer to identify the K data update frequencies, K data error rates and K equipment failure coefficients to determine the K data reliability factors.

[0029] Specifically, the K data update frequencies, K data error rates and K equipment failure coefficients of K databases are obtained. The data update frequency is the update frequency of database data, which is an important indicator for measuring the timeliness of database information. The higher the update frequency, the more the data can reflect the changes of the observed object in real time. Databases with high update frequency are usually more reliable. The data error rate is the proportion of erroneous data, which reflects the accuracy and quality of the data. Data sets with low error rates show higher consistency in observations, and data sources with low error rates are considered more reliable. The equipment failure coefficient is the failure frequency of the observed equipment within a specified time period, which is used to evaluate the stability of the equipment. Equipment with low failure frequency can usually provide more reliable observation data and have higher reliability. In order to comprehensively evaluate the data update frequency, data error rate and equipment failure coefficient, a database reliable identification network layer is constructed. The database reliable identification network layer adopts a neural network structure to optimize the contribution of each indicator by learning weights. The input layer of the network layer receives the update frequency, error rate and failure coefficient of each database. The hidden layer in the network layer analyzes these indicators and learns the reliability characteristics of different databases. The output layer of the network layer generates a data reliability factor for each database, which is used to measure the comprehensive reliability of the database. After training with historical data update frequency, historical data error rate and historical equipment failure coefficient, a trained database reliable identification network layer is obtained. The K data update frequencies, K data error rates and K equipment failure coefficients are used as inputs, and a comprehensive analysis is performed using the database reliable identification network layer to output K data reliability factors. The data reliability factor is a comprehensive indicator. The higher the value, the more reliable the database is, and vice versa. By constructing a database reliable identification network layer and analyzing the reliability factors of each database, the data reliability factor finally determined will effectively support the weighted fusion of data and ensure the accuracy and consistency of the fusion results.

[0030] The fused global observation data set is searched respectively with the preset first-level indicator set as the index to obtain a plurality of first-level indicator observation data sets, wherein each first-level indicator observation data set corresponds to a first-level indicator.

[0031] Furthermore, the preset first-level indicator set includes spatial resolution indicators, product type indicators, sharing mode indicators, application field indicators, capacity building indicators and policy support indicators.

[0032] Specifically, after data fusion, in order to further utilize the information in the global observation data set, the data is retrieved with a preset first-level indicator set to extract the observation data set corresponding to each first-level indicator. The first-level indicator represents the characteristics and application requirements of global observation data in different dimensions, which is helpful for systematic data acquisition. The preset first-level indicator set includes spatial resolution indicators, product type indicators, sharing mode indicators, application field indicators, capacity building indicators and policy support indicators, wherein the spatial resolution indicator represents the geographical fineness of the observation data, which is used to measure the detail of the observation information, such as the resolution in kilometers or meters. The product type indicator is used to distinguish the content type of the observation data, such as meteorological data, terrain data, environmental data, etc. The sharing mode indicator reflects the sharing form of the observation data, such as open data, authorized data, etc., which is used to control the access rights of the data. The application field indicator represents the main application fields of the observation data, such as agriculture, meteorology, disaster monitoring, etc. The capacity building indicator measures the application capabilities of the data in terms of technical support and knowledge dissemination, including training needs, the use of technical tools, etc. The policy support indicator reflects the supporting role of data at the policy level, such as support for policy formulation and evaluation. Then, each indicator in the preset first-level indicator set is used as an index to perform classified retrieval on the fused global observation data set. For example, data is retrieved based on the spatial resolution indicator to obtain observation data sets at different spatial resolutions, and data is classified by product type indicators to obtain data sets of specific types. For example, if the indicator is meteorological data, the system will retrieve and extract all meteorological observation-related data sets. Other indicators are used as indexes in the same way to perform classified retrieval on the fused global observation data set. Through the above step-by-step retrieval, multiple first-level indicator observation data sets are generated, and the retrieval results of each first-level indicator are defined as a first-level indicator observation data set, that is, a subset of observation data extracted on this dimension. Through these retrieval operations, the multiple first-level indicator observation data sets obtained provide efficient and accurate data support for subsequent data applications, and realize the analysis needs of global observation data in multiple fields and at multiple levels.

[0033] The fusion evaluation result and the preset first-level indicator weight set corresponding to the fusion global observation data set are obtained, and the evaluation model training is performed in combination with the multiple first-level indicator observation data sets to obtain a trained evaluation model.

[0034] Specifically, the fusion evaluation results are extracted from the fusion global observation data set. The fusion results are summary information of the development of GEOSS in historical time, representing the actual development process of the global observation system, and are used to guide the evaluation of the model at different time nodes. For example, the fusion results may include the coverage of the observation network, the degree of data sharing, the stability of the system, the quality of the data and other aspects. Then, the experts set the preset first-level indicator weight set based on their years of experience. The preset first-level indicator weight set is used to reflect the relative importance of each first-level indicator in the overall evaluation. For example, if the expert experience believes that the demand for geographic data is more important, then the weight setting for spatial resolution may be higher to indicate the importance of the demand for high-precision geographic data. Each weight value corresponds to a first-level indicator, ensuring that the importance of different indicators is considered during the evaluation. Then, the obtained fusion evaluation results and the preset first-level indicator weight set are combined with multiple first-level indicator observation data sets to form a training data set for the model. The first-level indicator observation data set provides data on different indicators, and the fusion evaluation results provide the model with a target value so that the model can be adjusted during training. The weight set provides a reference for the contribution of each indicator data, so that the model can more accurately allocate the influence of each indicator during the evaluation process. The evaluation model is trained through the model training data set to obtain a trained evaluation model. This evaluation model not only integrates historical development trends and expert knowledge, but also can adapt to new data inputs, accurately evaluate observation data, and timely discover and warn of potential environmental changes or disaster risks, thereby improving data evaluation efficiency and accuracy.

[0035] Furthermore, the method includes: constructing a weight distribution network layer according to the preset first-level indicator weight set; adding the weight distribution network layer between the input layer and the data analysis layer of the evaluation model constructed based on the feedforward neural network; using the multiple first-level indicator observation data sets and the fusion evaluation results to train the evaluation model, and updating and adjusting the weights in the weight distribution network layer according to the output results until the training converges, thereby obtaining the trained evaluation model.

[0036] Specifically, based on the preset set of first-level indicator weights, a weight allocation network layer is constructed. The role of the network layer is to assign initial weights to each first-level indicator so that indicators with high importance have a greater impact on the model output. The weight set is set by expert experience and reflects the relative priority of each first-level indicator in the overall evaluation. For example, for indicators with high spatial resolution importance, the weight value will be relatively high, so that indicators with high importance are more influential in the evaluation. Then, the weight allocation network layer is embedded between the input layer and the data analysis layer of the feedforward neural network model. The feedforward neural network is a one-way propagation network structure that can perform layer-by-layer feature extraction and analysis of input data. The introduction of the weight allocation network layer enables each first-level indicator observation data to be weighted accordingly before entering the model data analysis layer, thereby directly affecting the indicator priority set by experts in the model calculation process. Then, the evaluation model is trained using the training data, which includes multiple first-level indicator observation data sets and fusion evaluation results. During the training process, the evaluation model will continuously dynamically adjust the weight values ​​of the weight distribution network layer through error feedback, adjust the error between the output results based on the model and the fusion evaluation results, and update the weights in the weight distribution network layer through the back propagation algorithm, so that the model output is closer to the fusion evaluation results. After each update, the model will recalculate the contribution of each first-level indicator and gradually optimize the weight values ​​in the weight distribution network layer until the training reaches convergence, that is, the output error of the model meets the preset accuracy requirements, the model training is completed, and the final evaluation model is obtained. The model can accurately predict the development of GEOSS based on new observation data. The development of GEOSS (Global Earth Observation System) reflects the progress and status of the system in supporting global sustainable development, responding to environmental challenges and improving data sharing capabilities.

[0037] Collect multiple real-time first-level indicator observation data sets under the current window, use the evaluation model to evaluate the multiple real-time first-level indicator observation data sets, and obtain real-time evaluation results.

[0038] Specifically, multiple real-time first-level indicator observation data sets are extracted from the observation data within the current window period, and the current window refers to a preset time period for limiting the collection scope of real-time data, such as 7 days, one month, three months, etc. The real-time data set contains the latest observation data of each first-level indicator, including spatial resolution indicators, product type indicators, sharing mode indicators, application field indicators, capacity building indicators, and policy support indicators. After collecting and obtaining multiple real-time first-level indicator observation data sets, they are passed as input data to the evaluation model, which analyzes the data characteristics of each first-level indicator, combines the weights of each indicator, and performs weighted processing through the internal weight distribution network layer. The feedforward neural network structure processes and extracts data features layer by layer, and evaluates multiple real-time first-level indicator observation data sets according to the set weights and the mode in the training process to generate real-time evaluation results. The real-time evaluation results reflect the performance and development status of the global observation system in the current time period, such as global environmental change trends, changes in observation coverage, and data sharing and access conditions, etc., which can provide users and decision makers with instant information on the current status of the system, help users and decision makers to understand possible changes or anomalies in the world in a timely manner, and provide accurate, reliable and comprehensive data support.

[0039] Embodiment 2 is based on the same inventive concept as the GEOSS application analysis method based on big data in the previous embodiment. Figure 2 As shown, the present application provides a GEOSS application analysis platform based on big data, wherein the platform includes:

[0040] A data collection module 11, the data collection module 11 is used to perform multi-source global observation data collection on K databases to obtain K multi-source global observation data sets, wherein K is an integer greater than or equal to 1; a data processing module 12, the data processing module 12 is used to traverse the K multi-source global observation data sets to extract data format, data type and data missing degree, and perform data standardization processing according to the extraction results to obtain a standardized multi-source global observation data set; a data fusion module 13, the data fusion module 13 is used to perform multi-source data fusion on the standardized multi-source global observation data set according to the K data reliability factors of the K databases to obtain a fused global observation data set; a data retrieval module 14, the data retrieval module 14 It is used to retrieve the fused global observation data set respectively with the preset first-level indicator set as the index, and obtain multiple first-level indicator observation data sets, wherein each first-level indicator observation data set corresponds to one first-level indicator; an evaluation model acquisition module 15, the evaluation model acquisition module 15 is used to obtain the fusion evaluation result and the preset first-level indicator weight set corresponding to the fused global observation data set, and train the evaluation model in combination with the multiple first-level indicator observation data sets to obtain a trained evaluation model; an evaluation result acquisition module 16, the evaluation result acquisition module 16 is used to collect multiple real-time first-level indicator observation data sets under the current window, and use the evaluation model to evaluate the multiple real-time first-level indicator observation data sets to obtain real-time evaluation results.

[0041] Furthermore, the data acquisition module 11 is used to perform the following steps: Step 1: Select K URLs from a preset starting URL list to form a download queue, wherein the URL is an entry to a database; Step 2: Randomly extract a first URL from the download queue to download and parse the web page, and obtain multiple parsed URLs; Step 3: Filter the multiple parsed URLs using a web page analysis algorithm to obtain multiple newly added URLs, and add the multiple newly added URLs to the download queue; Step 4: Repeat steps 2 and 3 until the download queue is empty or the K databases are all collected, stop data collection, and obtain the K multi-source global observation data sets.

[0042] Further, the data processing module 12 is used to perform the following steps: respectively counting the number of missing fields of the K multi-source global observation data sets, corresponding to the total number of fields of the K multi-source global observation data sets, to obtain K data missing degree sets;

[0043] The K multi-source global observation data sets are traversed to extract data formats and data types, and K data format sets and K data type sets are obtained; Pandas is used to perform format standardization processing on the K multi-source global observation data sets according to the K data format sets and the K data type sets, and K format-standardized multi-source global observation data sets are obtained; the K format-standardized multi-source global observation data sets are processed according to the data missing degree sizes of the K data missing degree sets, and the K multi-source global observation data sets are obtained.

[0044] Furthermore, the data processing module 12 is also used to perform the following steps: determine whether the data missing degree of the K data missing degree sets is greater than or equal to the preset data missing degree, and if so, eliminate the corresponding data; if not, fill the data according to the mean filling method to obtain the K multi-source global observation data sets.

[0045] Furthermore, the data fusion module 13 is used to perform the following steps: obtain K data update frequencies, K data error rates and K equipment failure coefficients of the K databases; construct a database reliable identification network layer; use the database reliable identification network layer to identify the K data update frequencies, K data error rates and K equipment failure coefficients to determine the K data reliability factors.

[0046] Furthermore, the data retrieval module 14 includes the following execution steps: the preset first-level indicator set includes spatial resolution indicators, product type indicators, sharing mode indicators, application field indicators, capacity building indicators and policy support indicators.

[0047] Furthermore, the evaluation model acquisition module 15 includes the following execution steps: constructing a weight allocation network layer according to the preset first-level indicator weight set; adding the weight allocation network layer between the input layer and the data analysis layer of the evaluation model constructed based on the feedforward neural network; using the multiple first-level indicator observation data sets and the fusion evaluation results to train the evaluation model, and updating and adjusting the weights in the weight allocation network layer according to the output results until the training converges, thereby obtaining the trained evaluation model.

[0048] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0049] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A GEOSS application analysis method based on big data, characterized in that: The method comprises: Perform multi-source global observation data collection on K databases to obtain K multi-source global observation data sets, where K is an integer greater than or equal to 1; Traversing the K multi-source global observation data sets to extract data format, data type and data missing degree, and performing data standardization processing according to the extraction results to obtain a standardized multi-source global observation data set; According to the K data reliability factors of the K databases, multi-source data fusion is performed on the standardized multi-source global observation data set to obtain a fused global observation data set; Using the preset first-level indicator set as an index, the fused global observation data set is searched respectively to obtain multiple first-level indicator observation data sets, wherein each first-level indicator observation data set corresponds to a first-level indicator; Obtaining the fusion evaluation result and the preset first-level indicator weight set corresponding to the fusion global observation data set, and training the evaluation model in combination with the multiple first-level indicator observation data sets to obtain a trained evaluation model; Collect multiple real-time first-level indicator observation data sets under the current window, use the evaluation model to evaluate the multiple real-time first-level indicator observation data sets, and obtain real-time evaluation results.

2. The GEOSS application analysis method based on big data as claimed in claim 1, characterized in that: Perform multi-source global observation data collection on K databases to obtain K multi-source global observation data sets, including: Step 1: Select K URLs from the preset starting URL list to form a download queue, where the URL is the entry to the database; Step 2: randomly extracting a first URL from the download queue to download and parse the webpage, and obtaining a plurality of parsed URLs; Step 3: Filter the multiple parsed URLs using a web page analysis algorithm to obtain multiple newly added URLs, and add the multiple newly added URLs into the download queue; Step 4: Repeat steps 2 and 3 until the download queue is empty or the K databases are all collected, stop data collection, and obtain the K multi-source global observation data sets.

3. The GEOSS application analysis method based on big data as claimed in claim 1, characterized in that: Traversing the K multi-source global observation data sets to extract data format, data type and data missingness, and performing data standardization processing according to the extraction results to obtain a standardized multi-source global observation data set, including: Counting the number of missing fields of the K multi-source global observation data sets respectively, and correspondingly comparing the total number of fields of the K multi-source global observation data sets, to obtain K data missing degree sets; Traversing the K multi-source global observation data sets to extract data formats and data types, and obtaining K data format sets and K data type sets; Using Pandas to perform format standardization processing on the K multi-source global observation data sets according to the K data format sets and the K data type sets, so as to obtain K format-standardized multi-source global observation data sets; The K format-standardized multi-source global observation data sets are processed according to the data missingness degrees of the K data missingness degree sets to obtain the K multi-source global observation data sets.

4. The GEOSS application analysis method based on big data as claimed in claim 3, characterized in that: include: Determine whether the data missingness of the K data missingness sets is greater than or equal to a preset data missingness, and if so, remove the corresponding data; If not, the data is filled according to the mean filling method to obtain the K multi-source global observation data sets.

5. The GEOSS application analysis method based on big data according to claim 1, characterized in that: include: Obtaining K data update frequencies, K data error rates and K equipment failure coefficients of the K databases; Build a database to reliably identify the network layer; The database is used to reliably identify the network layer to identify the K data update frequencies, the K data error rates and the K equipment failure coefficients, and determine the K data reliability factors.

6. The GEOSS application analysis method based on big data according to claim 1, characterized in that: The preset first-level indicator set includes spatial resolution indicators, product type indicators, sharing mode indicators, application field indicators, capacity building indicators and policy support indicators.

7. The GEOSS application analysis method based on big data according to claim 1, characterized in that: include: Constructing a weight distribution network layer according to the preset first-level indicator weight set; Adding the weight distribution network layer between the input layer and the data analysis layer of the evaluation model constructed based on the feedforward neural network; The evaluation model is trained using the multiple primary indicator observation data sets and the fusion evaluation results, and the weights in the weight allocation network layer are updated and adjusted according to the output results until the training converges to obtain the trained evaluation model.

8. The GEOSS application analysis platform based on big data is characterized by: The steps for implementing the method according to any one of claims 1 to 7 include: A data collection module, wherein the data collection module is used to perform multi-source global observation data collection on K databases to obtain K multi-source global observation data sets, where K is an integer greater than or equal to 1; A data processing module, the data processing module is used to traverse the K multi-source global observation data sets to extract data format, data type and data missing degree, and perform data standardization processing according to the extraction results to obtain a standardized multi-source global observation data set; A data fusion module, wherein the data fusion module is used to perform multi-source data fusion on the standardized multi-source global observation data set according to the K data reliability factors of the K databases to obtain a fused global observation data set; A data retrieval module, wherein the data retrieval module is used to retrieve the fused global observation data set respectively with a preset first-level indicator set as an index to obtain a plurality of first-level indicator observation data sets, wherein each first-level indicator observation data set corresponds to a first-level indicator; An evaluation model acquisition module, the evaluation model acquisition module is used to obtain the fusion evaluation result and the preset first-level indicator weight set corresponding to the fusion global observation data set, and perform evaluation model training in combination with the multiple first-level indicator observation data sets to obtain a trained evaluation model; The evaluation result acquisition module is used to collect multiple real-time first-level indicator observation data sets under the current window, evaluate the multiple real-time first-level indicator observation data sets using the evaluation model, and obtain real-time evaluation results.

Images