A ground meteorological observation data profiling method and system based on metadata

Abstract

The present invention relates to the technical field of meteorological observation data management, and proposes a ground meteorological observation data profiling method and system based on metadata. The method includes: managing metadata throughout the entire life cycle of ground meteorological observation data; constructing the full-process blood relationship of ground meteorological observation data based on the metadata; and constructing a full-link data profiling of ground meteorological observation data using a knowledge graph based on the metadata. The present invention can improve the management efficiency of ground meteorological data throughout its entire life cycle. Through centralized management and unified analysis of metadata, a large amount of ground meteorological observation data can be better organized and managed, thereby improving the operating efficiency of the system and saving computing resources.

Description

Technical Field

[0001] The present invention generally relates to the field of meteorological observation data management technology. Specifically, the present invention relates to a ground meteorological observation data profiling method and system based on metadata. Background Art

[0002] The metadata of meteorological observation data is data that describes information such as meteorological observation elements, observation conditions, observation methods and data processing methods. It is expressed as metadata details of observation tasks in the information platform, including basic information, equipment information, business operation information, observation elements, historical evolution, basic environmental information, environmental picture information, personnel information, etc.

[0003] At present, meteorological departments and related institutions are paying more and more attention to the metadata management of ground meteorological observation data. By establishing norms and standards, and uniformly collecting, organizing and archiving metadata, the comprehensibility and usability of data can be improved. However, there are deviations and incompleteness in the metadata collection, recording and blood relationship construction of the entire life cycle of ground meteorological observation data. A complete full-link relationship portrait has not yet been formed from multiple links such as ground meteorological data observation, quality control, processing, service, and archiving. It is difficult to fully explore the potential information and relationships in meteorological metadata and realize the full-domain and full-process management of ground meteorological observation data and data relationship traceability. Summary of the invention

[0004] In order to at least partially solve the above problems in the prior art, the present invention proposes a ground meteorological observation data profiling method based on metadata, comprising the following steps:

[0005] Manage metadata of ground-based meteorological observation data throughout its life cycle;

[0006] Constructing the whole process blood relationship of ground meteorological observation data based on the metadata; and

[0007] Based on the metadata, a knowledge graph is used to construct a full-link data portrait of ground meteorological observation data.

[0008] Here, the term "full-process lineage" refers to the relationship between each link in the entire life cycle of data from generation to final use. This relationship is not just a simple parent-child relationship, but also includes detailed information such as each step of data processing, tools used, personnel, time, etc. By establishing this relationship, the source of data can be traced and the data processing process can be understood, thereby ensuring the quality and reliability of data.

[0009] In one embodiment of the present invention, it is provided that managing metadata of ground meteorological observation data throughout its life cycle includes:

[0010] Collect metadata of ground-based meteorological observation data throughout its life cycle, including docking with the multi-source business systems of the meteorological department, extracting, importing, cleaning and / or classifying metadata throughout its life cycle through data collection, storage management, processing, quality control, service utilization and / or data archiving;

[0011] Supplement the missing metadata of ground-based meteorological observation data, including missing metadata information, data classification information, intellectual property information, serviceable user information, and metadata information that cannot be automatically connected in the metadata warehouse standard framework;

[0012] An intelligent quality control engine for generating metadata of ground-based meteorological observation data, wherein the intelligent charge engine performs quality control on the metadata, provides a metadata charge result analysis report, and corrects abnormal data; and

[0013] The metadata of ground-based meteorological observation data is stored and managed throughout its life cycle.

[0014] In one embodiment of the present invention, the quality control of metadata by the intelligent charging engine includes:

[0015] The metadata is classified according to its properties, and integrity checks are performed on different types of metadata. The ground meteorological observation data transmission quality integrity rate statistics of n meteorological observation stations are performed. The total number of files transmitted by the i-th meteorological observation station during the observation operation is the number of files to be received i. The number of files arriving within the j-th minute after the punctual time is counted, and the transmission integrity quality statistics of the observation data of the i-th meteorological observation station in the j-th minute are expressed as follows:

[0016]

[0017] The unique mark of the main data of the metadata is used as the associated element value of its related data. After completing the integrity check of the main data, the consistency check of the related data of the main data is performed. The number of main data information that should be received by the i-th meteorological observation station during the observation operation is r, the amount of related data of each main data is s, and the actual amount of data received is m. The number of files that arrived is counted, and the statistical results of the consistency quality of the observation data transmission of the i-th meteorological observation station are expressed as follows:

[0018] ;as well as

[0019] The metadata is entered into the checklist for accuracy check, expressed as follows:

[0020]

[0021] Among them, m represents the total number of tables that need to be checked, ci represents the number of correct elements checked in the i-th table, s i Represents the total number of elements checked in the i-th table.

[0022] In one embodiment of the present invention, it is provided that the whole process of constructing the blood relationship of the ground meteorological observation data based on the metadata includes:

[0023] Constructing a meteorological metadata relationship matrix, wherein the rows and columns of the meteorological metadata relationship matrix represent metadata objects, and the checkpoints between metadata objects represent the relationships between metadata, wherein the meteorological metadata relationship matrix includes the following information: metadata type, metadata attribute, metadata relationship, metadata source, and metadata classification; and

[0024] Construct a full-process traceability model for ground meteorological metadata, which includes: recording the source of metadata; processing metadata data sets, including data filtering, data cleaning, data aggregation, and data analysis; conducting quality evaluation on the data accuracy, data integrity, and data consistency of metadata data sets; providing metadata services, including data analysis and mining, and data visualization; and archiving data, including archiving according to data type and managing and monitoring data after archiving.

[0025] In one embodiment of the present invention, it is provided that constructing a full-link data portrait of ground meteorological observation data using a knowledge graph based on the metadata includes:

[0026] Based on the full-process blood relationship of ground-based meteorological observation data, knowledge graph technology is used to associate and query metadata;

[0027] Analyze the transmission link of ground meteorological observation data and construct a full-link data portrait of ground meteorological observation data; and

[0028] The transmission link of ground meteorological observation data is monitored, and the full-link data portrait is updated.

[0029] In one embodiment of the present invention, according to the full-process blood relationship of ground meteorological observation data, using knowledge graph technology to perform associated storage and query of metadata includes the following steps:

[0030] Using the metadata as a data source for constructing a knowledge graph;

[0031] Analyze the fields, purposes, and scenarios for constructing knowledge graphs;

[0032] Define the concept hierarchy, attribute relationship and association relationship between concepts based on observation metadata, system and storage metadata, quality metadata, archiving metadata and service metadata;

[0033] Write OWL files and import them into knowledge graph building tools;

[0034] Add examples to generate knowledge graphs;

[0035] Extract specific instances and their relationship attributes through knowledge graph query analysis;

[0036] Define the dimensions and indicators required for data profiling goals and scenarios;

[0037] Perform data extraction and conversion, extract entity attributes from the knowledge graph, integrate the data into the format required for data profiling, and use machine learning to generate data profiling; and

[0038] Use and display data portraits.

[0039] In one embodiment of the present invention, it is provided that the transmission link of the ground meteorological observation data is analyzed, and the full-link data portrait of the ground meteorological observation data is constructed, including constructing a time dimension portrait and a space dimension portrait, wherein the construction of the time dimension portrait includes classifying the data according to the time quick reading to obtain the relationship of the data in the time dimension, which is expressed as the following formula:

[0040]

[0041] Among them, G y represents a data set with year and month y, y represents year and month, i represents the index of the data, m represents the total number of data, f(t i ) means that the time element t of each data i A function that maps to the year and month of the device; and

[0042] The construction of spatial dimension portrait includes extracting regional elements from ground meteorological observation data, and obtaining information on provinces, cities, and districts by comparing with the regional relationship table. The relationship of data in spatial dimension is obtained by combining the longitude and latitude of the observation station. m} represents a set of m data, with R ij Represents data m i With data m j The relationship between K={K1,K2,…,K n} represents the set of relationship coefficients.

[0043] The present invention also proposes a ground meteorological observation data image system based on metadata, comprising:

[0044] A metadata management module, which is configured to manage metadata of ground meteorological observation data throughout its life cycle;

[0045] A full-process lineage relationship module, which is configured to construct a full-process lineage relationship of ground meteorological observation data based on the metadata; and

[0046] A data profiling module is configured to construct a full-link data profiling of ground meteorological observation data using a knowledge graph based on the metadata.

[0047] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps according to the method when executed by a processor.

[0048] The present invention also provides a computer system, comprising:

[0049] a processor configured to execute machine-executable instructions; and

[0050] A memory having machine executable instructions stored thereon, which when executed by a processor perform the steps according to the method.

[0051] The present invention has at least the following beneficial effects:

[0052] (1) The management efficiency of the entire life cycle of ground meteorological data has been improved. The various metadata of ground meteorological data contain the main information about the entire life cycle of the data, including observation station information, data set information, data quality control information, data archiving information, and data service application information. Through the centralized management and unified analysis of these metadata, a large amount of ground meteorological observation data can be better organized and managed, improving the operating efficiency of the system and saving computing resources.

[0053] (2) Enhanced quality control of ground-based meteorological data. Metadata can be used to evaluate the accuracy, completeness, and consistency of data, identify possible data quality issues, and take appropriate measures to correct and improve them.

[0054] (3) Provide full-process quality traceability. Based on the blood relationship of various metadata of ground meteorological data, the quality of ground meteorological data can be traced throughout the entire process from data collection, storage management, processing, quality control, service utilization, and data archiving.

[0055] (4) Enhanced multi-field service capabilities. Through data profiling, we can better understand user needs and provide more personalized and accurate meteorological services for agriculture, transportation, energy, aviation and other fields.

[0056] (5) It has promoted data sharing among multiple industries. The standardization and regularization of data portraits will help promote the sharing and exchange of data among multiple industries such as tourism, logistics, electricity, and construction, and promote the widespread application of meteorological data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] To further illustrate the advantages and features of various embodiments of the present invention, a more detailed description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings only depict typical embodiments of the present invention and are therefore not to be considered as limiting the scope thereof. In the accompanying drawings, for clarity, identical or corresponding parts will be represented by identical or similar reference numerals.

[0058] Figure 1 A computer system is shown that implements the system and / or method according to the present invention.

[0059] Figure 2 A schematic flow chart of a method for ground meteorological observation data profiling based on metadata in one embodiment of the present invention is shown.

[0060] Figure 3 A schematic diagram of a process for associative storage and query of metadata based on knowledge graph technology according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0061] It should be noted that the components in the figures may be shown exaggeratedly for the sake of illustration and are not necessarily true to scale. In the figures, identical or functionally identical components are provided with the same reference numerals.

[0062] In the present invention, unless otherwise specified, "arranged on...", "arranged above..." and "arranged above..." do not exclude the existence of an intermediate between the two. In addition, "arranged on or above..." merely indicates the relative positional relationship between two components, and in certain cases, such as after reversing the product direction, it can also be converted into "arranged below or below...", and vice versa.

[0063] In the present invention, each embodiment is only intended to illustrate the aspects of the present invention and should not be construed as limiting.

[0064] In the present invention, unless otherwise specified, the quantifiers "a", "an" and "an" do not exclude the presence of a plurality of elements.

[0065] It should also be noted that in the embodiments of the present invention, for the sake of clarity and simplicity, only a portion of the parts or components may be shown, but those of ordinary skill in the art will understand that under the teachings of the present invention, the required parts or components may be added according to the needs of the specific scenario. In addition, unless otherwise specified, the features in different embodiments of the present invention may be combined with each other. For example, a feature in the second embodiment may be used to replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment also falls within the disclosure scope or recorded scope of the present application.

[0066] It should also be noted that within the scope of the present invention, the terms "same", "equal", "equal to" and the like do not mean that the values ​​of the two are absolutely equal, but allow a certain reasonable error, that is, the terms also cover "substantially the same", "substantially equal", "substantially equal to". By analogy, in the present invention, the terms "perpendicular to", "parallel to" and the like indicating directions also cover the meanings of "substantially perpendicular to" and "substantially parallel to".

[0067] In addition, the numbering of the steps of the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps can be executed in different orders.

[0068] The present invention is based on the following insight of the inventors: in recent years, the types of ground meteorological data have been continuously enriched, and the amount of data has grown rapidly. How to efficiently manage, share and utilize these massive data is a key issue that needs to be solved urgently. Although meteorological metadata construction has achieved many results, there are still problems such as decentralized metadata storage, insufficient mining application depth, and unclear benefits.

[0069] In response to the above problems, the present invention aims at the entire life cycle of ground meteorological observation data, dynamically collects meteorological metadata of multiple business systems in the meteorological department, realizes metadata management of the entire life cycle of ground meteorological observation data, constructs the whole process of ground meteorological data metadata blood relationship, and forms a panoramic view of the management and application of ground meteorological data based on knowledge graph technology for ground meteorological observation data from data collection, quality control, data processing, data archiving, and data service. It realizes the management of the entire life cycle of ground meteorological data, thereby improving the governance capacity of meteorological data assets. At the same time, it lays a solid foundation for the supervision of meteorological data services, and promotes the orderly sharing, compliant circulation and property rights protection of meteorological data.

[0070] The present invention will be further described below in conjunction with specific embodiments with reference to the accompanying drawings.

[0071] Figure 1 FIG. 1 shows a computer system 100 for implementing the system and / or method according to the present invention. Unless otherwise specified, the method and / or system according to the present invention may be implemented in Figure 1 The present invention may be implemented in the computer system 100 shown in the figure to achieve the purpose of the present invention, or the present invention may be implemented in a distributed manner in multiple computer systems 100 according to the present invention through a network, such as a local area network or the Internet. The computer system 100 of the present invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0072] like Figure 1 As shown, the computer system 100 includes a processor 111, a system bus 102, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a universal serial bus (USB) interface 112. The system bus 101 can be any of several types of bus structures, such as a memory bus or a memory controller, a peripheral bus, and a local bus using various types of bus architectures. The system bus 101 is used for communication between various bus devices. In addition to Figure 1 In addition to the bus devices or interfaces shown in , other bus devices or interfaces are also conceivable. The system memory 102 includes a read-only memory (ROM) 103 and a random access memory (RAM) 104, wherein the ROM 103 can, for example, store basic input / output system (BIOS) data for implementing basic routines for information transmission at startup, and the RAM 104 is used to provide the system with a running memory with a faster access speed. The computer system 100 also includes a hard disk drive 109 for reading and writing to the hard disk 110, an optical drive interface 113 for reading and writing to optical media such as CD-ROMs, and the like. The hard disk 110 can, for example, store an operating system and application programs. The drive and its associated computer-readable medium provide the computer system 100 with non-volatile storage of computer-readable instructions, data structures, program modules, and other data. The computer system 100 can also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as a display 106. The computer system 100 may further include an audio adapter 107 for audio processing and / or audio output, which is used to connect output devices such as speakers 108. In addition, the computer system 100 may further include a network interface 114 for network connection, wherein the network interface 114 may be connected to the Internet 116 through a network device such as a router 115, wherein the connection may be wired or wireless. In addition, the computer system 100 may further include a universal serial bus interface (USB) 112 for connecting peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices, such as a microphone, a camera, etc.

[0073] When the present invention Figure 1 When implemented on the computer system 100, the management efficiency of the entire life cycle of ground meteorological data can be improved. Through centralized management and unified analysis of metadata, a large amount of ground meteorological observation data can be better organized and managed, thereby improving the operating efficiency of the system and saving computing resources.

[0074] Furthermore, the embodiments may be provided as a computer program product that may include one or more machine-readable media having machine-executable instructions stored thereon, which, when executed by one or more machines such as a computer, a computer network or other electronic device, may cause the one or more machines to perform operations according to the embodiments of the present invention. The machine-readable medium may include, but is not limited to, a floppy disk, an optical disk, a CD-ROM (compact disk read-only memory) and a magneto-optical disk, a ROM (read-only memory), a RAM (random access memory), an EPROM (erasable programmable read-only memory), an EEPROM (electrically erasable programmable read-only memory), a magnetic or optical card, a flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0075] In addition, various embodiments may be downloaded as a computer program product, wherein the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Thus, a machine-readable medium as used herein may include such a carrier wave, but this is not required.

[0076] In the present invention, each module of the system according to the present invention can be implemented using software, hardware, firmware or a combination thereof. When the module is implemented using software, the function of the module can be implemented through a computer program flow, for example, the module can be implemented by a code segment (such as a code segment of a language such as C, C++, etc.) stored in a storage device (such as a hard disk, memory, etc.), wherein when the code segment is executed by a processor, the corresponding function of the module can be implemented. When the module is implemented using hardware, the function of the module can be implemented by setting a corresponding hardware structure, for example, the function of the module can be implemented by hardware programming a programmable device such as a field programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) including electronic devices such as multiple transistors, resistors and capacitors to implement the function of the module. When the module is implemented using firmware, the function of the module can be written into a read-only memory such as an EPROM or EEPROM of the device in the form of a program code, and when the program code is executed by a processor, the corresponding function of the module can be implemented. In addition, some functions of the module may need to be implemented by separate hardware or by collaboration with the hardware, for example, the detection function is implemented by corresponding sensors (such as proximity sensors, acceleration sensors, gyroscopes, etc.), the signal transmission function is implemented by corresponding communication devices (such as Bluetooth devices, infrared communication devices, baseband communication devices, Wi-Fi communication devices, etc.), the output function is implemented by corresponding output devices (such as displays, speakers, etc.), and so on.

[0077] The present invention realizes the standardized metadata management of the whole life cycle of ground meteorological observation data from the perspective of the whole process of meteorological data observation, quality control, processing, service application, and data archiving. By establishing the upstream and downstream metadata blood relationship of each link of ground meteorological observation data, the link relationship of the whole process of meteorological data is opened up, and the knowledge graph technology is used to construct a full-link portrait of ground meteorological observation data for multi-field and multi-industry applications, which provides a complete implementation plan for the quality traceability of ground meteorological observation data and high-value industry applications.

[0078] Figure 2 FIG. 1 is a flow chart showing a method for profiling ground meteorological observation data based on metadata in one embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0079] Step 201: Manage metadata of ground-based meteorological observation data throughout its entire life cycle.

[0080] Step 202: construct the whole process blood relationship of the ground meteorological observation data based on the metadata.

[0081] Step 203: construct a full-link data portrait of ground meteorological observation data using a knowledge graph based on the metadata.

[0082] The method is described in detail below.

[0083] In the present invention, firstly, metadata management of the ground meteorological observation data is performed throughout its life cycle. It is necessary to connect to the relevant business systems within the meteorological department, collect metadata information of the observation, storage and processing, quality, archiving, and service (utilization) of real-time and historical data of ground meteorological observation stations in each life cycle, and formulate metadata recording rules and quality control rules for ground meteorological observation data based on meteorological metadata industry standards, carry out classification, extraction, cleaning, recording, and quality control of various types of metadata, and form a metadata warehouse for ground meteorological observation data. Specifically, the metadata compilation process for the entire life cycle of ground meteorological observation data is as follows:

[0084] (1) Metadata collection of the entire life cycle of multi-source systems of ground meteorological observation data, including the formulation of a docking and collection strategy for metadata information of ground meteorological observation data, docking with the multi-source business systems within the meteorological department (observation system, quality control system, archiving system, service application system), and performing metadata extraction, importing, cleaning, and classification throughout the entire life cycle of data collection, storage management, processing, quality control, service utilization, data archiving, etc., to achieve dynamic collection of metadata of multi-system ground meteorological observation data within the meteorological department.

[0085] (2) Metadata supplementation of ground meteorological observation data. On the basis of connecting to the metadata of the multi-source system of ground meteorological observation data of the meteorological department, manual supplementation work is added to supplement and improve the existing metadata information. The metadata information entered for the first time includes metadata information missing in the relevant system, data classification information, intellectual property information, serviceable user information, etc., to supplement and improve metadata information that cannot be automatically connected in the metadata warehouse standard framework.

[0086] (3) Generate an intelligent quality control engine for ground meteorological observation data metadata. Based on various meteorological metadata industry standards such as meteorological observation metadata, meteorological data discovery metadata, meteorological archive metadata, and meteorological station metadata, an engine for metadata quality control rules for the entire life cycle of ground meteorological observation data is automatically generated. According to various metadata verification rules, the engine automatically completes the quality control of ground meteorological data metadata, provides metadata quality control result analysis reports, and performs error correction on abnormal data.

[0087] (4) The metadata of ground-based meteorological observation data will be stored and managed throughout its life cycle. In order to facilitate the quality management of data throughout its life cycle and taking into account the scalability and compatibility issues that the system will face in the future, the overall storage of various metadata of ground-based meteorological observation data will be constructed in accordance with the full life cycle of meteorological big data observation, storage, quality, archiving, and service (utilization).

[0088] The detailed process of quality control of metadata by the intelligent accusation engine is as follows:

[0089] Step 1: Classify metadata according to the nature of the data, and perform integrity check on each metadata according to the metadata integrity rule constraint check standard of different data types. During the data collection process, confirm whether the original data is correctly obtained based on the collection mark code. If the acquisition fails, enter the re-collection process. Convert the collected original data to form basic data objects. The integrity check includes:

[0090] Assume that there are n (n=1, 2, ..., i, ...) stations for observation data transmission quality statistics. The total number of files transmitted by the i-th station during the observation operation is the number of files to be received i. The number of files arriving within the j-th minute after the punctuality is counted, and the transmission integrity quality statistics of the j-th minute of the observation data of the station can be expressed as follows:

[0091]

[0092] Step 2: Use the unique identifier of the main data of the metadata as the associated element value of its related data. After verifying the integrity of the main data, a consistency check is also required to see if the related data has been fully acquired. With the main data as the center and the related data as the branch, a complete data relationship diagram is formed. Retrieve data from multiple angles and dimensions according to the rules and business rules agreed upon in the data relationship model. The consistency check includes: statistics on the quality of observation data transmission for a certain station. The number of main data information that should be received by the i-th station during the observation operation is r, and each main data is associated with other information data s, and the actual amount of data received is m. The number of files that have arrived is counted, and the statistical results of the consistency quality of the observation data transmission at this station can be expressed as follows:

[0093] .

[0094] Step 3: Develop an accuracy check control that integrates data format check, type check, boundary value check, uniqueness check, and business logic check for different metadata types and business rules. Data that fails the accuracy check will generate an accuracy check report after the check is completed. The report includes metadata unique identifier, data element error type, error analysis, this check process log, and other related information. The accuracy check algorithm can be as follows:

[0095] Among them, m represents the total number of tables that need to be checked, c i represents the number of correct elements checked in the i-th table, s i Represents the total number of elements checked in the i-th table.

[0096] Furthermore, based on the element characteristics of ground meteorological observation data, such as the four-level coding of meteorological data and station number, the whole process of lineage relationship of observation metadata, system and storage metadata, quality metadata, archiving metadata and service metadata is connected, starting from the analyzed meteorological data entity, recursive analysis is carried out back to the starting point of the data flow, providing metadata support for subsequent link analysis, root tracing, impact analysis, etc. The process of building lineage relationship of ground meteorological observation data includes:

[0097] (1) Construct a relationship matrix of various meteorological metadata. The meteorological metadata relationship matrix is ​​a structured metadata framework used to record the relationship between metadata. In the metadata relationship matrix, each row and column represents a metadata object, and the intersection of each metadata object represents the relationship between them. The metadata relationship matrix can conveniently display the dependencies between metadata, helping users to better understand and manage metadata. The meteorological metadata relationship matrix includes the following information: meteorological metadata type, which is used to describe the types of various meteorological metadata, such as metadata elements, metadata data sets, metadata quality, etc.; metadata attributes, which are used to record detailed descriptions of metadata, such as name, definition, data type, value range, etc.; metadata relationships, which are used to describe the dependencies between metadata, such as input and output of data conversion, data processing methods, data lineage, etc.; metadata sources, which are used to record the sources of various metadata of ground meteorological observation data; metadata classification, which is used to classify and archive metadata according to different categories, uses or functions for easy management and query.

[0098] (2) Establish a full-process traceability model for ground meteorological metadata.

[0099] The relevant data are associated, and the source, processing process, processing method, service application, archival storage and other information of each data are recorded, wherein the data source includes the original source of ground meteorological data, including observation stations, observation instruments, and time intervals. Further, the metadata data set is processed, including data filtering, data cleaning, data aggregation, data analysis, etc., wherein the data source and time are used as unique identifiers to remove data duplication; the data is escaped and analyzed according to the identification codes of different elements; the data is identified based on the analysis results; the data element values ​​are counted and calculated, and statistical results or reports are generated. Further, the quality of the metadata data set is evaluated, including indicators such as data accuracy, data integrity, and data consistency. The data accuracy, integrity, consistency, reliability, and timeliness are evaluated. Further, metadata services are provided, including: data analysis and mining, wherein real-time statistics, calculations, and personalized intelligent analysis of data are realized through real-time computing services and intelligent analysis services based on machine learning, and data value is mined; data visualization, wherein data is displayed in the form of charts, reports, etc., to help users intuitively understand the data situation. Furthermore, data archiving is performed, in which a special archiving plan is customized according to the characteristics of each type of data, including the data storage location, archiving standards, archiving cycle, cleanup strategy, etc. After archiving, the data needs to be managed and monitored to ensure the integrity, availability, credibility and security of the data.

[0100] Through the above steps, the blood relationship of the entire life cycle of ground meteorology can be constructed based on metadata, and the relationship between various data elements in the data flow process can be recorded, including information such as metadata input and output, data conversion methods, and data quality indicators.

[0101] (3) Realize the quality traceability of ground meteorological data throughout the entire process. Develop the meteorological data quality metadata traceability function based on the meteorological data quality metadata relationship matrix and traceability model. Trace the meteorological data quality metadata, starting from the analyzed entity and recursively analyzing backwards until the starting point of the data flow, to achieve the quality traceability of ground meteorological observation data throughout the entire process from observation, processing, service application, to archiving.

[0102] Furthermore, a full-link data portrait of ground meteorological observation data is constructed based on metadata, wherein a knowledge graph is used to construct a full-link data portrait of ground meteorological observation data, including:

[0103] (1) Relationship linking of ground meteorological observation metadata based on knowledge graph technology. Based on the relationship of metadata of ground meteorological observation data throughout its life cycle, such as the relationship between observation sites and observation elements, and the relationship between observation time and observation data, the knowledge graph entity linking technology is used to realize the associative storage and query of metadata. The detailed relationship construction process is as follows: Figure 3 shown.

[0104] (2) Construct a full-link data portrait, analyze the entire transmission link of data from observation equipment to final storage and application, including data collection points, transmission paths, transfer nodes, etc. Based on the analysis results, construct a full-link data portrait of ground meteorological observation, covering the source of ground observation data, processing methods, storage location, service industry, and utilization value, and intuitively and comprehensively display the characteristics and value of ground meteorological observation data. The full-link data portrait includes hourly ground meteorological data, and the data source is ground observation data. Data processing is required, including ground observation data, ground temperature hourly data, ground precipitation hourly data, hourly snow depth data set, and ground important weather data.

[0105] The time dimension portrait includes classification according to the time span of the data, so as to obtain the relationship between the data in the time dimension, which is expressed as the following formula:

[0106]

[0107] Among them, G y represents a data set with year and month y, y represents year and month, i represents the index of the data, m represents the total number of data, f(t i ) means that the time element t of each data i Maps to the function corresponding to the year and month.

[0108] Spatial dimension portrait includes extracting regional elements AREA from the data, comparing with the regional relationship table, obtaining information on provinces, cities, and districts, and combining the longitude and latitude analysis of the district station to obtain the relationship of the data in the spatial dimension.

[0109] Where M={m1,m2,…,m m} represents a set of m data, with R ij Represents data m i With data m j The relationship between (where i∈j) is represented by K={K1,K2,…,K n} represents the set of relationship coefficients. For each data m i , can have a set of relationship coefficients with other data, expressed as {R i1 ,R i2 ,…,R im}, where R ij is m i With data m j The relationship coefficient between ij Can belong to a certain K of the set K clock k For example, if the relationship between m1 and m2, m3, m4, and m5 is [K1:m2], [K2:m3], and [K3:m4,m5] respectively, then this can be interpreted as: 12 =K1, R 13 =K2,R 14 =R 15 =K3.

[0110] (3) Update and apply the full-link data portrait, continuously monitor the full link of ground meteorological observation data, and update the portrait in a timely manner to reflect the dynamic changes of data and links. The constructed ground meteorological observation data portrait will be applied to multiple fields such as meteorological forecasting, climate change research, and agricultural production guidance, providing scientific basis and data support for decision-making in related fields.

[0111] In one embodiment of the present invention, a ground meteorological observation data profiling system based on metadata is also proposed, including:

[0112] A metadata management module, which is configured to manage metadata of ground meteorological observation data throughout its life cycle;

[0113] A full-process lineage relationship module, which is configured to construct a full-process lineage relationship of ground meteorological observation data based on the metadata; and

[0114] A data profiling module is configured to construct a full-link data profiling of ground meteorological observation data using a knowledge graph based on the metadata.

[0115] In the present invention, the establishment of the blood relationship of ground meteorological observation data depends on the metadata collation of the ground meteorological observation data throughout its life cycle. The blood relationship between data sets is established by collating the title, abstract, data set code and other fields of the ground meteorological observation metadata. By establishing the blood relationship and associating the observation elements, observation time, site information and other data of the ground meteorological observation metadata, a complete data portrait with a certain blood relationship is formed, and a scientific basis and data support are provided for multiple fields such as meteorological change research, agricultural production guidance, and meteorological forecasting.

[0116] Although various embodiments of the present invention are described above, it should be understood that they are presented as examples only and not as limitations. It is obvious to those skilled in the relevant art that various combinations, modifications and changes can be made thereto without departing from the spirit and scope of the present invention. Therefore, the breadth and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should only be defined according to the attached claims and their equivalents.

Claims

1. A ground meteorological observation data profiling method based on metadata, characterized in that: The following steps are involved: Manage metadata of ground-based meteorological observation data throughout its life cycle; constructing the whole process blood relationship of ground meteorological observation data based on the metadata; as well as Based on the metadata, a knowledge graph is used to construct a full-link data portrait of ground meteorological observation data; The whole process of constructing ground meteorological observation data based on the metadata includes: Constructing a meteorological metadata relationship matrix, wherein the rows and columns of the meteorological metadata relationship matrix represent metadata objects, and the checkpoints between metadata objects represent the relationships between metadata, wherein the meteorological metadata relationship matrix includes the following information: metadata type, metadata attribute, metadata relationship, metadata source, and metadata classification; and Construct a full-process traceability model for ground meteorological metadata, which includes: recording the source of metadata; processing metadata data sets, including data filtering, data cleaning, data aggregation, and data analysis; conducting quality evaluation on the data accuracy, data integrity, and data consistency of metadata data sets; providing metadata services, including data analysis and mining, and data visualization; and archiving data, including archiving according to data type and managing and monitoring data after archiving. The full-link data portrait of ground meteorological observation data constructed based on the metadata using the knowledge graph includes: Based on the full-process blood relationship of ground-based meteorological observation data, knowledge graph technology is used to associate and query metadata; Analyze the transmission link of ground meteorological observation data and construct a full-link data portrait of ground meteorological observation data; and The transmission link of ground meteorological observation data is monitored, and the full-link data portrait is updated.

2. The method for ground meteorological observation data profiling based on metadata according to claim 1, characterized in that: The management of metadata throughout the life cycle of ground-based meteorological observation data includes: Collect metadata of ground-based meteorological observation data throughout its life cycle, including docking with the multi-source business systems of the meteorological department, extracting, importing, cleaning and / or classifying metadata throughout its life cycle through data collection, storage management, processing, quality control, service utilization and / or data archiving; Supplement the missing metadata of ground-based meteorological observation data, including missing metadata information, data classification information, intellectual property information, serviceable user information, and metadata information that cannot be automatically connected in the metadata warehouse standard framework; An intelligent quality control engine for generating metadata of ground-based meteorological observation data, wherein the intelligent charge engine performs quality control on the metadata, provides a metadata charge result analysis report, and corrects abnormal data; and The metadata of ground-based meteorological observation data is stored and managed throughout its life cycle.

3. The method for ground meteorological observation data profiling based on metadata according to claim 2, characterized in that: The quality control of metadata by the intelligent charge engine includes: The metadata is classified according to its properties, and integrity checks are performed on different types of metadata. The ground meteorological observation data transmission quality integrity rate statistics of n meteorological observation stations are performed. The total number of files transmitted by the i-th meteorological observation station during the observation operation is the number of files to be received i. The number of files arriving within the j-th minute after the punctual time is counted, and the transmission integrity quality statistics of the observation data of the i-th meteorological observation station in the j-th minute are expressed as follows: The unique mark of the main data of the metadata is used as the associated element value of its related data. After completing the integrity check of the main data, the consistency check of the related data of the main data is performed. The number of main data information that should be received by the i-th meteorological observation station during the observation operation is r, the amount of related data of each main data is s, and the actual amount of data received is m. The number of files that arrived is counted, and the statistical results of the consistency quality of the observation data transmission of the i-th meteorological observation station are expressed as follows: ;as well as The metadata is entered into the checklist for accuracy check, expressed as follows: Among them, m represents the total number of tables that need to be checked, c i represents the number of correct elements checked in the i-th table, s i Represents the total number of elements checked in the i-th table.

4. The method for ground meteorological observation data profiling based on metadata according to claim 1, characterized in that: According to the whole process of the ground meteorological observation data, the use of knowledge graph technology for metadata association storage and query includes the following steps: Using the metadata as a data source for constructing a knowledge graph; Analyze the domain, purpose, and scenario of constructing knowledge graphs; Define the concept hierarchy, attribute relationship and association relationship between concepts based on observation metadata, system and storage metadata, quality metadata, archiving metadata and service metadata; Write OWL files and import them into knowledge graph building tools; Add examples to generate knowledge graphs; Extract specific instances and their relationship attributes through knowledge graph query analysis; Define the dimensions and indicators required for data profiling goals and scenarios; Perform data extraction and conversion, extract entity attributes from the knowledge graph, integrate the data into the format required for data profiling, and use machine learning to generate data profiling; as well as Use and display data portraits.

5. The method for ground meteorological observation data profiling based on metadata according to claim 4, characterized in that: The transmission link of ground meteorological observation data is analyzed, and the full-link data portrait of ground meteorological observation data is constructed, including the construction of time dimension portrait and space dimension portrait. The construction of time dimension portrait includes classifying the data according to the time reading to obtain the relationship of data in the time dimension, which is expressed as the following formula: Among them, G y represents a data set with year and month y, y represents year and month, i represents the index of the data, m represents the total number of data, f(t i ) means that the time element t of each data i A function that maps to the year and month of the device; and The construction of spatial dimension portrait includes extracting regional elements from ground meteorological observation data, obtaining information on provinces, cities, and districts by comparing with the regional relationship table, and obtaining the relationship of data in spatial dimension by combining the longitude and latitude of the observation station. m } represents a set of m data, with R ij Represents data m i With data m j The relationship between K={K1,K2,…,K n } represents the set of relationship coefficients.

6. A ground meteorological observation data image system based on metadata, characterized in that: include: A metadata management module, which is configured to manage metadata of ground meteorological observation data throughout its life cycle; A full-process lineage relationship module, which is configured to construct a full-process lineage relationship of ground meteorological observation data based on the metadata; as well as A data profiling module is configured to construct a full-link data profiling of ground meteorological observation data using a knowledge graph based on the metadata, in The whole process blood relationship module constructs the whole process blood relationship of ground meteorological observation data based on the metadata, including: Constructing a meteorological metadata relationship matrix, wherein the rows and columns of the meteorological metadata relationship matrix represent metadata objects, and the checkpoints between metadata objects represent the relationships between metadata, wherein the meteorological metadata relationship matrix includes the following information: metadata type, metadata attribute, metadata relationship, metadata source, and metadata classification; and Construct a full-process traceability model for ground meteorological metadata, which includes: recording the source of metadata; processing metadata data sets, including data filtering, data cleaning, data aggregation, and data analysis; conducting quality evaluation on the data accuracy, data integrity, and data consistency of metadata data sets; providing metadata services, including data analysis and mining, and data visualization; and archiving data, including archiving according to data type and managing and monitoring data after archiving. The data profiling module uses the knowledge graph to construct a full-link data profiling of ground meteorological observation data based on the metadata, including: Based on the full-process blood relationship of ground-based meteorological observation data, knowledge graph technology is used to associate and query metadata; Analyze the transmission link of ground meteorological observation data and construct a full-link data portrait of ground meteorological observation data; and The transmission link of ground meteorological observation data is monitored, and the full-link data portrait is updated.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program, when executed by a processor, performs the steps of the method according to any one of claims 1 to 5.

8. A computer system, characterized in that: include: a processor configured to execute machine-executable instructions; as well as A memory having machine executable instructions stored thereon, wherein the machine executable instructions, when executed by a processor, perform the steps of the method according to any one of claims 1 to 5.

Images