Checking and processing method for ensuring data consistency of two-place three-center automatic meteorological station

Abstract

The invention discloses a checking and processing method for ensuring data consistency of two-place and three-center automatic weather stations. The method comprises the following steps: respectively constructing HashMap sets of a main center and a standby center for automatic station data stored in the main center and the standby center; comparing the HashMap sets of the main center and the standby center to obtain a record difference set and an element difference set; preferentially processing the record difference set and the element difference set of the Beijing standby center, and then performing the same processing on the Xian standby center. A comparison value character string is constructed for each row of records of an automatic weather station data table in the main and standby centers, MD5 encryption calculation is carried out to construct a Hash code, a HashMap set comparison file of the row record main and standby centers is established, and a record difference set file and an element difference set file generated by the main and standby centers are compared, so that a comparison result is obtained by taking the main center as a benchmark. And inconsistent data records in the standby center are quickly positioned, and the different data records are taken out from the main center to update the standby center, so that the consistency and reliability of the data are ensured.

Description

Technical Field

[0001] The present invention relates to the technical field of data consistency design, and more specifically, to a method for ensuring data consistency of automatic weather stations in two locations and three centers (a national data center in Beijing is the main center, and two backup centers are respectively set up in Beijing and Xi'an). Background Art

[0002] An automatic weather station refers to an observation device that can automatically collect and transmit meteorological information. It consists of meteorological sensors, microcomputer meteorological data acquisition instruments, power supply systems, radiation-proof ventilation hoods, all-weather protection boxes, meteorological observation brackets, communication modules, and other parts. The automatic weather station is used to conduct all-weather on-site monitoring of more than a dozen meteorological elements, including wind speed, wind direction, rainfall, air temperature, air humidity, light intensity, soil temperature, soil humidity, evaporation, and atmospheric pressure. At present, my country has established more than 70,000 automatic stations, which are uploaded to the national data center through provincial level. The data collected by the automatic weather station is one of the core data for meteorological monitoring, forecasting, and services.

[0003] In order to ensure the security of meteorological business processes and data, TianQing has established a backup center in Xi'an. TianQing needs to design backups in the main center and the backup center according to the business processes of data reception, processing, storage and service. Due to various reasons, such as the delay in data arrival or processing leading to deviations in basic statistical results, the same data currently has a daily difference of less than 5% in the main and backup centers, and efficient technical means or processes are needed to ensure data consistency. Summary of the invention

[0004] To this end, the technical problem to be solved by the present invention is to provide a method for checking and processing data consistency of automatic weather stations in two locations and three centers, which can ensure the consistency and reliability of data of the main center in Beijing and the backup centers in Beijing and Xi'an respectively.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] A method for checking and processing data consistency of automatic weather stations in two locations and three centers, with a national data center in Beijing as the main center and two backup centers in Beijing and Xi'an respectively, including the following steps:

[0007] (1) Construct a HashMap collection of the main center and a HashMap collection of the backup center for the automatic station data stored in the main center and the backup center respectively;

[0008] (2) Compare and check the HashMap set of the primary center and the HashMap set of the backup center to obtain the record difference set and the element difference set;

[0009] (3) Process the record difference set and the element difference set.

[0010] In the above-mentioned method for checking and processing the consistency of the data of the automatic weather stations in two locations and three centers, in step (1), the automatic station data is preprocessed: when the automatic station data is stored, the automatic station number and the observation time are used as the joint primary key to uniquely identify a row of data records R.

[0011] The above-mentioned method for checking and processing data consistency of automatic weather stations in two locations and three centers generates a joint primary key string set PK(R) and a comparison field string set V(R) for each data record R of the primary center and the backup center respectively;

[0012] PK(R) = Concatenate(Select(Fk, R)) (1);

[0013] V(R) = Concatenate(Select(Fc, R)) (2);

[0014] In formula (1) and formula (2), Select(F,R) is the field value of the data field set F in the data record R retrieved according to the agreed format, Fk represents the joint primary key field set, Fc represents the comparison field set, and Concatenate connects the field values ​​into a string set according to the agreed rules.

[0015] The above-mentioned method for checking and processing the data consistency of the automatic weather station in two locations and three centers performs MD5 encryption calculation on the comparison field string set V(R) of the main center and the backup center, thereby forming a Hashmap structure with the data of the comparison time period:

[0016] MD5_Hash(R)=MD5(V(R));

[0017] MD5(s): represents the function of performing MD5 encryption calculation on string s.

[0018] The above-mentioned method for checking and processing the data consistency of the automatic weather station in two locations and three centers constructs a HashMap collection for the two data sources of the main center and the backup center respectively:

[0019] H_m={(PK(R_m),MD5_Hash(R_m))|R_m∈main center data source};

[0020] H_b={(PK(R_b),MD5_Hash(R_b))|R_b∈backup center data source};

[0021] H_m represents the HashMap collection constructed by the main center data source, and m represents the main center system;

[0022] H_b represents the HashMap collection constructed by the backup center data source, and b represents the backup center system.

[0023] In the above-mentioned method for checking and processing data consistency of automatic weather stations in two locations and three centers, in step (2), the key values ​​of the HashMap collections of the primary center and the backup center are traversed to obtain the record difference collection C:

[0024] C={PK(R)|PK(R)∈(H_m.keys()-H_b.keys())};

[0025] C represents the set of joint primary keys of missing records in the backup center;

[0026] keys() represents the key value of the HashMap collection.

[0027] The above-mentioned method for checking and processing data consistency of automatic weather stations in two locations and three centers traverses the element values ​​of the HashMap sets of the primary center and the backup center, checks and compares the MD5 hash values ​​of the comparison field string sets with the same joint primary key records, and obtains the element difference set D: D = {PK(R)|(PK(R)∈H_m.keys()∩H_b.keys())∧(H_m[PK(R)]≠H_b[PK(R)])}.

[0028] The above-mentioned method for checking and processing data consistency of automatic weather stations in two locations and three centers, in step (3), obtains data information on inconsistency between the main and backup centers based on the difference comparison method, that is, records the difference set and the element difference set, takes the data of the main center as the benchmark, and synchronously updates the data of the backup center.

[0029] The above-mentioned method for checking and processing data consistency of automatic weather stations in two locations and three centers is to retrieve and write data records RS(C) missing from the backup center from the main center for the record difference set C to the backup center.

[0030] The above-mentioned method for checking and processing data consistency of automatic weather stations in three centers in two locations is for the primary and backup centers to have the same joint primary key row, that is, for the element difference set D, there are records RS(D) with inconsistent element values. First, the difference record RS_b(D) is backed up in the backup center, and then the RS_m(D) of the primary center is used to synchronously update the RS_b(D) of the backup center.

[0031] The data processing process gives priority to the same-site backup center. After the same-site backup center is processed, the different-site backup center is processed according to the same process.

[0032] The technical solution of the present invention achieves the following beneficial technical effects:

[0033] 1. Data consistency includes two aspects: one is the consistency of data records, and the other is the consistency of data element values. When comparing data, the data of the main center is used as the benchmark to check and synchronously update the data of the backup center.

[0034] 2. To ensure the safe operation of the business, TianQing established a backup center in Xi'an, which is consistent with the main center in the entire process from data access to data services. Automatic weather station data is one of the important data for meteorological monitoring, forecasting and application services. Focusing on the consistency of automatic weather station data in the main and backup centers, an efficient and reliable data comparison and data update solution is proposed. In the main and backup centers, a comparison value string is constructed for each row of records in the automatic weather station data table, a Hash code is constructed when performing MD5 encryption calculations, and a comparison file of the HashMap collection of row records in the main and backup centers is established. Compared with comparing individual elements one by one, the method of checking and processing the consistency of data elements based on the HashMap collection is more convenient and efficient, and effectively reduces the consumption of computing resources. By comparing and checking the record difference set and element difference set files generated by the main and backup centers, the inconsistent data records of the main and backup centers are quickly located based on the main center, and the difference data records are taken out from the main center and updated to the backup center synchronously to ensure data consistency and reliability.

[0035] 3. The functions have been integrated and fully tested in the TianQing storage management system. The verification results show that the solution effectively improves the consistency of the primary and backup data of the automatic weather station, and provides a guarantee for the primary and backup switching of meteorological services.

[0036] 4. Based on the automatic weather station data of the main center, the data record difference set and feature difference set of the main center and other backup centers are constructed based on HashMap, which can quickly locate the inconsistent data of other backup centers and synchronously update the corresponding data of the main center to other backup centers, effectively improving the efficiency of difference data comparison and ensuring the consistency and reliability of the overall data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Flowchart of data update processing;

[0038] Figure 2 Data processing display diagram. DETAILED DESCRIPTION

[0039] 1. Basic design ideas

[0040] Data consistency includes two aspects: one is the consistency of data records, and the other is the consistency of data element values. When comparing data, the data of the main center is used as the benchmark and the data of the backup center is updated synchronously.

[0041] To ensure the consistency of data records, a data record comparison method is used: the automatic station data is stored with the automatic station number and observation time as the joint primary key to uniquely identify a row of records. The difference between the primary and backup record data is mainly compared through the string set composed of the joint primary key to obtain the data record set C that belongs to the primary center but not to the backup center.

[0042] To ensure the consistency of data element values, a data element value comparison method is used: Since the data is stored according to multiple elements, there are more than 200 data fields, including observed element values, simple statistical values, and quality control codes corresponding to the values. To improve the comparison efficiency, the elements that need to be compared between the primary and backup are retrieved according to certain rules to form a string result set, and the result set is encrypted with MD5. The data of the comparison time period is formed into a Hashmap structure, with Key as the joint primary key string and Value as the value calculated by MD5 encryption. Then the two sets of Hashmaps of the primary and backup are compared according to the same primary key to find the difference value D.

[0043] 2. Specific methods

[0044] (1) Construct HashMap collections of the main center and the backup center for the automatic station data stored in the main center and the backup center respectively;

[0045] (1-1) Preprocessing of automatic station data:

[0046] When storing automatic station data, the automatic station number and observation time are used as the joint primary key to uniquely identify a row of data records R.

[0047] (1-2) Generate a joint primary key string and a comparison value string:

[0048] For each data record R of the primary center and the backup center, generate a joint primary key string set PK(R) and a comparison field string set V(R) respectively;

[0049] PK(R) = Concatenate(Select(Fk, R)) (1);

[0050] V(R) = Concatenate(Select(Fc, R)) (2);

[0051] In formula (1) and formula (2), Select(F,R) is the field value of the data field set F in the data record R retrieved according to the agreed format, Fk represents the joint primary key field set, Fc represents the comparison field set, and Concatenate connects the field values ​​into a string set according to the agreed rules (such as underscore).

[0052] (1-3) Perform MD5 calculation on the comparison result string of each line:

[0053] Perform MD5 encryption calculation on the comparison field string set V(R) of the primary center and the backup center, so as to form a Hashmap structure with the data of the comparison time period:

[0054] MD5_Hash(R)=MD5(V(R));

[0055] MD5(s): represents the function of performing MD5 encryption calculation on string s.

[0056] (1-4) Constructing the primary and backup HashMap collections

[0057] For the two data sources of the primary center and the backup center, construct HashMap collections respectively:

[0058] H_m={(PK(R_m),MD5_Hash(R_m))|R_m∈main center data source};

[0059] H_b={(PK(R_b),MD5_Hash(R_b))|R_b∈backup center data source};

[0060] H_m represents the HashMap collection constructed by the main center data source, and m represents the main center system;

[0061] H_b represents the HashMap collection constructed by the backup center data source, and b represents the backup center system.

[0062] (2) Compare and check the HashMap sets of the primary center and the backup center to obtain the record difference set and the element difference set;

[0063] (2-1) Compare the key value set of HashMap and obtain the record difference set C.

[0064] Traverse the key value sets of the two HashMaps of the primary center and the backup center to obtain the information of the record difference set C.

[0065] C={PK(R)|PK(R)∈(H_m.keys()-H_b.keys())};

[0066] C represents the set of joint primary keys of missing records in the backup center;

[0067] keys() represents the key value of the HashMap collection.

[0068] (2-2) Compare and check the element values ​​of the HashMap set with the same primary key, and find the element difference set D.

[0069] Traverse the two HashMap collections of the primary center and the backup center, check and compare the MD5 hash values ​​of the comparison field string collections with the same joint primary key records, and obtain the information of the feature difference collection D that needs to be updated.

[0070] D={PK(R)|(PK(R)∈H_m.keys()∩H_b.keys())∧(H_m[PK(R)]≠H_b[PK(R)])}

[0071] (3) Process the record difference set and the element difference set.

[0072] Based on the difference comparison method, the data information of the inconsistency between the main and backup centers is obtained, and the data of the backup center is synchronously updated based on the data of the main center.

[0073] (3-1) For the record difference set C, the data records RS(C) missing from the backup center are retrieved from the primary center and written to the backup center.

[0074] (3-2) For the same joint primary key row in the master and backup, that is, for the element difference set D, there are records RS(D) with inconsistent element values, first back up the difference record RS_b(D) in the backup center, and then use RS_m(D) in the master center to synchronously update RS_b(D) in the backup center. In this way, the data elements are completely consistent. The process is as follows Figure 1 shown.

[0075] (3-3) The data processing process gives priority to the same-site backup center. After the same-site backup center is processed, the different-site backup center is processed according to the same process.

[0076] 3. Application Testing and Integration

[0077] (1) Introduction to Data Storage Management System

[0078] It is a cluster software built based on distributed and big data technologies for storing and managing massive meteorological data. It mainly provides intensive management of multi-source heterogeneous distributed database clusters, including data storage model definition, synchronization of multiple copies of application data, data backup, data clearing, data recovery and data archiving; provides data resource list management, classification and grading management, data authority management and user behavior analysis; provides data consistency check and automatic update functions for the main and backup centers. At present, the main center has achieved the management of more than 2,000 types and 16PB of online meteorological data, and provides access services for nearly 200TB of data every day.

[0079] (2) Software integration design and implementation

[0080] Based on the storage management system of the primary and standby data centers, a data consistency comparison software is established. The software is designed using a microservice framework and developed based on springboot. The software functions include main metadata comparison, hash code calculation of primary and standby data table records, primary and standby data comparison, standby center data update, data comparison table configuration, and log recording.

[0081] After the software is deployed, enter the data comparison table configuration interface, select the data table to be compared, and select related attributes, such as the frequency of daily comparison, comparison elements, etc. Click Submit, and the program will compare the metadata information of the configuration table in the primary and backup centers, such as field type, length, etc. If there is any inconsistency, it will prompt that the configuration failed, and the metadata of the data table must be consistent first. When the successfully submitted task runs, the comparison file and difference file are generated from the primary and backup centers according to the comparison method, and then the corresponding data record rows of the backup center are updated based on the difference file and the primary center as the benchmark.

[0082] (3) Data verification effect analysis

[0083] The commonly used Chinese ground hourly and daily data tables were selected for configuration. By analyzing the average daily record difference set and element difference set of the main and backup centers, the record difference set is not large, which means that the data integrity of the main and backup centers is relatively good, but the element differences are about 3% and 5% respectively, and there are certain problems in consistency. Therefore, the method of this application was used to update the element difference records of about 5%, as shown in Table 1. The data consistency of the main and backup centers is effectively guaranteed.

[0084] Table 1

[0085]

[0086] Compared with the comparison of single elements, the comparison of consistency differences of data elements based on the hash codes of data table records greatly improves the comparison efficiency and reduces computing resources.

[0087] For example, in China's hourly ground data table, 205 fields are compared for consistency, which improves performance by 18 times compared to comparing each element one by one.

[0088] like Figure 2 For comparison with provincial data, Figure 2 The blue bar chart on the left is the data of the national main center, and the blue bar on the right is the data of the backup center. The method of this application is used for comparison, and the data with differences are updated.

[0089] Obviously, the above examples are only examples for clear explanation, and are not limitations on the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from this are still within the scope of protection of the claims of this patent application.

Claims

1. A method for checking and processing data consistency of automatic weather stations in two locations and three centers, characterized in that: The national data center in Beijing is the main center, and two backup centers are set up in Beijing and Xi'an respectively, including the following steps: (1) Construct a HashMap collection of the main center and a HashMap collection of the backup center for the automatic station data stored in the main center and the backup center respectively; (2) Compare and check the HashMap set of the primary center and the HashMap set of the backup center to obtain the record difference set and the element difference set; (3) Process the record difference set and the element difference set.

2. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 1, characterized in that: In step (1), the automatic station data is preprocessed: when storing the automatic station data, the automatic station number and the observation time are used as the joint primary key to uniquely identify a row of data records R.

3. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 2, characterized in that: For each data record R of the primary center and the backup center, generate a joint primary key string set PK(R) and a comparison field string set V(R) respectively; PK(R) = Concatenate(Select(Fk, R)) (1); V(R) = Concatenate(Select(Fc, R)) (2); In formula (1) and formula (2), Select(F,R) is the field value of the data field set F in the data record R retrieved according to the agreed format, Fk represents the joint primary key field set, Fc represents the comparison field set, and Concatenate connects the field values ​​into a string set according to the agreed rules.

4. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 3, characterized in that: Perform MD5 encryption calculation on the comparison field string set V(R) of the primary center and the backup center, so as to form a Hashmap structure with the data of the comparison time period: MD5_Hash(R)=MD5(V(R)); MD5(s): represents the function of performing MD5 encryption calculation on string s.

5. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 4, characterized in that: For the two data sources of the primary center and the backup center, construct HashMap collections respectively: H_m={(PK(R_m),MD5_Hash(R_m))|R_m∈main center data source}; H_b={(PK(R_b),MD5_Hash(R_b))|R_b∈central data source}; H_m represents the HashMap collection constructed by the main center data source, and m represents the main center system; H_b represents the HashMap collection constructed by the backup center data source, and b represents the backup center system.

6. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 5, characterized in that: In step (2), the key values ​​of the HashMap collections of the primary center and the backup center are traversed to obtain the record difference set C: C={PK(R)|PK(R)∈(H_m.keys()-H_b.keys())}; C represents the set of joint primary keys of missing records in the backup center; keys() represents the key value of the HashMap collection.

7. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 6, characterized in that: Traverse the feature values ​​of the HashMap collections of the primary center and the backup center, check and compare the MD5 hash values ​​of the comparison field string collections of the records with the same joint primary key, and obtain the feature difference collection D: D={PK(R)|(PK(R)∈H_m.keys()∩H_b.keys())∧(H_m[PK(R)]≠H_b[PK(R)])}.

8. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 7, characterized in that: In step (3), based on the difference comparison method, the data information of the inconsistency between the main and backup centers is obtained, that is, the difference set and the element difference set are recorded, and the data of the backup center is synchronously updated based on the data of the main center; the data processing process gives priority to the backup center at the same location, and when the processing of the backup center at the same location is completed, the backup center at the different location is processed according to the same process.

9. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 8, characterized in that: For the record difference set C, the data records RS(C) missing from the backup center are retrieved from the primary center and written to the backup center.

10. A method for checking and processing data consistency of automatic weather stations in two locations and three centers according to claim 9, characterized in that: For the same joint primary key rows in the primary and backup centers, that is, for the element difference set D, there are records RS(D) with inconsistent element values, first back up the difference record RS_b(D) in the backup center, and then use RS_m(D) in the primary center to synchronously update RS_b(D) in the backup center.

Images