Real estate registration data synchronization method based on seatunnel

Abstract

The invention discloses a real estate registration data synchronization method based on seatunnel, which comprises the following steps of: data source access: defining a real estate registration data source library as BDS (S1, S2, S3... Sn), defining a data target library as BDT (T1, T2, T3... Tn), setting database addresses of the BDS and the BDT, a complete link string, user information and driving information, and performing data source access by testing connectivity; the method is applied to data source management of real estate registration data synchronization. And task information configuration: setting a data synchronization task DST (D1, D2, D3... Dn), accessing a data source library S1 and a data target library T1 in a D1 configuration file, and appointing real estate registration data source tables (st1, st2, st3... stn) and result tables (rt1, rt2, rt3, rtn). According to the real estate registration data synchronization method based on the seatunnel, the problem of data real-time intercommunication of cross-platform, multi-network-segment and multi-source heterogeneous databases is solved through data source access plug-in development.

Description

Technical Field

[0001] The present invention belongs to the technical field of data synchronization methods, and in particular relates to a real estate registration data synchronization method based on seatunnel. Background Art

[0002] With the continuous advancement of real estate digitalization, the requirements for data security, integrity and stability are becoming increasingly stringent. In this context, current real estate data synchronization faces many key issues, as follows:

[0003] Cross-segment synchronization issues in a three-network separation environment: To enhance the security of the system and data, real estate information system builders actively promote the separation of the Internet, government extranet, and business intranet. In this mode, business data is stored in the government extranet, and highly confidential information is stored in the intranet, with data flow achieved through a one-way network gate. Although this approach improves security, it seriously affects the efficient exchange of information.

[0004] Database migration issues under the background of localization: Under the background of localization, it is urgent to migrate traditional commercial databases to domestic commercial databases. In this process, data integrity and business continuity must be ensured, while performance and compatibility issues must be fully considered.

[0005] Requirements for cadastral database construction: As an important part of the basic information platform for national land space, the cadastral database mainly stores cadastral data such as cadastral survey results and title registration results. Its construction requires the realization of cross-level data sharing and interconnection with related business systems.

[0006] Submission and collection requirements: The submission and collection of real estate registration data is the key to ensuring data consistency and accuracy. By establishing a data consistency comparison mechanism, incremental comparison and full database comparison between the real estate registration database and the real estate registration database of the subordinate districts can be achieved, thereby solving the problem of data inconsistency.

[0007] Data sharing requirements: In order to improve the efficiency of real estate registration and achieve the goal of "face recognition for registration" and "instant access" for real estate registration in the entire district, it is necessary to develop a shared exchange standardized service interface, complete the connection with the district government big data service platform, and automatically obtain shared information.

[0008] Decision analysis and push to analytical database requirements: To support decision analysis, data needs to be pushed to analytical databases. This requires data synchronization solutions to support real-time synchronization of large-scale data to ensure the timeliness and effectiveness of data.

[0009] By applying SeaTunnel technology, real estate data can be synchronized, effectively solving the above problems and providing strong data support for the digital construction of real estate. Summary of the invention

[0010] The present invention aims to provide a real estate registration data synchronization method based on seatunnel to solve the technical problems raised in the background technology.

[0011] To achieve the above purpose, the specific technical solution of the present invention is as follows: A real estate registration data synchronization method based on seatunnel, comprising the following steps:

[0012] Data source access: define the real estate registration data source library as BDS (S1, S2, S3...Sn), define the data target library as BDT (T1, T2, T3...Tn), set the database address, complete link string, user information, and driver information of BDS and BDT, and test the connectivity to apply it to the data source management of real estate registration data synchronization;

[0013] Task information configuration: Set the data synchronization task DST (D1, D2, D3...Dn). In the D1 configuration file, connect the data source library S1 and the data target library T1, specify the real estate registration data source table (st1, st2, st3...stn) and result table (rt1, rt2, rt3,,,rtn), mark the mapping relationship between the source field (sf1, sf2, sf3...sfn) and the result field (tf1, tf2, tf3...tfn), set up the filtering conditions (F1, F2, F3...Fn), and apply it to the task management of creating real estate registration data synchronization;

[0014] Data quality inspection and processing: Sort out and set the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn), package them into a JAR file through the business rule engine, and publish them as an optional plug-in to the SeaTunnel plug-in directory. When executing the data synchronization task DST, after executing the general filtering conditions (F1, F2, F3...Fn), call the business rule engine to execute the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn) to monitor and standardize data quality;

[0015] Multi-threaded parallel synchronization: execute data synchronization task D1, enable multi-threaded parallel synchronization in the background, divide the data in the data source table st1 into shards DS1, DS2, DS3...DSn, allocate idle and stable thread resources to each data shard, run the filtering conditions (F1, F2, F3...Fn), and write to the result table rt1 in real time and direction according to the field mapping relationship;

[0016] Abnormal status monitoring and recovery: Focusing on the entire life cycle of real estate registration data synchronization, including reading data source library information, converting data table fields, and writing data to the target library, we agree on the checkpoint CP1 of the Source enumerator, the checkpoint CP2 of the SourceReader, the checkpoint CP3 of the Sink Writer, the checkpoint CP4 of the Sink Writer, and the checkpoint CPn of other events. When running the data synchronization task DST, the checkpoint mechanism is triggered regularly to save the data processing status of each checkpoint to ensure that the last successful checkpoint can be recovered in the event of a failure, thereby ensuring data consistency.

[0017] Preferably, in the data source access step, the data source library and the data target library include a system and data environment built based on Windows and Linux operating systems in a non-domestic environment, using Oracle and MySQL databases, as well as a system and environment built based on the domestic operating systems of Kylin and Tongxin uos in a domestic environment, using Renmin University of China Jincang, DAMO and Hangao domestic databases.

[0018] Preferably, in the task information configuration step, the filtering conditions (F1, F2, F3...Fn) include one or more of classification by district and county code, classification by registration time, and classification by registration type.

[0019] Preferably, in the data quality inspection processing step, the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn) include the district and county code is not empty, the registration time is in the format of year, month, day, hour, minute, and second and is not empty, and whether the registration type complies with one or more of the national standard enumeration values.

[0020] The real estate registration data synchronization method based on seatunnel of the present invention has the following advantages:

[0021] 1. This real estate registration data synchronization method based on seatunnel solves the problem of real-time data intercommunication across platforms, multiple network segments, and multi-source heterogeneous databases through data source access plug-in development.

[0022] 2. This real estate registration data synchronization method based on seatunnel solves the timeliness and consistency issues of data sharing in different business platforms and at different stages of business handling through a real-time data synchronization process, while improving the business handling efficiency of government staff.

[0023] 3. This real estate registration data synchronization method based on seatunnel effectively utilizes system resources through dynamic thread allocation and shard data synchronization, and solves the problem of efficient and stable synchronization of massive data.

[0024] 4. This real estate registration data synchronization method based on seatunnel introduces a business rule verification engine through data quality inspection, checks data integrity and compliance from the business dimension, and performs standardized batch processing for common problems, thereby ensuring the data integrity and compliance of real estate registration business, reducing the risk of erroneous data in the business handling process, and improving user convenience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 This is a task structure diagram of a real estate registration data synchronization method based on seatunnel of the present invention;

[0027] Figure 2 The present invention is a flow chart of real estate registration information subject table synchronization based on seatunnel. DETAILED DESCRIPTION

[0028] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the embodiments of the present invention. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0029] In the description of the embodiments of the present invention, it should be understood that the terms "length", "vertical", "horizontal", "top", "bottom", etc. indicating the orientation or positional relationship are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the embodiments of the present invention.

[0030] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0031] In the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, an electrical connection, or a communication; it can be a direct connection, or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0032] The disclosure below provides many different embodiments or examples to implement different structures of the embodiments of the present invention. In order to simplify the disclosure of the embodiments of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the embodiments of the present invention. In addition, the embodiments of the present invention can repeat reference numbers and / or reference letters in different examples, and this repetition is for the purpose of simplification and clarity, which itself does not indicate the relationship between the various embodiments and / or settings discussed.

[0033] In order to better understand the purpose, structure and function of the present invention, the following is a further detailed description of a real estate registration data synchronization method based on seatunnel of the present invention in conjunction with the accompanying drawings.

[0034] like Figure 1-Figure 2 As shown, a real estate registration data synchronization method based on seatunnel of the present invention comprises the following steps:

[0035] Data source access: define the real estate registration data source library as BDS (S1, S2, S3...Sn), and the data target library as BDT (T1, T2, T3...Tn), set the database address, complete link string, user information, and driver information of BDS and BDT, and apply it to the data source management of real estate registration data synchronization by testing connectivity; in the data source access step, the data source library and data target library include systems and data environments built based on windows and linux operating systems in non-domestic environments, using oracle and mysql databases, as well as systems and environments built based on Zhongbiao Kylin and Tongxin uos domestic operating systems in a domestic environment, using Renmin University of China Jincang, Damang, and Hangao domestic databases.

[0036] Task information configuration: Set the data synchronization task DST (D1, D2, D3...Dn). In the D1 configuration file, connect the data source library S1 and the data target library T1, specify the real estate registration data source table (st1, st2, st3...stn) and result table (rt1, rt2, rt3,,,rtn), mark the mapping relationship between the source field (sf1, sf2, sf3...sfn) and the result field (tf1, tf2, tf3...tfn), set up the filtering conditions (F1, F2, F3...Fn), and apply it to the task management of creating real estate registration data synchronization;

[0037] Data quality inspection processing: sort out and set the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn), package them into a JAR file through the business rule engine, and publish them as an optional plug-in to the SeaTunnel plug-in directory. When executing the data synchronization task DST, after executing the general filtering conditions (F1, F2, F3...Fn), call the business rule engine to execute the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn) to monitor and standardize the data quality; in the data quality inspection processing step, the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn) include one or more of the district and county code is not empty, the registration time is in the year-month-day hour-minute-second format and is not empty, and whether the registration type conforms to the national standard enumeration value.

[0038] Multi-threaded parallel synchronization: execute data synchronization task D1, enable multi-threaded parallel synchronization in the background, divide the data in the data source table st1 into shards DS1, DS2, DS3...DSn, allocate idle and stable thread resources to each data shard, run the filtering conditions (F1, F2, F3...Fn), and write to the result table rt1 in real time according to the field mapping relationship; in the task information configuration step, the filtering conditions (F1, F2, F3...Fn) include one or more of classification by district and county code, classification by registration time, and classification by registration type.

[0039] Abnormal status monitoring and recovery: Focusing on the entire life cycle of real estate registration data synchronization, including reading data source library information, converting data table fields, and writing data to the target library, we agree on the checkpoint CP1 of the Source enumerator, the checkpoint CP2 of the SourceReader, the checkpoint CP3 of the Sink Writer, the checkpoint CP4 of the Sink Writer, and the checkpoint CPn of other events. When running the data synchronization task DST, the checkpoint mechanism is triggered regularly to save the data processing status of each checkpoint to ensure that the last successful checkpoint can be recovered in the event of a failure, thereby ensuring data consistency.

[0040] Furthermore, non-domestic environment: refers to the construction of system and data environment based on Windows and Linux operating systems, using Oracle, MySQL database and other resources. Domestic environment: refers to the construction of system and environment based on domestic operating systems such as Zhongbiao Kylin and Tongxin UOS, using domestic database resources such as Renmin University of China Jincang, Damang, and Hangao. Separation of three networks in real estate: the three networks refer to the government intranet, government extranet and the Internet, requiring physical isolation between the government intranet and the government extranet, and logical isolation between the government extranet and the Internet. Multi-source heterogeneity: refers to data coming from multiple channels such as different databases, file systems, etc.; and the data formats are diverse.

[0041] Configure the data source. Figure 2 As shown, the database address, complete link string, user information, and driver information of the data source library S1 (BDCDJ) are configured; the database address, complete link string, user information, and driver information of the data target library T1 (BDCFX) are configured.

[0042] Create a new real estate registration information subject table synchronization task.

[0043] 1) Access the configured data sources S1 and T1; 2) Specify the data source table (BDCDJ.Fact_YWZTXX) and result table (BDCFX.Fact_YWZTXX_1, BDCFX.Fact_YWZTXX_2...); 3) Figure 1 As shown, establish the field mapping relationship between the data source table and the result table; 4) set up filtering conditions, including F1: classification by district and county code, F2: classification by registration time, F3: classification by registration type....

[0044] Build quality inspection rules. 1) Business rule sorting. According to the needs of data synchronization and subsequent business analysis, sort out business rules such as "district and county code is not empty, registration time is in the format of year, month, day, hour, minute, and second, and is not empty, whether the registration type conforms to the national standard enumeration value..."; 2) Create new system quality inspection rules. Through rule code development, form system rules Q1, Q2, Q3...; 3) Introduce rules into the rule engine for unified system management.

[0045] Multi-threaded parallel synchronization, 1) Data sharding. Extract the data of BDCDJ.Fact_YWZTXX and perform data sharding; 2) Thread allocation. Allocate system threads for the synchronization service corresponding to the sharded data; 3) Run the filter conditions F1, F2, and F3 to classify the data according to the administrative district code, registration time, and registration type; 4) Run the quality inspection rules Q1, Q2, and Q3 to eliminate the data with empty district and county codes, empty registration time, and registration types that do not meet the national standard enumeration values; 5) According to the table and field mapping relationship, write the data to BDCFX.Fact_YWZTXX_1, BDCFX.Fact_YWZTXX_2....

[0046] Abnormal status monitoring and recovery. Focusing on the entire life cycle of data source library information reading, data table field conversion, and data target library writing involved in real estate registration data synchronization, the checkpoint CP1 of the Source enumerator, the checkpoint CP2 of the SourceReader, the checkpoint CP3 of the Sink Writer, the checkpoint CP4 of the Sink Writer, and the checkpoint CPn of other events are agreed upon. When running the data synchronization task DST, the checkpoint mechanism is triggered regularly to save the data processing status of each checkpoint, ensuring that it can be recovered from the last successful checkpoint in the event of a failure, thereby ensuring data consistency.

[0047] It is to be understood that the present invention is described by some embodiments, and it is known to those skilled in the art that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A real estate registration data synchronization method based on seatunnel, characterized by: The steps include: Data source access: define the real estate registration data source library as BDS (S1, S2, S3...Sn), define the data target library as BDT (T1, T2, T3...Tn), set the database address, complete link string, user information, and driver information of BDS and BDT, and test the connectivity to apply it to the data source management of real estate registration data synchronization; Task information configuration: Set the data synchronization task DST (D1, D2, D3...Dn). In the D1 configuration file, connect the data source library S1 and the data target library T1, specify the real estate registration data source table (st1, st2, st3...stn) and result table (rt1, rt2, rt3,,,rtn), mark the mapping relationship between the source field (sf1, sf2, sf3...sfn) and the result field (tf1, tf2, tf3...tfn), set up the filtering conditions (F1, F2, F3...Fn), and apply it to the task management of creating real estate registration data synchronization; Data quality inspection and processing: Sort out and set the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn), package them into a JAR file through the business rule engine, and publish them as an optional plug-in to the SeaTunnel plug-in directory. When executing the data synchronization task DST, after executing the general filtering conditions (F1, F2, F3...Fn), call the business rule engine to execute the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn) to monitor and standardize data quality; Multi-threaded parallel synchronization: execute data synchronization task D1, enable multi-threaded parallel synchronization in the background, divide the data in the data source table st1 into shards DS1, DS2, DS3...DSn, allocate idle and stable thread resources to each data shard, run the filtering conditions (F1, F2, F3...Fn), and write to the result table rt1 in real time and direction according to the field mapping relationship; Abnormal status monitoring and recovery: Focusing on the entire life cycle of real estate registration data synchronization, including reading data source library information, converting data table fields, and writing data to the target library, we agree on the checkpoint CP1 of the Source enumerator, the checkpoint CP2 of the SourceReader, the checkpoint CP3 of the Sink Writer, the checkpoint CP4 of the Sink Writer, and the checkpoint CPn of other events. When running the data synchronization task DST, the checkpoint mechanism is triggered regularly to save the data processing status of each checkpoint to ensure that the last successful checkpoint can be recovered in the event of a failure, thereby ensuring data consistency.

2. According to claim 1, a real estate registration data synchronization method based on seatunnel is characterized in that: In the data source access step, the data source library and data target library include a system and data environment built based on Windows and Linux operating systems in a non-domestic environment, using Oracle and MySQL databases, as well as a system and environment built based on the domestic operating systems of Kylin and Tongxin uos in a domestic environment, using Renmin University of China Jincang, Damang and Hangao domestic databases.

3. The real estate registration data synchronization method based on seatunnel according to claim 1 is characterized in that: In the task information configuration step, the filtering conditions (F1, F2, F3...Fn) include one or more of classification by district and county code, classification by registration time, and classification by registration type.

4. The method for synchronizing real estate registration data based on seatunnel according to claim 1, characterized in that: In the data quality inspection processing step, the real estate registration data quality inspection rules BQCR (Q1, Q2, Q3...Qn) include that the district and county code is not empty, the registration time is in the format of year, month, day, hour, minute, and second and is not empty, and whether the registration type complies with one or more of the national standard enumeration values.

Images