Data synchronization method and device, equipment and storage medium

Abstract

The embodiment of the invention relates to the technical field of big data processing, and discloses a data synchronization method, device and equipment and a storage medium, and the method comprises the steps: collecting a data synchronization demand of each branch mechanism, carrying out the multi-stage splitting of the data synchronization demand according to a plurality of predetermined dimension levels, and obtaining a plurality of atomic demands, the multiple dimension levels comprise a data system level, a database level and a data table level; formulating a data synchronization task based on the atomic demand, and packaging the synchronization task of each branch mechanism into a data synchronization plan; a first data center corresponding to each branch mechanism is determined from a plurality of preset data centers, and the data center closest to the branch mechanism is the first data center of the branch mechanism; and based on the data synchronization plan of each branch mechanism, synchronizing the data from the database of the first data center to the database of the corresponding branch mechanism. Through the above mode, the embodiment of the invention can efficiently and timely synchronize the data.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of big data processing technology, and specifically to a data synchronization method, device, equipment and storage medium. Background Art

[0002] Data synchronization refers to the process of copying data from one database to another between two or more databases to maintain the consistency and synchronization of data between databases. For enterprises including headquarters and multiple branches, data synchronization is often required frequently so that the party in need can obtain the corresponding data for consumption.

[0003] At present, data synchronization technology mainly relies on centralized architecture and regular data replication technology. For example, when multiple branches synchronize data with the headquarters, they need to first download the data from the headquarters server to the branch or sales department, clean the data using local tools, and then import it into the local database. The data can only be used after it is imported into the database. This method is cumbersome and inefficient, especially when the data files are large. Cross-regional transmission between the headquarters and multiple branches consumes more bandwidth, which can easily lead to slow transmission rates and failure to synchronize data in a timely manner. Summary of the invention

[0004] In view of the above problems, embodiments of the present invention provide a data synchronization method, apparatus, device and storage medium, which are used to solve the problems of low efficiency and untimely data synchronization in the prior art.

[0005] According to one aspect of an embodiment of the present invention, there is provided a data synchronization method, comprising:

[0006] Collecting data synchronization requirements of various branches, splitting the data synchronization requirements into multiple levels according to a plurality of predetermined dimensional levels, and obtaining a plurality of atomic requirements, wherein the plurality of dimensional levels include a data system level, a database level, and a data table level;

[0007] Formulate data synchronization tasks based on the atomic requirements, and package the synchronization tasks of each branch into a data synchronization plan;

[0008] Determine a first data center corresponding to each branch from a plurality of predetermined data centers, wherein the data center closest to the branch is the first data center of the branch;

[0009] Based on the data synchronization plan of each of the branches, the data is synchronized from the database of the first data center to the database of the corresponding branch.

[0010] In an optional manner, after formulating the data synchronization task based on the atomic requirement, the method further includes:

[0011] Based on the atomic demand, determining first table structure information of a plurality of first data tables from a database of the first data center;

[0012] Based on the atomic requirement, obtaining second table structure information of a plurality of pre-stored second data tables;

[0013] Comparing the first table structure information with the second table structure information, and determining whether to perform standardization processing on the data tables of each of the branches according to the comparison result, wherein the standardization processing includes one of table creation and table structure update;

[0014] After packaging the synchronization tasks of each branch into a data synchronization plan, the method further includes:

[0015] The operation status of the database of each branch office is determined, and when the operation status of the database of each branch office is a normal operation status, a data synchronization operation is performed.

[0016] In an optional manner, after formulating the data synchronization task based on the atomic requirement, the method further includes:

[0017] Configuring multiple task nodes for the data synchronization task, and monitoring the data synchronization task based on the multiple task nodes;

[0018] After packaging the synchronization tasks of each branch into a data synchronization plan, the method further includes:

[0019] A plurality of plan nodes of the data synchronization plan are configured, and the data synchronization plan is monitored based on the plurality of plan nodes.

[0020] In an optional manner, the method further includes:

[0021] Generate execution logs of the multiple task nodes and the multiple plan nodes, and output the execution logs visually.

[0022] In an optional manner, the method further includes:

[0023] During the data synchronization process, monitoring whether the computer room of the first data center has any abnormality;

[0024] If so, the first data center is switched to the second data center to synchronize data from the second data center to the database of the corresponding branch of the first data center, wherein the data center that is next closest to the corresponding branch is the second data center of the corresponding branch.

[0025] In an optional manner, the method further includes:

[0026] During the data synchronization process, extracting an operation log of the database of the first data center, and determining a data change operation of the database of the first data center based on the operation log;

[0027] If the data changed by the data change operation is currently synchronized data, a change message corresponding to the data change operation is generated, and the change message is sent to the database of the corresponding branch.

[0028] In an optional manner, the method further includes:

[0029] During the data synchronization process, an extraction process is started, data change information of the database of the first data center is extracted based on the extraction process, and the data change information is written into a track file;

[0030] A delivery process is started, and the trajectory file is sent to a database of a corresponding branch based on the delivery process.

[0031] According to another aspect of an embodiment of the present invention, there is provided a data synchronization device, comprising:

[0032] A splitting module is used to collect data synchronization requirements of various branches, and split the data synchronization requirements into multiple levels according to a predetermined multiple dimensional levels to obtain multiple atomic requirements, wherein the multiple dimensional levels include a data system level, a database level, and a data table level;

[0033] A formulation module, used to formulate data synchronization tasks based on the atomic requirements, and to package the synchronization tasks of each branch into a data synchronization plan;

[0034] A determination module, configured to determine a first data center corresponding to each of the branch offices from a plurality of predetermined data centers, wherein the data center closest to the branch office is the first data center of the branch office;

[0035] The synchronization module is used to synchronize data from the database of the first data center to the database of the corresponding branch office based on the data synchronization plan of each branch office.

[0036] According to another aspect of an embodiment of the present invention, there is provided a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0037] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method as described above.

[0038] According to another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and when the executable instruction is executed on a computer device, the computer device executes the method described above.

[0039] The embodiment of the present invention only needs to perform relevant operations in the data center of the head office, and there is no need to perform any operations in the branch office. By splitting data synchronization requirements, formulating tasks, packaging plans, and determining the nearest data center in the head office, multiple branch offices can synchronize the data of the head office simultaneously and orderly, simplifying data synchronization operations, making data synchronization efficient, and being able to synchronize data in a timely manner.

[0040] The above description is only an overview of the technical solution of the embodiment of the present invention. In order to more clearly understand the technical means of the embodiment of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiment of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0042] Figure 1 A schematic diagram of an environment in which a data synchronization method according to an embodiment of the present invention is used is shown;

[0043] Figure 2 A schematic diagram showing a flow chart of a data synchronization method provided by an embodiment of the present invention;

[0044] Figure 3 A schematic diagram showing a configuration task node in a data synchronization method provided by an embodiment of the present invention is shown;

[0045] Figure 4 A schematic diagram showing a configuration plan node in a data synchronization method provided by an embodiment of the present invention is shown;

[0046] Figure 5 A schematic diagram showing the structure of a data synchronization device provided by an embodiment of the present invention is shown;

[0047] Figure 6 A schematic diagram of the structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0048] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0049] Figure 1 A schematic diagram of the use environment of the data synchronization method provided by an embodiment of the present invention is shown, wherein a head office is deployed with multiple data centers, wherein each data center includes a computer room, a server and other related equipment (for example, monitoring equipment and security equipment, etc.), and the multiple data centers can be deployed in the same city, or some in the same city and some in different locations.

[0050] like Figure 1 As shown, the data synchronization service control center and the data synchronization operation center are deployed on the server, and the corresponding data storage devices can also be deployed on the server. In the data synchronization service control center, various types of databases are uniformly managed, including databases gaussdb, tdsql, standard databases SQLSERVER and ORACLE, and confidential data will be encrypted and stored. The data synchronization service control center is used to collect data synchronization requirements of each branch, split the data synchronization requirements into multiple atomic requirements, formulate data synchronization tasks corresponding to the atomic requirements, package the synchronization tasks of each branch into a data synchronization plan, and send the data synchronization plan to the data synchronization operation center. The data synchronization operation center obtains the corresponding data from the database of the data storage device according to the data synchronization plan, and synchronizes the data to the database of each corresponding branch or business department, so that the consumption platform of the branch or business department can obtain data from the corresponding database for consumption.

[0051] In the entire data synchronization of the embodiment of the present invention, only relevant operations need to be performed in the data center of the head office, including the collection and splitting of data synchronization requirements of each branch office, formulation of synchronization tasks, packaging of synchronization plans, determination of the nearest data center and synchronization of data. No operations need to be performed in the branch offices. By performing the splitting of synchronization requirements, formulation of tasks, packaging of plans and determination of the nearest data center in the head office, multiple branches can synchronize the data of the head office simultaneously and orderly, simplifying data synchronization operations, making data synchronization efficient, and being able to synchronize data in a timely manner.

[0052] Figure 2 A flow chart of a data synchronization method according to an embodiment of the present invention is shown. Figure 2 As shown, the method comprises the following steps:

[0053] Step 101: Collect data synchronization requirements of various branches, and split the data synchronization requirements into multiple levels according to multiple predetermined dimensional levels to obtain multiple atomic requirements.

[0054] In this embodiment, the branch includes the branch itself and the sales department, both of which belong to the front line of the business. When multiple branches have data needs, the data synchronization service control center of the head office collects the data synchronization needs of the branches, and then splits the data synchronization needs into multiple levels according to the predetermined multiple dimensional levels to obtain multiple atomic needs.

[0055] In this embodiment, multiple data systems are deployed in each data center, each data system includes multiple databases, and each database includes multiple data tables. The dimension level includes the data system level, the database level, and the data table level. Therefore, according to the order of the dimension level, the data synchronization requirements of each branch are first split at the system level to determine the target system where the data is to be synchronized, and then the data synchronization requirements are split at the database level to determine the corresponding target database in the target system, and finally the data synchronization requirements are split at the data table level to determine the target system, the corresponding target database, and the corresponding data table in the target database of the data synchronization requirements.

[0056] For example, when a branch synchronizes private data belonging to the branch from the head office, such as customer portrait data, it can be seen through analysis that the customer portrait data includes customer transaction data, capital share data, customer panoramic information, and customer suitability data. These data are stored in different data systems of the head office. Therefore, the customer portrait data synchronization requirements are split into multiple levels according to the data system level, database level, and data table level. After the split, the following atomic requirements are obtained:

[0057] Atomic requirement 1: Extract the client fund share data of the headquarters according to the branch / business department code from the fundasset data table of the trading system database;

[0058] Atomic requirement 2: Extract customer transaction data of the headquarters according to branch / business department code from the data table trdasset of the transaction system database;

[0059] Atomic requirement 3: Extract the panoramic data of the headquarters’ customers from the customer table in the account system database according to the branch / business department code;

[0060] Atomic requirement 4: Extract the suitability data of the headquarters’ customers according to the branch / business department code from the surveyRating data table in the account system database;

[0061] Atomic requirement 5: Extract the customer account opening records and business handling data details of the headquarters according to the branch / business department code from the data table opp_busi_data of the centralized operation platform database;

[0062] Therefore, the above customer portrait data synchronization requirements are split into 5 atomic requirements, and data is extracted from 5 data tables in 3 databases according to the branch / business department code.

[0063] Step 102: Formulate data synchronization tasks based on the atomic requirements, and package the synchronization tasks of each branch into a data synchronization plan.

[0064] In this embodiment, a data synchronization task is formulated for each atomic requirement, and one or more data synchronization tasks are formulated for one data synchronization requirement. In this way, personalized synchronization tasks can be formulated for the data synchronization requirements of branches, without the need to synchronize the entire data table to the branches. After formulating the data synchronization tasks, each data synchronization task is visualized, and the task information of each data synchronization task can be viewed on the page. The task information includes task number, task type, task name, task description, and formulation time.

[0065] In addition, all data synchronization tasks of each branch are packaged into a data synchronization plan for the branch. Each data synchronization plan can be triggered according to actual needs, for example, triggered at a predetermined time to achieve data synchronization. After packaging the data synchronization plan, the data synchronization plan is visualized, and the plan information of each data synchronization plan can be viewed on the page. The plan information includes the plan number, plan name, plan description, execution mode, and update time, etc. In addition, each data synchronization plan on the page is also configured with an operation component, which includes immediate execution and view administrator. By clicking the immediate execution component, the execution of the data synchronization plan can be triggered.

[0066] In an optional embodiment, after formulating the data synchronization task based on the atomic requirement, it also includes: based on the atomic requirement, determining the first table structure information of multiple first data tables from the database of the first data center; based on the atomic requirement, obtaining the second table structure information of multiple pre-stored second data tables; comparing the first table structure information with the second table structure information, and determining whether to standardize the data tables of each of the branches based on the comparison result, wherein the standardization includes one of table creation and table structure update.

[0067] In this embodiment, the second table structure information of the plurality of pre-stored second data tables is the table structure information of the data tables of the branch offices. The branch offices can upload the table structure information of the data tables to the data synchronization service control center in real time or at a fixed time, or the data synchronization service control center can collect all the table structure information and other information of the data tables of the branch offices locally, and store them in the data storage device of the first data center, and perform visual management, so that the basic information, table structure information, index information, dependency relationship, etc. of the tables can be viewed. Generally speaking, the structure information of the data tables of the database of the first data center as the data source may change according to actual needs. By comparing the table structure information of the first data table from the database of the first data center with the second data table from the branch offices, it is possible to determine which data structure information of the first data center has changed. If the table structure information of the first data table of the database of the first data center has changed, an instruction is sent to the branch offices to timely instruct the corresponding data tables of the branch offices to perform standardization processing, that is, to create tables or update table structures, so as to ensure the consistency of the data structure and prevent data import failure or data anomalies.

[0068] In an optional embodiment, after packaging the synchronization tasks of each branch into a data synchronization plan, it also includes: determining the operating status of the database of each branch, and when the operating status of the database of each branch is a normal operating status, performing a data synchronization operation.

[0069] In this embodiment, first, all the link information and table structure information of the branch office are collected to the data synchronization service control center for visual management, and the link information is used to meet the requirements of the end-to-end data synchronization status check of the data synchronization service. Specifically, the data synchronization service control center connects to the database of the branch office through the link information, and then obtains the operating status of the database of the branch office. The operating status includes normal operation, offline status, suspended status, etc., and checks to confirm whether the database of the branch office is in a normal operating state. In its normal operating state, data synchronization operations can be performed, while in other states, data synchronization operations cannot be performed, thereby ensuring that the data synchronization operation can be performed normally.

[0070] In an optional embodiment, after formulating the data synchronization task based on the atomic requirement, it also includes: configuring multiple task nodes of the data synchronization task, and monitoring the data synchronization task based on the multiple task nodes.

[0071] In this embodiment, for each data synchronization task, a corresponding task node is configured to monitor each data synchronization task throughout the process, such as Figure 3As shown, the configured task nodes include 7 task nodes, including whether to create a table for the branch, clear the table structure, create a data table, extract the data table, write synchronization pipeline, write data to the branch, and update the synchronization pipeline status. Each task node is monitored for its successful execution, and a corresponding log is generated for each task node to determine whether each data synchronization task can be executed normally.

[0072] Optionally, generate execution logs for multiple task nodes and output the execution logs visually. The log number, time level, log information, and exception information can be viewed on the page. The log information records detailed synchronization information.

[0073] In an optional embodiment, after packaging the synchronization tasks of each branch into a data synchronization plan, the method further includes: configuring multiple plan nodes of the data synchronization plan, and monitoring the data synchronization plan based on the multiple plan nodes.

[0074] In this embodiment, for each data synchronization plan, a corresponding plan node is configured, such as Figure 4 As shown, the configured plan nodes include branch synchronization preparation (e.g., whether the operation status of the branch database is normal), branch table 1-n synchronization, and branch synchronization repair. Each plan node is monitored and a corresponding log is generated for each plan node to determine whether each data synchronization plan can be executed normally.

[0075] Optionally, generate execution logs for multiple plan nodes and output the execution logs visually. The execution logs for multiple plan nodes of each data synchronization plan are displayed on the page. The execution log records the name, execution status, execution result (execution success, execution failure), execution parameters, execution time (start time, end time) and exception information of the data synchronization plan. You can also view the plan flow chart of each data synchronization plan and the data synchronization tasks it contains, the task flow chart of each data synchronization task and the task log on this page to achieve full-link monitoring and visual management.

[0076] Step 103: determining a first data center corresponding to each branch from a plurality of predetermined data centers, wherein the data center closest to the branch is the first data center of the branch;

[0077] In this embodiment, the data center closest to the branch office is determined from multiple data centers to perform data synchronization. For example, a data center in the same city is selected as the first data center of the branch office, so that the first data center corresponding to each branch office is not necessarily the same. Selecting a data center that is close to perform data synchronization can reduce the delay of data transmission and improve the efficiency and real-time performance of data synchronization.

[0078] Step 104: Based on the data synchronization plan of each of the branches, synchronize the data from the database of the first data center to the database of the corresponding branch.

[0079] In this embodiment, the first data center corresponding to each branch is the closest data center. Based on the data synchronization plan of each branch, data is synchronized from one or more data centers to the database of the corresponding branch, so that data can be synchronized in batches from the head office to each branch according to one or more data synchronization plans.

[0080] Optionally, in this embodiment, according to the subscription mode of the branch or business department, the data in each data table is extracted individually or in batches according to the code of the branch or business department, and the data is written into the database table of the subscriber.

[0081] In this embodiment, for each data synchronization task, after the data synchronization of the task is completed, the synchronization status of the data synchronization task is updated to completed; for each data synchronization plan, after the data synchronization of the plan is completed, the synchronization status of the data synchronization plan is updated to completed.

[0082] In an optional embodiment, during the data synchronization process, the data synchronization method also includes the following steps: monitoring whether the computer room of the first data center has any abnormalities; if so, switching from the first data center to the second data center to synchronize data from the second data center to the database of the corresponding branch of the first data center, wherein the data center that is next closest to the corresponding branch is the second data center of the corresponding branch.

[0083] Among them, whether the computer room of the first data center has abnormal conditions includes network failure, hardware failure and security issues. Network failure includes failure to connect or inability to connect of network devices such as routers and switches. Hardware failure includes hard disk failure, memory failure and power failure. When the device cannot start normally or cannot read data, it is generally a hardware failure. Security issues include illegal intrusion or virus attack. Of course, the abnormality of the computer room also includes other aspects, such as abnormalities caused by environmental factors such as temperature and humidity, disaster events and human errors, etc., which are not limited here.

[0084] In the process of synchronizing data from one or more first data centers to corresponding branches, when it is detected that an abnormality occurs in the computer room of the first data center that is undergoing data synchronization, the first data center initiates a switching instruction, and the data synchronization operation is promptly switched from the first data center to the second data center. The second data center is the data center that is the second closest to the corresponding branch, and can be a data center in the same city or a different city. The purpose of selecting the second data center that is the second closest to perform data synchronization is also to reduce the delay in data transmission and to improve the efficiency and real-time performance of data synchronization.

[0085] In an optional embodiment, during the data synchronization process, the data synchronization method also includes the following steps: during the data synchronization process, extracting the operation log of the database of the first data center, and determining the data change operation of the database of the first data center based on the operation log; if the data changed by the data change operation is the currently synchronized data, then generating a change message corresponding to the data change operation, and sending the change message to the database of the corresponding branch.

[0086] In this embodiment, during the data synchronization process, changes made to the data in the database of the first data center will be promptly identified and captured, and the changes include data insertion, updating, and deletion. When the database completes the change of a data table, these changes will be recorded in the order in which they occurred and transmitted to the database of the branch office. This data change method of this embodiment supports real-time analysis, can synchronize data without stopping, reduces the amount of data transmitted over the network, reduces network bandwidth and storage requirements, is suitable for complex data synchronization, and can provide efficient and low-latency data synchronization.

[0087] Specifically, when the database completes the change of a data table, the change operation is recorded in the operation log of the database. The operation log can be an online log or an archived log. After the operation log is extracted, it is parsed. Optionally, the database dictionary can be used to translate the operation log, and the object ID in the log is translated into an object name, such as a table name, a field name, and a field value. By analyzing the translated content, the data change operation in the database can be determined. If the data changed by the data change operation is the currently synchronized data, a conversion process is performed to generate a unified and standardized change message corresponding to the data change operation. The change message is sent to the database of the branch in the form of a message stream. The message stream includes four message state semantics: insert, update_before, update_after, and delete. Among them, insert represents a newly inserted data record row, update_before represents the data before the data record row is updated, update_after represents the data after the data record row is updated, and delete represents a deleted data record row.

[0088] In an optional embodiment, during the data synchronization process, the data synchronization method also includes the following steps: during the data synchronization process, starting an extraction process, extracting data change information of the database of the first data center based on the extraction process, and converting the data change information into a track file; starting a delivery process, and sending the track file to the database of the corresponding branch based on the delivery process.

[0089] In this embodiment, during the data synchronization process, the first data center will run a management process, and start an extraction process and a delivery process based on the management. The extraction process extracts the source data and captures data changes, and writes the data change information into a track file. The delivery process sends the track file to the database of the corresponding branch. In the specific deployment process, an extraction component and a transmission component are installed in the first data center to extract and transmit the changed data in the database of the first data center, and an application component is installed in the branch to receive and execute the data transmitted from the first data center. This embodiment can capture the changes in the data in the database of the first data center in real time and transmit it to the branch in a timely manner, so that the data of the two are kept synchronized, and data synchronization can be performed without downtime, while providing efficient and low-latency data synchronization.

[0090] Among them, the management process is a control process running in the first data center. In fact, the branch office will also run a control process to control the reception of the above-mentioned track file. The management process is started as a service. The management process can start, monitor and restart other processes, report error events and allocate storage space, etc. The management process starts the extraction process. The extraction process captures data changes from the operation log. The operation log can be an online log or an archive log. The extraction process directly extracts data from the database of the first data center during the initial data loading phase. In the synchronous change capture phase, the data changes are captured through the operation log. Specifically, the extraction process uses its inherent checkpoint mechanism to periodically check and record the read and write locations to capture data changes, and write the data change information into the local track file and cache it. The management process starts the delivery process. The delivery process sends the track file to the branch office in the form of data blocks through the TCP / IP protocol for the branch office's database to synchronize the changed data. The management process of the branch office starts the receiving process, receives the data blocks transmitted by the first data center and reassembles them into a track file, performs remote caching, and then starts the application process, reads the content of the track file and converts it into a JSON format file, parses the JSON format file to obtain a structured query statement, and applies the structured query statement to the database of the branch office, thereby realizing the synchronization of the changed data. In this embodiment, the track file exists in both the first data center and the branch office, and the track file can prevent single point failures, make the information persistent, and use the checkpoint mechanism to record the read and write locations. If a failure occurs, the data can be retransmitted according to the checkpoint record of the read and write locations.

[0091] In an embodiment of the present invention, the data synchronization requirements of each branch are collected, and the data synchronization requirements are multi-level split according to a plurality of predetermined dimensional levels to obtain a plurality of atomic requirements, wherein the plurality of dimensional levels include a data system level, a database level, and a data table level; data synchronization tasks are formulated based on the atomic requirements, and the synchronization tasks of each branch are respectively packaged into a data synchronization plan; the first data center corresponding to each branch is determined from a plurality of predetermined data centers, wherein the data center closest to the branch is the first data center of the branch; based on the data synchronization plan of each branch, data is synchronized from the database of the first data center to the database of the corresponding branch. The embodiment of the present invention only needs to perform relevant operations in the data center of the head office, and no operations are required in the branch office. By splitting the data synchronization requirements, formulating tasks, packaging plans, and determining the closest data center in the head office, multiple branches can synchronize the data of the head office simultaneously and orderly, simplifying the data synchronization operation, making data synchronization efficient, and being able to synchronize data in a timely manner.

[0092] Figure 5 FIG. 2 shows a schematic diagram of the structure of a data synchronization device according to an embodiment of the present invention. Figure 5 As shown, the device 500 includes:

[0093] The splitting module 501 is used to collect data synchronization requirements of each branch, and split the data synchronization requirements into multiple levels according to a predetermined multiple dimensional levels to obtain multiple atomic requirements, wherein the multiple dimensional levels include a data system level, a database level, and a data table level;

[0094] A formulation module 502 is used to formulate data synchronization tasks based on the atomic requirements, and to package the synchronization tasks of each branch into a data synchronization plan;

[0095] A determination module 503 is used to determine a first data center corresponding to each branch from a plurality of predetermined data centers, wherein the data center closest to the branch is the first data center of the branch;

[0096] The synchronization module 504 is used to synchronize data from the database of the first data center to the database of the corresponding branch office based on the data synchronization plan of each branch office.

[0097] The data synchronization device of this embodiment is basically the same as the above-mentioned data synchronization method embodiment, and the above-mentioned data synchronization method embodiment can be referred to, and will not be described in detail here.

[0098] Figure 6 The schematic diagram of the structure of the computer device embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0099] like Figure 6 As shown, the computer device may include: a processor (processor) 602 , a communication interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

[0100] The processor 602, the communication interface 604, and the memory 606 communicate with each other via a communication bus 608. The communication interface 604 is used to communicate with other computer devices such as client devices or other server network elements. The processor 602 is used to execute the program 610, which can specifically execute the relevant steps in the above-mentioned computer device embodiment.

[0101] Specifically, the program 610 may include program code including computer executable instructions.

[0102] The processor 602 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The one or more processors included in the computer device may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0103] The memory 606 is used to store the program 610. The memory 606 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0104] The program 610 may be specifically called by the processor 602 to enable the computer device to perform the following operations:

[0105] Collecting data synchronization requirements of various branches, splitting the data synchronization requirements into multiple levels according to a plurality of predetermined dimensional levels, and obtaining a plurality of atomic requirements, wherein the plurality of dimensional levels include a data system level, a database level, and a data table level;

[0106] Formulate data synchronization tasks based on the atomic requirements, and package the synchronization tasks of each branch into a data synchronization plan;

[0107] Determine a first data center corresponding to each branch from a plurality of predetermined data centers, wherein the data center closest to the branch is the first data center of the branch;

[0108] Based on the data synchronization plan of each of the branches, the data is synchronized from the database of the first data center to the database of the corresponding branch.

[0109] In an optional manner, after formulating the data synchronization task based on the atomic requirement, the method further includes:

[0110] Based on the atomic demand, determining first table structure information of a plurality of first data tables from a database of the first data center;

[0111] Based on the atomic requirement, obtaining second table structure information of a plurality of pre-stored second data tables;

[0112] Comparing the first table structure information with the second table structure information, and determining whether to perform standardization processing on the data tables of each of the branches according to the comparison result, wherein the standardization processing includes one of table creation and table structure update;

[0113] After packaging the synchronization tasks of each branch into a data synchronization plan, the method further includes:

[0114] The operation status of the database of each branch office is determined, and when the operation status of the database of each branch office is a normal operation status, a data synchronization operation is performed.

[0115] In an optional manner, after formulating the data synchronization task based on the atomic requirement, the method further includes:

[0116] Configuring multiple task nodes for the data synchronization task, and monitoring the data synchronization task based on the multiple task nodes;

[0117] After packaging the synchronization tasks of each branch into a data synchronization plan, the method further includes:

[0118] A plurality of plan nodes of the data synchronization plan are configured, and the data synchronization plan is monitored based on the plurality of plan nodes.

[0119] In an optional manner, the method further includes:

[0120] Generate execution logs of the plurality of task nodes and the plurality of plan nodes, and output the execution logs in a visual manner.

[0121] In an optional manner, the method further includes:

[0122] During the data synchronization process, monitoring whether the computer room of the first data center has any abnormality;

[0123] If so, the first data center is switched to the second data center to synchronize data from the second data center to the database of the corresponding branch of the first data center, wherein the data center that is next closest to the corresponding branch is the second data center of the corresponding branch.

[0124] In an optional manner, the method further includes:

[0125] During the data synchronization process, extracting an operation log of the database of the first data center, and determining a data change operation of the database of the first data center based on the operation log;

[0126] If the data changed by the data change operation is currently synchronized data, a change message corresponding to the data change operation is generated, and the change message is sent to the database of the corresponding branch.

[0127] In an optional manner, the method further includes:

[0128] During the data synchronization process, an extraction process is started, data change information of the database of the first data center is extracted based on the extraction process, and the data change information is written into a track file;

[0129] A delivery process is started, and the trajectory file is sent to a database of a corresponding branch based on the delivery process.

[0130] An embodiment of the present invention provides a computer-readable storage medium, wherein the storage medium stores at least one executable instruction. When the executable instruction is executed on a computer device, the computer device executes any of the above method embodiments.

[0131] An embodiment of the present invention provides a computer program, which can be called by a processor to enable a computer device to execute any of the above method embodiments.

[0132] An embodiment of the present invention provides a computer program product, which includes a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed on a computer, the computer is enabled to execute any of the above method embodiments.

[0133] The algorithm or display provided herein is not inherently related to any particular computer, virtual system or other equipment. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description made to specific languages ​​above is for disclosing the best mode of the present invention.

[0134] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0135] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be interpreted as reflecting an intention that the claimed invention requires more features than those expressly recited in each claim.

[0136] Those skilled in the art will appreciate that the modules in the computer device in the embodiment can be adaptively changed and set in one or more computer devices different from the embodiment. The modules or units or components in the embodiment can be combined into one module or unit or component, and they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or computer device disclosed in this manner can be combined in any combination. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0137] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be understood as limitations on the order of execution.

Claims

1. A data synchronization method, characterized in that: The method comprises: Collecting data synchronization requirements of various branches, splitting the data synchronization requirements into multiple levels according to a plurality of predetermined dimensional levels, and obtaining a plurality of atomic requirements, wherein the plurality of dimensional levels include a data system level, a database level, and a data table level; Formulate data synchronization tasks based on the atomic requirements, and package the synchronization tasks of each branch into a data synchronization plan; Determine a first data center corresponding to each branch from a plurality of predetermined data centers, wherein the data center closest to the branch is the first data center of the branch; Based on the data synchronization plan of each of the branches, the data is synchronized from the database of the first data center to the database of the corresponding branch.

2. The method according to claim 1, characterized in that After formulating the data synchronization task based on the atomic requirement, the method further includes: Based on the atomic demand, determining first table structure information of a plurality of first data tables from a database of the first data center; Based on the atomic requirement, obtaining second table structure information of a plurality of pre-stored second data tables; Comparing the first table structure information with the second table structure information, and determining whether to perform standardization processing on the data tables of each of the branches according to the comparison result, wherein the standardization processing includes one of table creation and table structure update; After packaging the synchronization tasks of each branch into a data synchronization plan, the method further includes: The operation status of the database of each branch office is determined, and when the operation status of the database of each branch office is a normal operation status, a data synchronization operation is performed.

3. The method according to claim 1, characterized in that After formulating the data synchronization task based on the atomic requirement, the method further includes: Configuring multiple task nodes for the data synchronization task, and monitoring the data synchronization task based on the multiple task nodes; After packaging the synchronization tasks of each branch into a data synchronization plan, the method further includes: A plurality of plan nodes of the data synchronization plan are configured, and the data synchronization plan is monitored based on the plurality of plan nodes.

4. The method according to claim 3, characterized in that The method further comprises: Generate execution logs of the plurality of task nodes and the plurality of plan nodes, and output the execution logs in a visual manner.

5. The method according to claim 1, characterized in that The method further comprises: During the data synchronization process, monitoring whether the computer room of the first data center has any abnormality; If so, the first data center is switched to the second data center to synchronize data from the second data center to the database of the corresponding branch of the first data center, wherein the data center that is next closest to the corresponding branch is the second data center of the corresponding branch.

6. The method according to claim 1, characterized in that The method further comprises: During the data synchronization process, extracting an operation log of the database of the first data center, and determining a data change operation of the database of the first data center based on the operation log; If the data changed by the data change operation is currently synchronized data, a change message corresponding to the data change operation is generated, and the change message is sent to the database of the corresponding branch.

7. The method according to claim 1, characterized in that The method further comprises: During the data synchronization process, an extraction process is started, data change information of the database of the first data center is extracted based on the extraction process, and the data change information is written into a track file; A delivery process is started, and the trajectory file is sent to a database of a corresponding branch based on the delivery process.

8. A data synchronization device, characterized in that: include: A splitting module is used to collect data synchronization requirements of various branches, and split the data synchronization requirements into multiple levels according to a predetermined multiple dimensional levels to obtain multiple atomic requirements, wherein the multiple dimensional levels include a data system level, a database level, and a data table level; A formulation module, used to formulate data synchronization tasks based on the atomic requirements, and to package the synchronization tasks of each branch into a data synchronization plan; A determination module, configured to determine a first data center corresponding to each of the branch offices from a plurality of predetermined data centers, wherein the data center closest to the branch office is the first data center of the branch office; The synchronization module is used to synchronize data from the database of the first data center to the database of the corresponding branch office based on the data synchronization plan of each branch office.

9. A computer device, characterized in that: include: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one executable instruction, and when the executable instruction is executed on a computer device, the computer device executes the method according to any one of claims 1 to 7.

Images