Master data real-time synchronization method, device and equipment and computer readable storage medium

Through Flink CDC technology, the problem of poor real-time synchronization of master data is solved, the synchronization delay at millisecond level is achieved, synchronization performance and timeliness are improved, and it is suitable for a variety of master data synchronization scenarios.

CN118981502APending Publication Date: 2024-11-19WUHAN SOUTH SAGITTARIUS INTEGRATION CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411108753.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In the prior art, the real-time synchronization of master data is poor, resulting in a time difference between the business system and the master data management system before and after it is launched, increasing the risk of misinformation exchange.

Method used

Using Flink CDC technology, a dictionary comparison table is generated by determining the set of main data fields to be synchronized, dependent dictionaries and other main data sets, and a Flink CDC task is generated based on the field distribution situation to achieve real-time synchronization of the main data.

Benefits of technology

It realizes millisecond-level delay synchronization of the main data, improves synchronization performance and timeliness, simplifies the data synchronization process between the main data system and the business system, and is suitable for most master data synchronization scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981502A_ABST
    Figure CN118981502A_ABST
Patent Text Reader

Abstract

The invention discloses a master data real-time synchronization method, device and equipment and a computer readable storage medium. The method comprises the following steps: determining a field set of master data to be synchronized, a dictionary on which the master data depends and other master data sets on which the master data depends; synchronizing other master data sets on which the master data depends into a service system; when the enumeration values of the business system fields and the dictionary on which the main data depend are the same but the coding values are different, or the business system fields and the dictionary on which the main data depend are inconsistent but the enumeration values have a conversion corresponding relation, taking the enumeration values as main keys to generate a dictionary comparison table; associating the main data with the dictionary comparison table to generate a result table; taking the result table as a data reading table, and generating a Flink CDC task based on the distribution condition of the field set; and running the Flink CDC task so as to synchronize the main data into the business system. Compared with the prior art, the synchronous performance and the synchronous timeliness are greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a master data real-time synchronization method, device, equipment and computer-readable storage medium. Background Art

[0002] As the golden data of an enterprise, master data must be cross-departmental, cross-business, and cross-system. When there are many business systems before the master data management system is launched, how to ensure that the business systems launched before the master data management system is launched and the business systems launched after the master data management system are consistent at the second level is a challenging issue. At present, the mainstream method in the industry is to synchronize batches at regular intervals and manually. The time difference may be a few minutes, a few hours, or even a few days. This time difference makes business personnel at risk of exchanging wrong information when communicating with users or other internal members. Summary of the invention

[0003] The present application provides a master data real-time synchronization method, device, equipment and computer-readable storage medium, which can solve the technical problem of poor real-time master data synchronization in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a master data real-time synchronization method, the master data real-time synchronization method comprising:

[0005] Determine the field set of the master data to be synchronized, the dictionary on which the master data depends, and other master data sets on which the master data depends;

[0006] Synchronize other master data sets that the master data depends on into the business system;

[0007] When the business system field and the dictionary enumeration value that the master data depends on are the same but the coding value is different, or the business system field and the dictionary that the master data depends on are inconsistent but the enumeration value has a conversion correspondence relationship, a dictionary comparison table is generated with the enumeration value as the primary key;

[0008] Associate the master data with the dictionary comparison table to generate a result table;

[0009] Use the result table as the data reading table and generate a Flink CDC task based on the distribution of the field set;

[0010] Run the Flink CDC task to synchronize master data to the business system.

[0011] In combination with the first aspect, in one implementation, after the step of synchronizing other master data sets that the master data depends on into the business system, the method further includes:

[0012] When the business system dictionary is consistent with the dictionary that the master data depends on, the table is read with the master data as the data and a Flink CDC task is generated based on the distribution of the field set;

[0013] Run the Flink CDC task to synchronize master data to the business system.

[0014] In combination with the first aspect, in one implementation, the step of running the Flink CDC task includes:

[0015] Run the Flink CDC task through the Flink cluster.

[0016] In combination with the first aspect, in one implementation, the master data real-time synchronization method further includes:

[0017] Monitor the running status while the Flink CDC task is running.

[0018] In combination with the first aspect, in one implementation, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set both exist in the same data table, the step of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set includes:

[0019] Generate a Flink CDC task with the result table as the data reading table and the same data table as the data writing table.

[0020] In combination with the first aspect, in one implementation, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set exists in a table Tsink_c, and the primary key and the extended field set in the core field set exist in another table Tsink_e, the step of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set includes:

[0021] Generate Flink CDC task 1 with the result table as the data reading table and Tsink_c as the data writing table;

[0022] Generate Flink CDC task 2 with the result table as the data reading table and Tsink_e as the data writing table.

[0023] In combination with the first aspect, in one implementation, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set are distributed in data table 1 to data table N, N is a positive integer greater than 2, and the step of taking the result table as the data reading table and generating a FlinkCDC task based on the distribution of the field set includes:

[0024] When data tables 1 to N are all dimension tables and have primary data primary keys, the result table is used as the data reading table, and data tables 1 to N are used as data writing tables, and Flink CDC tasks 1 to N are generated respectively.

[0025] In a second aspect, an embodiment of the present application provides a master data real-time synchronization device, the master data real-time synchronization device comprising:

[0026] A determination module is used to determine the field set of the master data to be synchronized, the dictionary on which the master data depends, and other master data sets on which the master data depends;

[0027] The first synchronization module is used to synchronize other master data sets that the master data depends on into the business system;

[0028] The first generation module is used to generate a dictionary comparison table using the enumeration value as the primary key when the business system field and the dictionary enumeration value that the master data depends on are the same but the coding value is different or the business system field and the dictionary that the master data depends on are inconsistent but the enumeration value has a conversion correspondence relationship;

[0029] The second generation module is used to associate the master data with the dictionary comparison table to generate a result table;

[0030] The third generation module is used to use the result table as the data reading table and generate a Flink CDC task based on the distribution of the field set;

[0031] The second synchronization module is used to run the Flink CDC task to synchronize the master data to the business system.

[0032] In a third aspect, an embodiment of the present application provides a master data real-time synchronization device, which includes a processor, a memory, and a master data real-time synchronization program stored in the memory and executable by the processor, wherein when the master data real-time synchronization program is executed by the processor, the steps of the master data real-time synchronization method described in the first aspect are implemented.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a master data real-time synchronization program is stored, wherein when the master data real-time synchronization program is executed by a processor, the steps of the master data real-time synchronization method described in the first aspect are implemented.

[0034] The beneficial effects brought by the technical solution provided in the embodiments of the present application include:

[0035] In an embodiment of the present application, a field set of master data to be synchronized, a dictionary on which the master data depends, and other master data sets on which the master data depends are determined; the other master data sets on which the master data depends are synchronized to the business system; when the business system field and the dictionary enumeration value on which the master data depends are the same but the encoding value is different or the business system field and the dictionary on which the master data depends are inconsistent but the enumeration value has a conversion correspondence, a dictionary comparison table is generated using the enumeration value as the primary key; the master data is associated with the dictionary comparison table to generate a result table; the result table is used as a data reading table and a Flink CDC task is generated based on the distribution of the field set; and the Flink CDC task is run to synchronize the master data to the business system. The embodiment of the present application adopts Flink CDC technology, and the master data delay is at the millisecond level. Compared with the solution of master data synchronization through timed batch synchronization combined with manual synchronization in the prior art, the synchronization performance and synchronization timeliness are greatly improved; and the embodiment of the present application is easy to use, and there is no need to carry out cumbersome data synchronization API interface development between the master data system and the business system; the embodiment of the present application directly acts on the underlying data, bypassing the complex business logic relationship; the application scenarios of the embodiment of the present application are diverse and practical, and the actual development cycle is short; the embodiment of the present application has a wide range of application and is suitable for master data synchronization in most situations. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of an embodiment of a real-time master data synchronization method of the present application;

[0037] Figure 2 This is a flow chart of another embodiment of the real-time master data synchronization method of the present application;

[0038] Figure 3 This is a functional module diagram of an embodiment of a master data real-time synchronization device of the present application;

[0039] Figure 4 This is a schematic diagram of the hardware structure of the master data real-time synchronization device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0041] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0042] In a first aspect, an embodiment of the present application provides a method for real-time synchronization of master data.

[0043] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the real-time master data synchronization method of this application. Figure 1 As shown, the master data real-time synchronization method includes:

[0044] Step S10, determining the field set of the master data to be synchronized, the dictionary on which the master data depends, and other master data sets on which the master data depends;

[0045] In this embodiment, the master data Tsrc to be synchronized is determined, and on this basis, the field set of Tsrc is determined. The field set of Tsrc includes the core field set C_CORE, the extended field set C_EXT and the derived field set C_OTH. The dictionary T_DICT on which Tsrc depends and other master data sets C_OTH on which Tsrc depends are determined. <t>.

[0046] Among them, C_CORE is the field set that must be synchronized, and C_EXT and C_OTH are the field sets that exist in the business system. It should be noted that the C_OTH field set will not be designed as a field of the main data table in most cases, but C_EXT will definitely be. The following embodiments will only consider the case where C_OTH does not exist. If C_OTH exists in actual situations, it can be processed according to C_EXT.

[0047] Specifically, object-oriented methods can be used to divide the attributes of master data. Generally speaking, the master data system has defined C_CORE and C_EXT when it comes to the master data. The dictionary T_DICT that Tsrc depends on can be obtained through field annotations or by consulting relevant developers. <to>It can be obtained through the blood relationship between master data.

[0048] Step S20, synchronizing other master data sets that the master data depends on into the business system;

[0049] In this embodiment, the relevant fields of the master data may depend on other master data sets. For example, the employee table master data depends on the department table master data. Therefore, before synchronizing the master data, other master data sets that the master data depends on should be synchronized to the business system.

[0050] Step S30, when the business system field and the dictionary enumeration value that the master data depends on are the same but the coding value is different or the business system field and the dictionary that the master data depends on are inconsistent but the enumeration value has a conversion correspondence relationship, a dictionary comparison table is generated using the enumeration value as the primary key;

[0051] In this embodiment, according to the correspondence between the dictionary on which the master data depends and the business system, the following situations may exist:

[0052] Scenario 1: The business system dictionary is consistent with the dictionary that the master data depends on;

[0053] Scenario 2: The business system field and the dictionary enumeration value that the master data depends on are the same, but the coded values ​​are different;

[0054] Scenario 3: The business system field is inconsistent with the dictionary that the master data depends on, but there is a conversion correspondence between the enumeration values.

[0055] For case 1, no additional processing is required; for case 2 or 3, it is necessary to generate a dictionary comparison table Tmat using the enumeration value as the primary key, and Tmat is a static dimension table.

[0056] Specifically, the main data system dictionary table is used as the main table, the business system dictionary table is used as the secondary table, and the dictionary value is used as the primary key to associate them to generate a dictionary CODE correspondence table Tmat between the main data system dictionary table and the business system dictionary table.

[0057] It should be noted that most companies with master data management systems will uniformly manage dictionaries according to the level of master data. New business systems will adopt the dictionary of the master data system, and existing business systems may refuse to adopt it. However, due to business needs, the enumeration values ​​will be the same. In the extreme case, one enumeration value of the existing business system may correspond to the enumeration values ​​of multiple master data systems. For example, the approval status of the existing system may be simply "pending approval", "under approval", "approval passed", "approval failed", and the master data system may be "pending submission", "pending approval", etc., among which the "pending approval" of the existing business system corresponds to the "pending submission" and "pending approval" of the master data system.

[0058] Step S40, associating the master data with the dictionary comparison table to generate a result table;

[0059] In this embodiment, by associating the master data Tsrc with Tmat, the dictionary code in Tsrc is converted into the dictionary code of the business system, and the result table Tsrc-r is obtained, where Tmat is a static dimension table and Tsrc-r is a Flink CDC dynamic table. Here, "association" refers to the JOIN operation in the database.

[0060] Considering the stability of the dictionary, Tmat is used as a static dimension table. That is, Tmat of the synchronization task is only read once when the program starts. If the dictionary changes later, Tmat will not be aware of it. At this time, the master data synchronization service needs to be restarted. In actual situations, the dictionary of the online system rarely changes the existing dictionary. This is because if the existing dictionary is changed, the historical data cannot read the dictionary value correctly.

[0061] Step S50, taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set;

[0062] In this embodiment, the distribution of the field set may be as follows:

[0063] Case 1: C_CORE and C_EXT (or C_EXT does not exist) both exist in the same table Tsink, that is, C <tsink>The length is 1;

[0064] Case 2: C_CORE exists in one table Tsink_c, the primary key in C_CORE and C_EXT exist in another table Tsink_e, C <tsink>The length is 2;

[0065] Case 3: C_CORE and C_EXT are scattered across multiple tables.

[0066] It is necessary to generate corresponding Flink CDC tasks based on different situations.

[0067] Further, in one embodiment, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set both exist in the same data table, the step of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set includes:

[0068] Generate a Flink CDC task with the result table as the data reading table and the same data table as the data writing table.

[0069] In this embodiment, when it is case 1, the Flink CDC task is written with the result table Tsrc-r as the data reading table (ie, source table) and Tsink as the data writing table (ie, sink table). In this case, only one Flink CDC task is generated.

[0070] Further, in one embodiment, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set exists in a table Tsink_c, and the primary key and the extended field set in the core field set exist in another table Tsink_e, the step of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set includes:

[0071] Generate Flink CDC task 1 with the result table as the data reading table and Tsink_c as the data writing table;

[0072] Generate Flink CDC task 2 with the result table as the data reading table and Tsink_e as the data writing table.

[0073] In this embodiment, when it is case 2, the result table Tsrc-r is used as the data reading table (i.e., source table) and Tsink_c is used as the data writing table (i.e., sink table) to write Flink CDC task 1, and the result table Tsrc-r is used as the data reading table (i.e., source table) and Tsink_e is used as the data writing table (i.e., sink table) to write Flink CDC task 2; in this case, two Flink CDC tasks are generated. The specific process is as follows:

[0074] Check whether the C_CORE and C_EXT field types in Tsink_c and Tsink_e are consistent with the data types in the master data Tsrc. If they are inconsistent, perform necessary conversion operations. Check whether the primary key of the Tsink_c table is consistent with that of Tsrc. If they are inconsistent, use the natural key as the primary key for synchronization. It should be noted that when the natural key is used as the primary key, the uniqueness of the natural key must be guaranteed. It is best to add a unique index constraint to the Tsink_c table and the Tsrc table. Check whether the primary key of the Tsink_e table is consistent with that of Tsrc. If it is inconsistent, try to use the method for handling when the primary key of the Tsink_c table is inconsistent with that of Tsrc. If not, use the dynamicTable method in flink to synchronize Tsink_e. It should be noted that the dynamicTable synchronization method in flink is full update synchronization, which is not suitable for synchronization of master data tables with more than 1 million. Use Tsrc-r as the source table and Tsink_c as the sink table to write the Flink CDC task Task-C using FlinkSQL; use Tsrc-r as the source table and Tsink_e as the sink table to write the Flink CDC synchronization task Task-E using FlinkSQL.

[0075] Further, in one embodiment, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set are distributed in data table 1 to data table N, N is a positive integer greater than 2. The step of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set includes:

[0076] When data tables 1 to N are all dimension tables and have primary data primary keys, the result table is used as the data reading table, and data tables 1 to N are used as data writing tables, and Flink CDC tasks 1 to N are generated respectively.

[0077] In this embodiment, when it is situation 3, it is necessary to first consider whether there is a possibility of synchronization. Specifically, it is necessary to consider whether all the data tables involved are dimension tables and whether all the data tables involved have primary data primary keys.

[0078] If data tables 1 to N are all dimension tables and all have primary data primary keys, use the result table Tsrc-r as the data reading table (i.e., source table) and data table 1 as the data writing table (i.e., sink table) to write Flink CDC task 1, use the result table Tsrc-r as the data reading table (i.e., source table) and data table 2 as the data writing table (i.e., sink table) to write Flink CDC task 2, use the result table Tsrc-r as the data reading table (i.e., source table) and data table 3 as the data writing table (i.e., sink table) to write Flink CDC task 3; and so on, generate N Flink CDC tasks.

[0079] Step S60: Run the Flink CDC task to synchronize the master data to the business system.

[0080] In this embodiment, the Flink CDC task is run to synchronize the master data to the business system.

[0081] Furthermore, in one embodiment, the step of running the Flink CDC task includes:

[0082] Run the Flink CDC task through the Flink cluster.

[0083] In this embodiment, the generated Flink CDC task is started to synchronize the master data. The synchronization task between a single master data and a single system should be managed as a task. The main reason for running synchronization tasks in Flink cluster mode is that the number of synchronization tasks is large, but the logic of each synchronization task is simple, the amount of data is small, and the state that needs to be cached is also small, so sharing the same JobManger can save a lot of computing resources. It is recommended to use FlinkSQL as the code type.

[0084] Furthermore, in one embodiment, the master data real-time synchronization method further includes:

[0085] Monitor the running status while the Flink CDC task is running.

[0086] In this embodiment, since all synchronization tasks are Flink CDC tasks, the stability of the Flink cluster is the focus of monitoring. Since Flink CDC can ensure strong end-to-end consistency when supported by source and sink components, it is sufficient to monitor the running status of the Flink CDC task.

[0087] In an embodiment of the present application, a field set of master data to be synchronized, a dictionary on which the master data depends, and other master data sets on which the master data depends are determined; the other master data sets on which the master data depends are synchronized to the business system; when the business system field and the dictionary enumeration value on which the master data depends are the same but the encoding value is different or the business system field and the dictionary on which the master data depends are inconsistent but the enumeration value has a conversion correspondence, a dictionary comparison table is generated using the enumeration value as the primary key; the master data is associated with the dictionary comparison table to generate a result table; the result table is used as a data reading table and a Flink CDC task is generated based on the distribution of the field set; and the Flink CDC task is run to synchronize the master data to the business system. The embodiment of the present application adopts Flink CDC technology, and the master data delay is at the millisecond level. Compared with the solution of master data synchronization through timed batch synchronization combined with manual synchronization in the prior art, the synchronization performance and synchronization timeliness are greatly improved; and the embodiment of the present application is easy to use, and there is no need to carry out cumbersome data synchronization API interface development between the master data system and the business system; the embodiment of the present application directly acts on the underlying data, bypassing the complex business logic relationship; the application scenarios of the embodiment of the present application are diverse and practical, and the actual development cycle is short; the embodiment of the present application has a wide range of application and is suitable for master data synchronization in most situations.

[0088] Further, in one embodiment, referring to Figure 2 , Figure 2 This is a flow chart of another embodiment of the real-time master data synchronization method of the present application. Figure 2 As shown, after step S20, the following further includes:

[0089] Step S70: When the business system dictionary is consistent with the dictionary that the master data depends on, the master data is used as the data reading table and a Flink CDC task is generated based on the distribution of the field set.

[0090] Step S80: Run the Flink CDC task to synchronize the master data to the business system.

[0091] In this embodiment, when the correspondence between the dictionary on which the master data depends and the business system is situation 1, a Flink CDC task is generated in a manner substantially the same as that of the step S50 embodiment. The difference is that in situation 1, since steps S30 to S40 are not performed, that is, a result table is not generated, the master data is used as the data reading table when the Flink CDC task is generated.

[0092] In a second aspect, an embodiment of the present application also provides a master data real-time synchronization device.

[0093] In one embodiment, referring to Figure 3 , Figure 3 This is a functional module diagram of an embodiment of the master data real-time synchronization device of the present application. Figure 3 As shown, the master data real-time synchronization device includes:

[0094] A determination module 10 is used to determine a set of fields of the master data to be synchronized, a dictionary on which the master data depends, and other sets of master data on which the master data depends;

[0095] A first synchronization module 20, used to synchronize other master data sets that the master data depends on into the business system;

[0096] The first generating module 30 is used to generate a dictionary comparison table using the enumeration value as the primary key when the business system field and the dictionary enumeration value that the master data depends on are the same but the coding value is different or the business system field and the dictionary that the master data depends on are inconsistent but the enumeration value has a conversion correspondence relationship;

[0097] The second generating module 40 is used to associate the master data with the dictionary comparison table to generate a result table;

[0098] The third generation module 50 is used to use the result table as the data reading table and generate a Flink CDC task based on the distribution of the field set;

[0099] The second synchronization module 60 is used to run the Flink CDC task to synchronize the master data to the business system.

[0100] Furthermore, in one embodiment, the master data real-time synchronization device further includes a fourth generation module 50, which is used to:

[0101] When the business system dictionary is consistent with the dictionary that the master data depends on, the table is read with the master data as the data and a Flink CDC task is generated based on the distribution of the field set;

[0102] The second synchronization module 60 is also used to run the Flink CDC task to synchronize the master data to the business system.

[0103] Furthermore, in one embodiment, the second synchronization module 60 is used to:

[0104] Run the Flink CDC task through the Flink cluster.

[0105] Furthermore, in one embodiment, the master data real-time synchronization device further includes a monitoring module, which is used to:

[0106] Monitor the running status while the Flink CDC task is running.

[0107] Further, in one embodiment, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set are both present in the same data table, the third generating module 50 is used to:

[0108] Generate a Flink CDC task with the result table as the data reading table and the same data table as the data writing table.

[0109] Further, in one embodiment, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set exists in one table Tsink_c, and the primary key and the extended field set in the core field set exist in another table Tsink_e, the third generation module 50 is used to:

[0110] Generate Flink CDC task 1 with the result table as the data reading table and Tsink_c as the data writing table;

[0111] Generate Flink CDC task 2 with the result table as the data reading table and Tsink_e as the data writing table.

[0112] Further, in one embodiment, the field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set are distributed in data table 1 to data table N, N is a positive integer greater than 2, the third generation module 50 is used to:

[0113] When data tables 1 to N are all dimension tables and have primary data primary keys, the result table is used as the data reading table, and data tables 1 to N are used as data writing tables, and Flink CDC tasks 1 to N are generated respectively.

[0114] Among them, the functional implementation of each module in the above-mentioned master data real-time synchronization device corresponds to each step in the above-mentioned master data real-time synchronization method embodiment, and its functions and implementation processes are no longer repeated here.

[0115] In a third aspect, an embodiment of the present application provides a master data real-time synchronization device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0116] Reference Figure 4 , Figure 4 The hardware structure diagram of the master data real-time synchronization device involved in the embodiment of the present application is shown in FIG. In the embodiment of the present application, the master data real-time synchronization device may include a processor, a memory, a communication interface, and a communication bus.

[0117] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0118] The communication interface includes input / output (I / O) interface, physical interface and logical interface, etc., which are used to realize the interconnection of devices inside the master data real-time synchronization device, and the interface used to realize the interconnection of the master data real-time synchronization device with other devices (such as other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber optic interface, an ATM interface, etc.; the user device can be a display, a keyboard, etc.

[0119] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0120] The processor may be a general-purpose processor, and the general-purpose processor may call the master data real-time synchronization program stored in the memory and execute the master data real-time synchronization method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the master data real-time synchronization program is called may refer to the various embodiments of the master data real-time synchronization method of the present application, and will not be repeated here.

[0121] Those skilled in the art will understand that Figure 4 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0122] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0123] The computer-readable storage medium of the present application stores a master data real-time synchronization program, wherein when the master data real-time synchronization program is executed by a processor, the steps of the master data real-time synchronization method described above are implemented.

[0124] Among them, the method implemented when the master data real-time synchronization program is executed can refer to the various embodiments of the master data real-time synchronization method of the present application, and will not be repeated here.

[0125] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0126] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0127] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0128] In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; the "and / or" in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" refers to two or more than two.

[0129] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0131] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.< / tsink> < / tsink> < / to> < / t>

Claims

1. A real-time master data synchronization method, It is characterized in that The master data real-time synchronization method comprises: Determine the field set of the master data to be synchronized, the dictionary on which the master data depends, and other master data sets on which the master data depends; Synchronize other master data sets that the master data depends on into the business system; When the business system field and the dictionary enumeration value that the master data depends on are the same but the coding value is different, or the business system field and the dictionary that the master data depends on are inconsistent but the enumeration value has a conversion correspondence relationship, a dictionary comparison table is generated with the enumeration value as the primary key; Associate the master data with the dictionary comparison table to generate a result table; Use the result table as the data reading table and generate a Flink CDC task based on the distribution of the field set; Run the Flink CDC task to synchronize master data to the business system.

2. The master data real-time synchronization method according to claim 1, It is characterized in that After the step of synchronizing other master data sets that the master data depends on into the business system, the process further includes: When the business system dictionary is consistent with the dictionary that the master data depends on, the table is read with the master data as the data and a Flink CDC task is generated based on the distribution of the field set; Run the Flink CDC task to synchronize master data to the business system.

3. The master data real-time synchronization method according to claim 1 or 2, It is characterized in that The steps of running the Flink CDC task include: Run the Flink CDC task through the Flink cluster.

4. The master data real-time synchronization method according to claim 3, It is characterized in that The master data real-time synchronization method also includes: Monitor the running status while the Flink CDC task is running.

5. The master data real-time synchronization method according to claim 1, It is characterized in that The field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set exist in the same data table, the steps of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set include: Generate a Flink CDC task with the result table as the data reading table and the same data table as the data writing table.

6. The master data real-time synchronization method according to claim 1, It is characterized in that The field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set exists in a table Tsink_c, and the primary key and the extended field set in the core field set exist in another table Tsink_e, the steps of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set include: Generate Flink CDC task 1 with the result table as the data reading table and Tsink_c as the data writing table; Generate Flink CDC task 2 with the result table as the data reading table and Tsink_e as the data writing table.

7. The master data real-time synchronization method according to claim 1, It is characterized in that The field set includes a core field set and an extended field set. When the distribution of the field set is that the core field set and the extended field set are distributed in data table 1 to data table N, N is a positive integer greater than 2. The step of taking the result table as the data reading table and generating a Flink CDC task based on the distribution of the field set includes: When data tables 1 to N are all dimension tables and have primary data primary keys, the result table is used as the data reading table, and data tables 1 to N are used as data writing tables, and Flink CDC tasks 1 to N are generated respectively.

8. A master data real-time synchronization device, It is characterized in that The master data real-time synchronization device comprises: A determination module is used to determine the field set of the master data to be synchronized, the dictionary on which the master data depends, and other master data sets on which the master data depends; The first synchronization module is used to synchronize other master data sets that the master data depends on into the business system; The first generation module is used to generate a dictionary comparison table using the enumeration value as the primary key when the business system field and the dictionary enumeration value that the master data depends on are the same but the coding value is different or the business system field and the dictionary that the master data depends on are inconsistent but the enumeration value has a conversion correspondence relationship; The second generation module is used to associate the master data with the dictionary comparison table to generate a result table; The third generation module is used to use the result table as the data reading table and generate a FlinkCDC task based on the distribution of the field set; The second synchronization module is used to run the Flink CDC task to synchronize the master data to the business system.

9. A master data real-time synchronization device, It is characterized in that The master data real-time synchronization device includes a processor, a memory, and a master data real-time synchronization program stored in the memory and executable by the processor, wherein when the master data real-time synchronization program is executed by the processor, the steps of the master data real-time synchronization method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a master data real-time synchronization program, wherein when the master data real-time synchronization program is executed by a processor, the steps of the master data real-time synchronization method according to any one of claims 1 to 7 are implemented.