Master data item determination method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a main data item determination method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the specific scheme is as follows: determining a source service table of a source service system and a target service table of a target service system based on a target main data theme and a data consanguinity map of a data processing platform, a plurality of service systems are deployed on the data processing platform, and the data consanguinity map represents a dependency relationship among data items in service tables of the plurality of service systems; dividing source data items in the source service table to obtain a plurality of source data item sets; based on the blood relationship map, obtaining the frequency of each source data item set depended by each target service table; and on the basis of the dependent frequency of each source data item set, determining main data items under the target main data theme from the source data items included in each source data item set. By applying the scheme provided by the embodiment of the invention, the efficiency of determining the master data can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for determining a master data item. Background Art

[0002] In today's highly digitalized world, digital transformation has become an inevitable path for enterprise development, and the application of master data is crucial in this process. Master data refers to the fundamental information that meets the needs of organizational collaboration and reflects the status and attributes of core business entities. It is authoritative, comprehensive, shareable, stable, long-term, scalable, and highly valuable, making it a vital enterprise data asset.

[0003] Given the above situation, practical applications require determining which data items qualify as master data. Currently, staff members primarily screen potential master data items based on their personal experience, then negotiate with other staff members before finalizing the selection. This process is significantly influenced by staff subjective factors and requires repeated consultations between different staff members, which is time-consuming and inefficient. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a method, device, electronic device, and storage medium for determining master data items to improve the efficiency of determining master data. The specific technical solution is as follows:

[0005] According to one aspect of an embodiment of the present invention, a method for determining a master data item is provided, the method comprising:

[0006] Determining a source business table of a source business system and a target business table of a target business system based on a target master data theme and a data lineage map of a data processing platform, wherein the data processing platform is deployed with multiple business systems, and the data lineage map represents the dependency relationships between data items in the business tables of the multiple business systems;

[0007] Divide the source data items in the source business table to obtain multiple source data item sets;

[0008] Based on the bloodline graph, obtaining the frequency of each source data item set being dependent on by each target business table;

[0009] Based on the frequency of dependency of each source data item set, master data items under the target master data theme are determined from the source data items included in each source data item set.

[0010] In one embodiment of the present invention, the source data items in the source business table are divided to obtain multiple source data item sets, including:

[0011] Based on the data lineage graph, source data items in the source business table that have a dependency relationship with target data items in the target business table are determined, and the determined source data items are divided to obtain multiple source data item sets;

[0012] or

[0013] All source data items in the source business table are divided to obtain multiple source data item sets.

[0014] In one embodiment of the present invention, obtaining the frequency of each source data item set being dependent on each target business table based on the lineage graph includes:

[0015] For each source data item set, the frequency of each source data item set being depended on by each target business table is obtained in the following manner:

[0016] For each target business table, based on the data lineage graph, determine whether all source data items in the source data item set are dependent on the target business table; if so, determine that the target business table is a dependent table of the source data item set;

[0017] Count the number of dependent tables of the source data item set as the frequency of the source data item set being dependent.

[0018] In one embodiment of the present invention, determining the master data items under the target master data theme from the source data items included in each source data item set based on the frequency of dependency of each source data item set includes:

[0019] Determine, from each set of source data items, a first set whose frequency of dependency is greater than a preset first threshold and a second set whose frequency of dependency is less than or equal to the first threshold and greater than a preset second threshold;

[0020] Based on the source data items included in the first set and the source data items included in the second set, master data items under the target master data subject are determined.

[0021] In one embodiment of the present invention, determining the master data items under the target master data subject based on the source data items included in the first set and the source data items included in the second set includes:

[0022] If there are no duplicate source data items between the first set and the second set, the source data items included in the first set are determined as core data items of the master data under the target master data subject, and the source data items included in the second set are determined as subsidiary data items of the master data under the target master data subject;

[0023] If there are repeated source data items between the first set and the second set, the source data items included in the first set are determined as core data items of the master data under the target master data subject, and the source data items in the second set that do not exist in the first set are determined as subsidiary data items of the master data under the target master data subject; or the repeated source data items between the first set and the second set are determined as core data items of the master data under the target master data subject, and the non-repeated source data items between the first set and the second set are determined as subsidiary data items of the master data under the target master data subject.

[0024] In one embodiment of the present invention, determining the source business table of the source business system and the target business table of the target business system based on the target master data subject and the data lineage map of the data processing platform includes:

[0025] Obtain the data items corresponding to the target master data subject;

[0026] Based on the data lineage map of the data processing platform, determine the first business table to which the upstream data item of the obtained data item belongs, the second business table to which the downstream data item belongs, and the third business table to which the obtained data item belongs;

[0027] Determine a source business table of a source business system based on the first business table and the third business table;

[0028] A target business table of a target business system is determined based on the second business table and the third business table.

[0029] According to another aspect of an embodiment of the present invention, a device for determining a master data item is provided, the device comprising:

[0030] A business table determination module is configured to determine a source business table of a source business system and a target business table of a target business system based on a target master data theme and a data lineage map of a data processing platform, wherein the data processing platform is deployed with multiple business systems and the data lineage map represents the dependency relationships between data items in the business tables of the multiple business systems;

[0031] A source data item partitioning module is used to partition source data items in a source business table to obtain multiple source data item sets;

[0032] A frequency acquisition module, configured to obtain, based on the bloodline graph, the frequency at which each source data item set is relied upon by each target business table;

[0033] The data item determination module is configured to determine the master data items under the target master data theme from the source data items included in each source data item set based on the frequency of dependency of each source data item set.

[0034] In one embodiment of the present invention, the source data item partitioning module is specifically used to determine, from the source data items in the source business table, source data items that have a dependency relationship with the target data items in the target business table based on the data lineage map, and partition the determined source data items to obtain multiple source data item sets; or partition all source data items in the source business table to obtain multiple source data item sets.

[0035] In one embodiment of the present invention, the frequency acquisition module is specifically used to obtain, for each source data item set, the frequency of each source data item set being dependent on by each target business table in the following manner: for each target business table, based on the data lineage map, determine whether each source data item in the source data item set is dependent on the target business table; if all are dependent on the target business table, determine that the target business table is the dependent table of the source data item set; and count the number of dependent tables of the source data item set as the frequency of the source data item set being dependent.

[0036] In one embodiment of the present invention, the data item determination module is specifically used to determine, from each set of source data items, a first set whose frequency of dependence is greater than a preset first threshold and a second set whose frequency of dependence is less than or equal to the first threshold and greater than a preset second threshold; and determine the master data items under the target master data subject based on the source data items included in the first set and the source data items included in the second set.

[0037] In one embodiment of the present invention, the data item determination module is specifically used to determine the source data items included in the first set as the core data items of the master data under the target master data subject, and determine the source data items included in the second set as the subsidiary data items of the master data under the target master data subject if there are no repeated source data items between the first set and the second set; if there are repeated source data items between the first set and the second set, determine the source data items included in the first set as the core data items of the master data under the target master data subject, and determine the source data items in the second set that do not exist in the first set as the subsidiary data items of the master data under the target master data subject; or determine the repeated source data items between the first set and the second set as the core data items of the master data under the target master data subject, and determine the non-repeated source data items between the first set and the second set as the subsidiary data items of the master data under the target master data subject.

[0038] In one embodiment of the present invention, the business table determination module is specifically used to obtain data items corresponding to the target master data subject; based on the data lineage map of the data processing platform, determine the first business table to which the upstream data item of the obtained data item belongs, the second business table to which the downstream data item belongs, and the third business table to which the obtained data item belongs; determine the source business table of the source business system based on the first business table and the third business table; and determine the target business table of the target business system based on the second business table and the third business table.

[0039] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0040] Memory for storing computer programs;

[0041] The processor is configured to implement any of the above-mentioned methods for determining a master data item when executing a program stored in the memory.

[0042] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for determining a master data item.

[0043] According to another aspect of the embodiments of the present invention, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for determining a master data item.

[0044] Beneficial effects of the embodiments of the present invention:

[0045] In the master data item determination method provided by the embodiment of the present invention, the target business table of the target business system and the source business table of the source business system are determined based on the data lineage map. In this way, the target business table can be used as a basis to determine the master data items under the target master data theme from the source data items included in the source business table. In this way, quickly and accurately determining the master data items can reduce labor costs and improve the efficiency of determining master data. In addition, the frequency of source data items being relied upon by each target business table can be characterized by: the globality, sharing, stability, long-term, importance, and coreness of the source data items in multiple business systems deployed on the data processing platform. The frequency of source data items being relied upon by each target business table is used as a judgment basis. The frequency of dependence can be used as data support for determining the master data items, thereby improving the credibility of the determined master data items, reducing repeated negotiations between staff members, reducing personnel communication costs, and improving the efficiency of determining master data.

[0046] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0048] Figure 1 A schematic flow chart of a method for determining a master data item provided by an embodiment of the present invention;

[0049] Figure 2 A schematic diagram of a data lineage map provided by an embodiment of the present invention;

[0050] Figure 3 A flowchart of a method for determining a service table provided by an embodiment of the present invention;

[0051] Figure 4 A schematic structural diagram of a device for determining a master data item provided by an embodiment of the present invention;

[0052] Figure 5 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention are within the scope of protection of the present invention.

[0054] The following describes the execution subject of the embodiment of the present invention.

[0055] The solution provided by the embodiment of the present invention can be applied to electronic devices such as desktop computers, laptops, tablet computers, and servers, and can also be applied to a data item determination platform for determining master data items. For ease of expression, the execution entity of the master data item determination method provided by the embodiment of the present invention is collectively referred to as a master data determination platform.

[0056] The concepts involved in the embodiments of the present invention are described below.

[0057] 1. Data items

[0058] A data item is the smallest data unit that describes the attributes of an entity. For example, for the customer entity, the customer's attributes may include: "customer number", "customer name" and "contact information". Then, the data table that records the customer may include the "customer number" data item, the "customer name" data item and the "contact information" data item.

[0059] 2. Master Data Subject

[0060] Master data topics are master data categories that are logically categorized based on business needs. For example, master data topics may include employee master data topics, customer master data topics, supplier master data topics, product master data topics, material master data topics, and so on.

[0061] A correspondence can be set between master data topics and business tables. For example, the "Customer Information" business table can correspond to the customer master data topic. The "Employee Information" business table can correspond to the employee master data topic.

[0062] A corresponding relationship can also be set between master data topics and data items. For example, the "customer number" data item, the "customer name" data item, and the "contact information" data item can correspond to the customer master data topic, and the "employee number" data item, the "employee name" data item, and the "department" data item can correspond to the employee master data topic.

[0063] When determining master data items, a target master data topic can be pre-determined. This can be determined based on the usage rate of the data items within the master data topic and the value of the data items within the master data topic to the company. The company can be the company corresponding to the data processing platform. When determining the target master data topic, master data topics with a usage rate greater than a preset usage rate threshold and / or master data topics with a value to the company greater than a preset value threshold can be selected as the target master data topic.

[0064] The value of a data item within a master data theme to a company can be expressed through a pre-set value. This value can be determined based on the company's business. For example, for a service-oriented company primarily providing various services to clients, the data items within the customer master data theme are more important to the company and can be assigned a higher value. The usage rate of a data item within a master data theme can be determined by the frequency with which the data is accessed during various business processes within the data processing platform.

[0065] 3. Data Lineage Map

[0066] The data lineage map represents the dependencies between data items in the business tables of multiple business systems. The data lineage map can record the data source, data mapping relationships, and data applications of each data item.

[0067] For example, the business table of business system 1 records the data of data item 1, data item 2, and data item 3. The business table of business system 2 records data item 4. The data of data item 4 needs to be obtained by data processing based on the data of data item 1, data item 2, and data item 3 stored in system 1. Then, the data lineage map can record that data item 4 in the business table of business system 2 depends on data item 1, data item 2, and data item 3 in the business table of business system 1. Conversely, data item 1, data item 2, and data item 3 in the business table of business system 1 are dependent on data item 4 in the business table of business system 2.

[0068] For another example, data item 2 and data item 3 are recorded in the business table of business system 3. The data of data item 2 and data item 3 in the business table of business system 3 are read from the data of data item 2 and data item 3 recorded in the business table of business system 1. Then, the bloodline map can record that data item 2 in the business table of business system 3 depends on data item 2 in the business table of business system 1, and data item 3 in the business table of business system 3 depends on data item 3 in the business table of business system 1. Conversely, data item 2 in the business table of business system 1 is dependent on data item 2 in the business table of business system 3, and data item 3 in the business table of business system 1 is dependent on data item 3 in the business table of business system 3.

[0069] 4. Data Processing Platform

[0070] A data processing platform is a platform used by companies, enterprises or institutions to process various types of data.

[0071] 5. Source Business System and Target Business System

[0072] The source business system is the system that maintains and stores core business data. It is the data provider and can be used to provide various types of data to support its own or other systems to implement various functions. The target business system is the system that implements various functions based on the data provided by the source business system and is the data user.

[0073] 6. Source data items and target data items

[0074] Both source data items and target data items are data items. Source data items are data items in the source business table, while target data items are data items in the target business table. Different business tables can contain the same data items or different data items. For example, the business table of business system 1 may contain data items 1, 2, and 3, while the business table of business system 4 may contain data items 1, 6, and 7.

[0075] The following describes in detail the method for determining master data items provided by an embodiment of the present invention.

[0076] In one embodiment of the present invention, see Figure 1 A flowchart of a method for determining a master data item is provided, the method comprising the following steps S101-S104.

[0077] Step S101: Based on the target master data subject and the data lineage map of the data processing platform, determine the source business table of the source business system and the target business table of the target business system.

[0078] Among them, the data processing platform is deployed with multiple business systems.

[0079] The implementation of step S101 of determining the source service table and the target service table will be described in the following embodiments and will not be described in detail here.

[0080] Step S102: Divide the source data items in the source business table to obtain multiple source data item sets.

[0081] In one implementation, the master data determination platform determines, based on the data lineage graph, source data items in the source business table that have a dependency relationship with target data items in the target business table, and divides the determined source data items to obtain multiple source data item sets.

[0082] For example, see Figure 2 A schematic diagram of a data lineage map provided. Figure 2 In the example, the source business tables on the left include source business table 1, source business table 2, and source business table 3, and the target business tables on the right include target business table 1, target business table 2, and target business table 3. Figure 2The arrows in the table indicate that the data items in the source business table are dependent on the data items in the target business table. For example, data item A, data item B, data item C, data item D, data item L, and data item M in source business table 1 are all source data items, wherein data item A, data item B, data item C, and data item D in source business table 1 are all connected to data items in the target business table. Then, data item A, data item B, data item C, and data item D in source business table 1 are all source data items that have a dependency relationship with the target data item in the target business table. Similarly, data item A, data item E, data item F, data item G in source business table 2 and data item A, data item H, data item I, data item J, data item K in source business table 3 are also source data items that have a dependency relationship with the target data item in the target business table. Then, the source data items in each source business table that have a dependency relationship with the target data item in the target business table include: data item A, data item B, data item C, data item D, data item E, data item F, data item G, data item H, data item I, data item J, data item K, and data item L. Then, the source data items determined above are divided to obtain multiple source data item sets. The master data determination platform can divide the determined source data items in a combination manner based on the number of source data items in the source data item set being 1 to a first number of elements, where the first number is the number of the determined source data items. For example, if the determined source data item is the data item AL, then the determined data item AL can be divided in a combination manner based on the number of source data items in the source data item set being 1 to 12, to obtain multiple source data item sets.

[0083] in addition, Figure 2 In the example, data item M, data item N, and data item O in each source business table are not connected to the data item in the target business table, that is, data item M, data item N, and data item O have no dependency relationship with the target data item in the target business table. Then, the source data items that have no dependency relationship with the target data item in the target business table will not be divided, and the obtained source data item set will not include data item M, data item N, and data item O.

[0084] In this way, based on the target data items in the target business table, the source data items in the source business table that have a dependency relationship with the target data items in the target business table are determined. The source data items that can be accurately determined from a large number of source data items can be used as candidate master data items, which reduces the data items that need to be calculated and improves the efficiency of determining the master data items.

[0085] In another implementation, the master data determination platform divides all source data items in the source business table to obtain multiple source data item sets.

[0086] by Figure 2 In the data lineage graph shown, all source data items in the source business table include: data item A, data item B, data item C, data item D, data item E, data item F, data item G, data item H, data item I, data item J, data item K, data item L, data item M, data item N, and data item O. The master data determination platform can divide all source data items in a combination manner based on the number of source data items in the source data item set being 1 to a second number of elements, where the second number is the number of all source data items in the source business table.

[0087] In this way, it can be ensured that all source data items are included in each source data item set, comprehensively covering each source data item and reducing the omission of data items.

[0088] Step S103: Based on the lineage graph, obtain the frequency at which each source data item set is dependent on each target business table.

[0089] In one implementation, for each source data item set, the frequency with which each target business table depends on each source data item set is obtained in the following manner: for each target business table, based on the data lineage graph, determine whether each source data item in the source data item set is all dependent on the target business table; if all are dependent on the target business table, determine that the target business table is the dependent table of the source data item set; and count the number of dependent tables of the source data item set as the frequency with which the source data item set is dependent.

[0090] by Figure 2 Taking the source business table in as an example, the above data item AK is divided, and it is assumed that: source data item set 1 includes data item A, data item B, and data item C, source data item set 2 includes data item A, data item B, data item C, and data item D, source data item set 3 includes data item A, data item B, data item E, data item F, and data item L, and so on.

[0091] Source data item set 1

[0092] For business table 1, based on the lineage map, the master data determination platform can determine that every source data item in source data item set 1 is dependent on target business table 1. For business table 2, based on the lineage map, the master data determination platform can determine that every source data item in source data item set 1 is dependent on target business table 2. For business table 3, based on the lineage map, the master data determination platform can determine that data items A and B in source data item set 1 are dependent on target business table 3, but data item C is not. Therefore, the dependency frequency of source data item set 1 can be determined to be 2.

[0093] Source data item set 2

[0094] For business table 1, based on the lineage map, the master data determination platform can determine that every source data item in source data item set 2 is dependent on target business table 1. For business table 2, based on the lineage map, the master data determination platform can determine that data item D in source data item set 2 is not dependent on target business table 2. For business table 3, based on the lineage map, the master data determination platform can determine that data item D in source data item set 2 is not dependent on target business table 3. Therefore, the dependency frequency of source data item set 2 can be determined to be 1.

[0095] Source data item set 3

[0096] For business table 1, based on the lineage map, the master data determination platform can determine that every source data item in source data item set 3 is dependent on target business table 1. For business table 2, based on the lineage map, the master data determination platform can determine that every source data item in source data item set 3 is dependent on target business table 1. For business table 3, based on the lineage map, the master data determination platform can determine that every source data item in source data item set 3 is dependent on target business table 1. Therefore, the dependency frequency of source data item set 3 can be determined to be 3.

[0097] It can be seen that when all source data items in the same source data item set are dependent on the target business table, it is determined that the source data item set is dependent on the target business table. In this way, determining the frequency of dependence of the source data item set based on the business table can accurately represent the frequency of dependence of each source data item in the source data item set, that is, it can accurately determine the coreness and importance of each source data item, thereby improving the accuracy of the determined master data items.

[0098] In another implementation, for each set of source data items, the frequency with which each target business table depends on each set of source data items is obtained in the following manner: for each source data item, the number of times the source data item is depended on by each target business table is obtained, and the number of times the source data item with the smallest number of times in the set of source data items is depended on by each target business table is determined as the frequency with which the source data item set is depended on by each target business table.

[0099] For example, let's take the source data item set 2 described above, which includes data item A, data item B, data item C, and data item D. Data item A in source data item set 2 is dependent on target business table 1, target business table 2, and target business table 3, respectively. Therefore, the number of times data item A is dependent on each target business table is 3. Data item B in source data item set 2 is dependent on target business table 1, target business table 2, and target business table 3, respectively. Therefore, the number of times data item B is dependent on each target business table is 3. Data item C in source data item set 2 is dependent on target business table 1 and target business table 2, respectively. Therefore, the number of times data item A is dependent on each target business table is 2. Data item D in source data item set 2 is dependent on target business table 1, and the number of times data item A is dependent on each target business table is 1. Clearly, the source data item with the smallest number of dependencies in source data item set 2 is dependent on by each target business table 2, and the frequency of this source data item set being dependent on by each target business table is 2.

[0100] Step S104: Based on the frequency of dependency of each source data item set, determine the master data items under the target master data theme from the source data items included in each source data item set.

[0101] From the above, it can be seen that the target business table of the target business system and the source business table of the source business system are determined based on the data lineage map. In this way, the target business table can be used as a basis to determine the master data items under the target master data theme from the source data items included in the source business table. In this way, quickly and accurately determining the master data items can reduce labor costs and improve the efficiency of determining master data. In addition, the frequency of source data items being relied upon by each target business table can be characterized as follows: the globality, sharing, stability, long-term, importance, and coreness of the source data items in multiple business systems deployed on the data processing platform. The frequency of source data items being relied upon by each target business table is used as the judgment basis. The frequency of dependence can be used as data support for determining the master data items, thereby improving the credibility of the determined master data items, reducing repeated negotiations between staff members, reducing personnel communication costs, and improving the efficiency of determining master data.

[0102] The following describes the method of determining the master data item in step S104.

[0103] In one implementation, the master data determination platform may determine a candidate source data item set whose frequency of dependency on each source data item set is greater than a preset frequency threshold, and determine the master data items under the target master data subject from the source data items included in the candidate source data item set.

[0104] In another implementation, the master data determination platform may determine the master data items according to the following steps A and B.

[0105] Step A: From each set of source data items, determine a first set whose frequency of dependence is greater than a preset first threshold and a second set whose frequency of dependence is less than or equal to the first threshold and greater than a preset second threshold.

[0106] Assume that the source data item sets obtained include: source data item set 3 including data item A, data item B, data item E, data item F, and data item L, the frequency of dependency of source data item set 3 is 3, source data item set 4 including data item A, data item B, data item E, data item F, data item H, and data item I, the frequency of dependency of source data item set 4 is 2, source data item set 5 including data item A, data item B, data item C, data item E, data item F, source data item set 6 including data item A, data item B, data item C, data item E, data item F, source data item set 7 including data item A, data item B, data item C, data item E, data item F, source data item set 8 including data item A, data item B, data item C, data item E, data item F, source data item set 9 including data item A, data item B, data item C, data item E, data item F, source data item set 10 including data item A, data item B, data item C, data item E, data item F, source data item set 11 including data item A, data item B, data item C, data item E, data item F, source data item set 12 including data item A, data item B, data item C, data item E, data item F, source data item set 13 including data item A, data item The frequency of dependence of source data item set 5 is 2. The frequency of dependence of source data item set 6 including data item A, data item B, data item E, data item F, and data item J is 2. The frequency of dependence of source data item set 6 is 2. The frequency of dependence of source data item set 7 including data item D is 1. The frequency of dependence of source data item set 8 including data item G is 1. The frequency of dependence of source data item set 8 is 1. The frequency of dependence of source data item set 9 including data item K is 1.

[0107] If the first threshold is 2 and the second threshold is 1, then the first set includes source data item set 3, and the second set includes source data item set 4, source data item set 5, and source data item set 6.

[0108] If the first threshold is 1 and the second threshold is 0, then the first set includes source data item set 3, source data item set 4, source data item set 5, and source data item set 6, and the second set includes source data item set 7, source data item set 8, and source data item set 9.

[0109] Step B: Determine the master data items under the target master data subject based on the source data items included in the first set and the source data items included in the second set.

[0110] Case 1

[0111] If there are no duplicate source data items between the first set and the second set, the source data items included in the first set are determined as core data items of the master data under the target master data subject, and the source data items included in the second set are determined as subsidiary data items of the master data under the target master data subject.

[0112] For example, if the first threshold is 1 and the second threshold is 0, then the first set includes source data item set 3, source data item set 4, source data item set 5, and source data item set 6, and the second set includes source data item set 7, source data item set 8, and source data item set 9.

[0113] Then, the source data items included in the first set are: data item A, data item B, data item C, data item E, data item F, data item H, data item I, data item J, and data item L. The source data items included in the second set are: data item D, data item G, and data item K. In this case, there are no duplicate source data items between the first and second sets. Therefore, data item A, data item B, data item C, data item E, data item F, data item H, data item I, and data item J are determined as core data items of the master data under the target master data theme, and data item D, data item G, and data item K are determined as subsidiary data items of the master data under the target master data theme.

[0114] Case 2

[0115] If there are duplicate source data items between the first set and the second set, the data items of the master data are determined according to the following step B1 or step B2.

[0116] For example, if the first threshold is 2 and the second threshold is 1, then the first set includes source data item set 3, and the second set includes source data item set 4, source data item set 5, and source data item set 6. Therefore, the source data items included in the first set are: data item A, data item B, data item E, data item F, and data item L. The source data items included in the second set are: data item A, data item B, data item C, data item E, data item F, data item H, data item I, and data item J. The source data items that are repeated between the first and second sets are: data item A, data item B, data item E, and data item F.

[0117] Step B1: Determine the source data items included in the first set as core data items of the master data under the target master data subject, and determine the source data items in the second set that do not exist in the first set as subsidiary data items of the master data under the target master data subject.

[0118] For example, according to the above example, the core data items determined by step B1 are: data item A, data item B, data item E, data item F, data item L, and the subsidiary data items are: data item C, data item H, data item I, data item J.

[0119] Step B2: Determine the source data items that are repeated between the first set and the second set as the core data items of the master data under the target master data theme, and determine the source data items that are not repeated between the first set and the second set as the subsidiary data items of the master data under the target master data theme.

[0120] For example, according to the above example, the core data items determined by step B2 are: data item A, data item B, data item E, data item F, and the subsidiary data items are: data item C, data item H, data item I, data item J, data item L.

[0121] In this way, the main data items are distinguished into core data items and ancillary data items. Based on this, the main data items can be distinguished more reasonably, making the determined main data items more detailed, so as to facilitate subsequent different processing of the core data items and ancillary data items in the main data items, thereby improving the applicability of the main data items.

[0122] As can be seen from the above, by using different first thresholds and second thresholds, a first set and a second set with different frequencies of reliance are determined from each set of source data items. The data of the source data items in such first set and second set are used at different frequencies. The master data items determined based on the first set in which the data of the source data items are used more frequently are more core, important, representative, stable, and versatile, etc., while the master data items determined based on the second set in which the data of the source data items are used less frequently can reduce the omission of source data items with higher importance but lower usage frequency, thereby improving the accuracy of the determined master data items.

[0123] The following describes the method of determining the source service table and the target service table in step S101.

[0124] In one embodiment of the present invention, see Figure 3 A flowchart of a method for determining a business table is provided, wherein the method includes the following steps S301-S304.

[0125] Step S301: Obtain data items corresponding to the target master data subject.

[0126] In one implementation, a correspondence can be established between master data themes and data items. Each data item can be assigned a tag that records the master data theme to which it belongs. The master data identification platform can determine which data items belong to the target master data theme based on the data item's tag.

[0127] Step S302: Based on the data lineage map of the data processing platform, determine the first business table to which the upstream data item of the obtained data item belongs, the second business table to which the downstream data item belongs, and the third business table to which the obtained data item belongs.

[0128] In one implementation, the master data determination platform can determine the first business table and the second business table based on the data lineage map of the data processing platform, the dependency and dependent relationships of the obtained data items, and the dependency and dependent relationships of the obtained data items.

[0129] Specifically, based on the dependency and dependent relationships of the obtained data items, the data determination platform identifies the data items that depend on the obtained data items as downstream data items, and identifies the business table to which the downstream data items belong as the second business table. The data determination platform may identify the data items that the obtained data items depend on as upstream data items, and identify the business table to which the upstream data items belong as the first business table.

[0130] Step S303: Determine the source business table of the source business system based on the first business table and the third business table.

[0131] In one implementation, the data determination platform may use the first business table and the third business table as source business tables of the source business system.

[0132] In another implementation, the data determination platform may use the first business table as the source business table of the source business system, and determine, in the third business table, the third business table to which the data item including the non-existent upstream data item belongs as the source business table of the source business system.

[0133] Step S304: determining a target business table of the target business system based on the second business table and the third business table.

[0134] In one implementation, the data determination platform may determine a business table that is not determined as a source business table in the second business table and the third business table as a target business table of the target business system.

[0135] In another implementation, the data determination platform may use the second business table as the target business table of the target business system, and in the third business table, determine the third business table to which the data items including upstream data items and no downstream data items belong as the target business table of the target business system.

[0136] From the above, it can be seen that by comprehensively considering the first business table of the upstream data item of the obtained data item, the second business table to which the downstream data item belongs, and the third business table to which the obtained data item belongs, the source business table of the source business system and the target business table of the target business system are determined. This can comprehensively and systematically determine the source data items used to determine the master data items, reduce the situation of missing data items, and the accurately determined target business table of the target business system can provide a basis for obtaining the frequency of each source data item set being relied upon by each target business table, thereby improving the accuracy of determining the master data items.

[0137] In the technical solution provided by the embodiment of the present invention, the operations involved in obtaining, storing, using, processing, transmitting, providing and disclosing user personal information and company information are all carried out with authorization.

[0138] Corresponding to the above-mentioned method for determining a master data item, an embodiment of the present invention further provides a device for determining a master data item.

[0139] In one embodiment of the present invention, see Figure 4 A schematic diagram of the structure of a device for determining a master data item is provided, the device comprising:

[0140] A business table determination module 401 is configured to determine a source business table of a source business system and a target business table of a target business system based on a target master data theme and a data lineage map of a data processing platform, wherein the data processing platform is deployed with multiple business systems and the data lineage map represents the dependency relationships between data items in the business tables of the multiple business systems;

[0141] A source data item partitioning module 402 is configured to partition source data items in a source business table to obtain multiple source data item sets;

[0142] The frequency obtaining module 403 is used to obtain the frequency of each source data item set being dependent on each target business table based on the blood relationship map;

[0143] The data item determination module 404 is configured to determine the master data items under the target master data theme from the source data items included in each source data item set based on the frequency of dependency of each source data item set.

[0144] From the above, it can be seen that the target business table of the target business system and the source business table of the source business system are determined based on the data lineage map. In this way, the target business table can be used as a basis to determine the master data items under the target master data theme from the source data items included in the source business table. In this way, quickly and accurately determining the master data items can reduce labor costs and improve the efficiency of determining master data. In addition, the frequency of source data items being relied upon by each target business table can be characterized as follows: the globality, sharing, stability, long-term, importance, and coreness of the source data items in multiple business systems deployed on the data processing platform. The frequency of source data items being relied upon by each target business table is used as the judgment basis. The frequency of dependence can be used as data support for determining the master data items, thereby improving the credibility of the determined master data items, reducing repeated negotiations between staff members, reducing personnel communication costs, and improving the efficiency of determining master data.

[0145] In one embodiment of the present invention, the source data item partitioning module is specifically used to determine, from the source data items in the source business table, source data items that have a dependency relationship with the target data items in the target business table based on the data lineage map, and partition the determined source data items to obtain multiple source data item sets; or partition all source data items in the source business table to obtain multiple source data item sets.

[0146] In this way, based on the target data item in the target business table, source data items in the source business table that have a dependency relationship with the target data item in the target business table are determined. This allows accurate identification of source data items from a large number of source data items as candidate master data items, reducing the number of data items that need to be calculated and improving the efficiency of determining master data items. Alternatively, it can ensure that all source data items are included in each source data item set, comprehensively covering all source data items and reducing the possibility of missing data items.

[0147] In one embodiment of the present invention, the frequency acquisition module is specifically used to obtain, for each source data item set, the frequency of each source data item set being dependent on by each target business table in the following manner: for each target business table, based on the data lineage map, determine whether each source data item in the source data item set is dependent on the target business table; if all are dependent on the target business table, determine that the target business table is the dependent table of the source data item set; and count the number of dependent tables of the source data item set as the frequency of the source data item set being dependent.

[0148] It can be seen that when all source data items in the same source data item set are dependent on the target business table, it is determined that the source data item set is dependent on the target business table. In this way, determining the frequency of dependence of the source data item set based on the business table can accurately represent the frequency of dependence of each source data item in the source data item set, that is, it can accurately determine the coreness and importance of each source data item, thereby improving the accuracy of the determined master data items.

[0149] In one embodiment of the present invention, the data item determination module is specifically used to determine, from each set of source data items, a first set whose frequency of dependence is greater than a preset first threshold and a second set whose frequency of dependence is less than or equal to the first threshold and greater than a preset second threshold; and determine the master data items under the target master data subject based on the source data items included in the first set and the source data items included in the second set.

[0150] As can be seen from the above, by using different first thresholds and second thresholds, a first set and a second set with different frequencies of reliance are determined from each set of source data items. The data of the source data items in such first set and second set are used at different frequencies. The master data items determined based on the first set in which the data of the source data items are used more frequently are more core, important, representative, stable, and versatile, etc., while the master data items determined based on the second set in which the data of the source data items are used less frequently can reduce the omission of source data items with higher importance but lower usage frequency, thereby improving the accuracy of the determined master data items.

[0151] In one embodiment of the present invention, the data item determination module is specifically used to determine the source data items included in the first set as the core data items of the master data under the target master data subject, and determine the source data items included in the second set as the subsidiary data items of the master data under the target master data subject if there are no repeated source data items between the first set and the second set; if there are repeated source data items between the first set and the second set, determine the source data items included in the first set as the core data items of the master data under the target master data subject, and determine the source data items in the second set that do not exist in the first set as the subsidiary data items of the master data under the target master data subject; or determine the repeated source data items between the first set and the second set as the core data items of the master data under the target master data subject, and determine the non-repeated source data items between the first set and the second set as the subsidiary data items of the master data under the target master data subject.

[0152] In this way, the main data items are distinguished into core data items and ancillary data items. Based on this, the main data items can be distinguished more reasonably, making the determined main data items more detailed, so as to facilitate subsequent different processing of the core data items and ancillary data items in the main data items, thereby improving the applicability of the main data items.

[0153] In one embodiment of the present invention, the business table determination module is specifically used to obtain data items corresponding to the target master data subject; based on the data lineage map of the data processing platform, determine the first business table to which the upstream data item of the obtained data item belongs, the second business table to which the downstream data item belongs, and the third business table to which the obtained data item belongs; determine the source business table of the source business system based on the first business table and the third business table; and determine the target business table of the target business system based on the second business table and the third business table.

[0154] From the above, it can be seen that by comprehensively considering the first business table of the upstream data item of the obtained data item, the second business table to which the downstream data item belongs, and the third business table to which the obtained data item belongs, the source business table of the source business system and the target business table of the target business system are determined. This can comprehensively and systematically determine the source data items used to determine the master data items, reduce the situation of missing data items, and the accurately determined target business table of the target business system can provide a basis for obtaining the frequency of each source data item set being relied upon by each target business table, thereby improving the accuracy of determining the master data items.

[0155] The embodiment of the present invention further provides an electronic device, such as Figure 5As shown, it includes a processor 501 , a communication interface 502 , a memory 503 and a communication bus 504 , wherein the processor 501 , the communication interface 502 and the memory 503 communicate with each other via the communication bus 504 .

[0156] Memory 503, used for storing computer programs;

[0157] The processor 501 is configured to implement any of the above methods for determining a master data item when executing a program stored in the memory 503 .

[0158] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0159] The communication interface is used for communication between the above electronic device and other devices.

[0160] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0161] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0162] In another embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for determining master data items are implemented.

[0163] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the master data item determination methods in the above embodiments.

[0164] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0165] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0166] Each embodiment in this specification is described in a related manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device, electronic device, storage medium, and computer program product embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For related portions, reference can be made to the descriptions of the method embodiments.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A method for determining a master data item, characterized in that: The method comprises: Determining a source business table of a source business system and a target business table of a target business system based on a target master data theme and a data lineage map of a data processing platform, wherein the data processing platform is deployed with multiple business systems, and the data lineage map represents the dependency relationships between data items in the business tables of the multiple business systems; Divide the source data items in the source business table to obtain multiple source data item sets; Based on the bloodline graph, obtaining the frequency of each source data item set being dependent on by each target business table; Based on the frequency of dependency of each source data item set, master data items under the target master data theme are determined from the source data items included in each source data item set.

2. The method according to claim 1, characterized in that The source data items in the source business table are divided to obtain multiple source data item sets, including: Based on the data lineage graph, source data items in the source business table that have a dependency relationship with target data items in the target business table are determined, and the determined source data items are divided to obtain multiple source data item sets; or All source data items in the source business table are divided to obtain multiple source data item sets.

3. The method according to claim 1, characterized in that The obtaining, based on the bloodline graph, of the frequency at which each source data item set is relied upon by each target business table includes: For each source data item set, the frequency of each source data item set being depended on by each target business table is obtained in the following manner: For each target business table, based on the data lineage graph, determine whether all source data items in the source data item set are dependent on the target business table; if so, determine that the target business table is a dependent table of the source data item set; Count the number of dependent tables of the source data item set as the frequency of the source data item set being dependent.

4. The method according to any one of claims 1 to 3, characterized in that The determining, based on the frequency of dependency of each source data item set, the master data items under the target master data subject from the source data items included in each source data item set, includes: Determine, from each set of source data items, a first set whose frequency of dependency is greater than a preset first threshold and a second set whose frequency of dependency is less than or equal to the first threshold and greater than a preset second threshold; Based on the source data items included in the first set and the source data items included in the second set, master data items under the target master data subject are determined.

5. The method according to claim 4, characterized in that The determining, based on the source data items included in the first set and the source data items included in the second set, the master data items under the target master data subject includes: If there are no duplicate source data items between the first set and the second set, the source data items included in the first set are determined as core data items of the master data under the target master data subject, and the source data items included in the second set are determined as subsidiary data items of the master data under the target master data subject; If there are repeated source data items between the first set and the second set, the source data items included in the first set are determined as core data items of the master data under the target master data subject, and the source data items in the second set that do not exist in the first set are determined as subsidiary data items of the master data under the target master data subject; or the repeated source data items between the first set and the second set are determined as core data items of the master data under the target master data subject, and the non-repeated source data items between the first set and the second set are determined as subsidiary data items of the master data under the target master data subject.

6. The method according to any one of claims 1 to 3, characterized in that The step of determining the source business table of the source business system and the target business table of the target business system based on the data lineage map of the target master data subject and the data processing platform includes: Obtain the data items corresponding to the target master data subject; Based on the data lineage map of the data processing platform, determine the first business table to which the upstream data item of the obtained data item belongs, the second business table to which the downstream data item belongs, and the third business table to which the obtained data item belongs; Determine a source business table of a source business system based on the first business table and the third business table; A target business table of a target business system is determined based on the second business table and the third business table.

7. A device for determining a master data item, characterized in that: The device comprises: A business table determination module is configured to determine a source business table of a source business system and a target business table of a target business system based on a target master data theme and a data lineage map of a data processing platform, wherein the data processing platform is deployed with multiple business systems and the data lineage map represents the dependency relationships between data items in the business tables of the multiple business systems; A source data item partitioning module is used to partition source data items in a source business table to obtain multiple source data item sets; A frequency acquisition module, configured to obtain, based on the bloodline graph, the frequency at which each source data item set is relied upon by each target business table; The data item determination module is configured to determine the master data items under the target master data theme from the source data items included in each source data item set based on the frequency of dependency of each source data item set.

8. The device according to claim 7, characterized in that The source data item partitioning module is specifically configured to determine, from the source data items in the source business table, source data items that have a dependency relationship with the target data items in the target business table based on the data lineage graph, and partition the determined source data items to obtain multiple source data item sets; or partition all source data items in the source business table to obtain multiple source data item sets; and / or The frequency obtaining module is specifically configured to obtain, for each source data item set, the frequency of each source data item set being dependent on each target business table in the following manner: for each target business table, based on the data lineage graph, determine whether each source data item in the source data item set is all dependent on the target business table; if all source data items are all dependent on the target business table, determine the target business table as a dependent table of the source data item set; and count the number of dependent tables of the source data item set as the frequency of the source data item set being dependent; and / or The data item determination module is specifically configured to determine, from each set of source data items, a first set whose frequency of dependency is greater than a preset first threshold and a second set whose frequency of dependency is less than or equal to the first threshold and greater than a preset second threshold; and determine the master data items under the target master data subject based on the source data items included in the first set and the source data items included in the second set; and / or The data item determination module is specifically configured to, if there are no duplicate source data items between the first set and the second set, determine the source data items included in the first set as core data items of the master data under the target master data subject, and determine the source data items included in the second set as subsidiary data items of the master data under the target master data subject; if there are duplicate source data items between the first set and the second set, determine the source data items included in the first set as core data items of the master data under the target master data subject, and determine the source data items in the second set that do not exist in the first set as subsidiary data items of the master data under the target master data subject; Alternatively, the source data items that are repeated between the first set and the second set are determined as core data items of the master data under the target master data subject, and the source data items that are not repeated between the first set and the second set are determined as subsidiary data items of the master data under the target master data subject; and / or The business table determination module is specifically configured to obtain a data item corresponding to a target master data subject; determine, based on a data lineage graph of a data processing platform, a first business table to which an upstream data item of the obtained data item belongs, a second business table to which a downstream data item belongs, and a third business table to which the obtained data item belongs; and determine a source business table of a source business system based on the first business table and the third business table; A target business table of a target business system is determined based on the second business table and the third business table.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method according to any one of claims 1 to 6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images