A rule-based automated data governance system and method

Abstract

The present invention discloses a rule-based automated data governance system and method, which relates to the technical field of data management. The system of the present invention includes: a historical data analysis and rule library construction module, a real-time requirement analysis and requirement determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module; the historical data analysis and rule library construction module collects and analyzes historical business requirement information and historical data to construct a data governance rule library; the real-time requirement analysis and requirement determination module obtains real-time business requirement information and data to be governed, and determines whether it is new business requirement information; the data governance simulation and effect evaluation module conducts data governance simulation on new requirements based on historical data governance rules and evaluates the data governance effect; the data governance rule adjustment and application module outputs adjustment suggestions according to the evaluation results to support personnel in adjusting and applying real-time data governance rules and optimizing the data governance process.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and particularly to a rule-based automated data governance system and method. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, enterprises and organizations are facing challenges in managing massive and complex data. Data has not only become the core asset for decision-making support but also plays a crucial role in daily operations. Traditional data management methods usually rely on manual intervention and manual processes, making it difficult to meet the real-time and accurate data governance requirements. Especially in the case of extremely large data volumes and diverse data types, manual management is often inefficient and error-prone. Therefore, there is an urgent need for a system that can automate, efficiently, and flexibly perform data governance to cope with changing business needs and complex compliance requirements.

[0003] To solve the above problems, more and more enterprises and research institutions have begun to explore rule-based automated data governance methods in order to improve the efficiency and accuracy of data governance through automated and intelligent means. In the prior art, although some data governance methods can achieve automation to a certain extent, in practical applications, they still face some key challenges, especially in how to handle real-time data streams and emerging business needs. Existing automated data governance systems often rely on a pre-set rule library and perform data governance according to a fixed process. However, with the continuous change of enterprise business needs, especially the continuous emergence of real-time business needs, existing methods are difficult to flexibly adapt and adjust to new requirements dynamically. This has led to a significant decline in the applicability and governance efficiency of the system when dealing with complex and dynamic data environments. Summary of the Invention

[0004] The purpose of the present invention is to provide a rule-based automated data governance system and method to solve the problems raised in the above background art.

[0005] To solve the above technical problems, the present invention provides the following technical solutions:

[0006] A rule-based automated data governance method includes the following steps:

[0007] Step S100. Collect historical business requirement information and historical data through a cloud platform, analyze the historical business requirement information and historical data, and correspond the historical business requirement information with the corresponding historical data; combine the processing process of the historical data to obtain historical data governance rules corresponding to the historical business requirement information, and build a data governance rule library;

[0008] Step S200. Obtain real-time business requirement information and real-time data to be governed, divide the real-time data to be governed according to the real-time business requirement information, and correspond the real-time business requirement information with the corresponding real-time data to be governed; Compare and analyze the real-time business requirement information with the historical business requirement information, and based on the comparison and analysis results, determine whether the real-time business requirement information is new business requirement information;

[0009] Step S300. For the determined new business requirement information, match the new business requirement information with the historical business requirement information; Based on the historical data governance rules of the matched historical business requirement information, simulate the data governance process of the real-time data to be governed, and evaluate the data governance effect according to the feedback results of the simulation;

[0010] Step S400. According to the evaluation results of the data governance effect, and in combination with the matching process between the real-time business requirement information and the matched historical business requirement information, generate adjustment suggestions for the real-time data governance rules corresponding to the new business requirement information, output the adjustment suggestions to relevant personnel, and let the relevant personnel define the real-time data governance rules corresponding to the new business requirement information according to the adjustment suggestions.

[0011] Further, step S100 includes:

[0012] S101. Collect historical business requirement information and historical data through the cloud platform. The historical business requirement information refers to the requirement descriptions and records formed by an enterprise or organization in past business activities and related to business goals and operation requirements. These information usually include content such as business goals, decision-making basis, requirement descriptions, scenario settings, and processing flows; The historical data refers to the actual data generated during past business execution and related to business operations and decision-making; These data are usually directly related to the business requirement information and are the results of business activity execution; For each piece of historical business requirement information, extract keywords to form a set of historical business requirement keywords G, and G = {g1, g2,..., gn}, where g1 represents the first keyword of the historical business requirement information, g2 represents the second keyword of the historical business requirement information, and so on, and gn represents the nth keyword of the historical business requirement information;

[0013] S102. Mark and match the historical data according to the set of historical business requirement keywords G, analyze the mapping relationship between the historical business requirement keywords and the historical data, and calculate the correlation coefficient L(dk, gi) between each piece of historical record information dk in the historical data and each keyword gi. The specific calculation formula is:

[0014] L(dk, gi)=Count((dk, gi) / Length(dk);

[0015] Among them, dk represents the k-th historical record information in the historical data, and k represents the historical record information number in the historical data; gi represents the i-th keyword in the set of historical business requirement keywords, and i takes values from 1 to n; Count((dk,gi) represents the number of times the keyword gi appears in the k-th record information of the historical data; Length(dk) represents the length of the k-th record information of the historical data, such as the number of fields or characters; for each historical record information dk, summarize the correlation coefficient L(dk,gi) between the historical record information dk and all keywords in the set G of historical business requirement keywords and calculate the average value to obtain the average correlation coefficient L0; compare the average correlation coefficient L0 with the correlation threshold L. If L0≥L, then correspond the historical record information dk with the historical business requirement information corresponding to the set G of historical business requirement keywords, otherwise do not perform any processing; traverse all historical record information in the historical data, so as to correspond the historical business requirement information with the corresponding historical data.

[0016] S103. Obtain the processing process of the historical data. According to the processing process of the historical data, obtain the historical data governance rules corresponding to the historical business requirement information, and summarize the historical data governance rules corresponding to the historical business requirement information to form a data governance rule library Z, and Z = {z1, z2,..., zm}, where z1 represents the historical data governance rule corresponding to the first historical business requirement information, z2 represents the historical data governance rule corresponding to the second historical business requirement information, and so on. zm represents the historical data governance rule corresponding to the m-th historical business requirement information, and m represents the data number of the historical business requirement information.

[0017] Further, step S200 includes:

[0018] S201. Obtain the real-time business requirement information and the real-time data to be governed. The real-time data to be governed refers to the data that needs to be subjected to data governance in combination with the real-time business requirement information; according to the analysis method of the historical requirement information for the real-time requirement information, for each real-time business requirement information, form a set S of real-time business requirement keywords, and S = {s1, s2,..., sf}, where s1 represents the first keyword of the real-time business requirement information, s2 represents the second keyword of the real-time business requirement information, and so on. sf represents the f-th keyword of the real-time business requirement information; referring to the analysis process of corresponding the historical business requirement information with the corresponding historical data, correspond the real-time business requirement information with the corresponding real-time data to be governed.

[0019] S202. Extract the set G of historical business requirement keywords corresponding to each piece of historical business requirement information, calculate the similarity between the set G of historical business requirement keywords corresponding to each piece of historical business requirement information and the set S of real-time business requirement keywords corresponding to each piece of real-time business requirement information, and the similarity calculation formula is: C = N(G∩S) / N(G∪S), where N(G∩S) represents the number of keywords in the intersection result of the set G of historical business requirement keywords and the set S of real-time business requirement keywords, and N(G∪S) represents the number of keywords in the union result of the set G of historical business requirement keywords and the set S of real-time business requirement keywords; when the similarity C is equal to 1 or the set S of real-time business requirement keywords is a subset of the set G of historical business requirement keywords, it means that the current real-time business requirement information is not new business requirement information; when the similarity C is less than 1 or the set G of historical business requirement keywords is a subset of the set S of real-time business requirement keywords, go to S203 for further analysis;

[0020] When the similarity is low, it indicates that the real-time business requirement contains some new keywords, which may represent new requirements. However, this does not mean that all cases different from historical requirements are new requirements. Some cases with low similarity may also be due to minor adjustments or different expressions of the same requirement, rather than completely new business requirements. Therefore, in addition to similarity calculation, other supplementary judgment means may be needed to further confirm whether it is a new requirement.

[0021] S203. Calculate the keyword difference rate B, and the specific calculation formula is: B = [N(S) - N(G∩S)] / N(S), where N(S) represents the number of keywords in the real-time requirement information; calculate the semantic association degree Yg between the set X of new keywords and the union of the set G of historical business requirement keywords and the set S of real-time business requirement keywords, and X = S - G, and the calculation formula of the semantic association degree Yg is: Yg = 1 / |X|Σ xj∈X [α·Sim(xj,G∩S)], where Sim(xj,G∩S) represents the semantic similarity between the new keyword xj and the union of the set G of historical business requirement keywords and the set S of real-time business requirement keywords, and α represents the weight parameter; according to the keyword difference rate B and the semantic association degree Yg, calculate the new requirement judgment index P, and P = β×B - γ×Yg, where β and γ respectively represent the weights of the keyword difference rate B and the semantic association degree Yg, and β + γ = 1; if the new requirement judgment index P is greater than or equal to the threshold P0, it means that the current real-time business requirement information is new business requirement information, otherwise, the current real-time business requirement information is not new business requirement information.

[0022] Further, step S300 includes:

[0023] S301. For the newly judged business requirement information, based on the similarity calculation between the historical business requirement information and the real-time business requirement information in S202, select the historical business requirement information with the largest similarity C as the matching result, and C < 1; according to the matched historical business requirement information, search in the data governance rule library to find the corresponding historical data governance rule; according to the corresponding historical data governance rule, simulate the data governance process of the real-time data to be governed corresponding to the newly added business requirement information, and record the simulated data as the simulated data;

[0024] S302. Obtain the feedback result of the simulated data, and combine it with the simulated data to calculate the accuracy improvement index F, and F = Nv / N, where Nv represents the number of records whose accuracy is corrected after data governance in the simulated data, and N represents the total number of records in the simulated data; calculate the missing value ratio Q, and Q = Mv / N, where Mv represents the number of missing records after data governance in the simulated data; calculate the consistency coefficient E, and E = Cs / N, where Cs represents the record data that conforms to data consistency after data governance in the simulated data; comprehensively consider the accuracy improvement index F, the missing value ratio Q, and the consistency coefficient E to calculate the comprehensive evaluation index R, and R = w1 × F - w2 × Q + w3 × E, where w1, w2, and w3 represent the weights of the accuracy improvement index F, the missing value ratio Q, and the consistency coefficient E respectively, and w1 + w2 + w3 = 1.

[0025] Further, step S400 includes:

[0026] S401. Based on the evaluation result of the data governance effect, and combined with the matching process between the real-time business requirement information and the matched historical business requirement information, use the matched historical business requirement information as the basis for adjusting the data governance rule corresponding to the newly added business requirement information; extract the newly added keyword set X, and use the data governance rule adjustment basis and the newly added keyword set X as the adjustment suggestions for the real-time data governance rule corresponding to the newly added business requirement information;

[0027] S402. Output the adjustment suggestions for the real-time data governance rules corresponding to the newly added business requirement information to the relevant personnel. The relevant personnel define the real-time data governance rules corresponding to the newly added business requirement information in combination with the adjustment suggestions; perform data governance on the real-time data to be governed corresponding to the newly added business requirement information according to the real-time data governance rules, so as to obtain real-time data, extract the feedback results of the real-time data, and calculate the comprehensive evaluation index R' of the real-time data with reference to the calculation process of the comprehensive evaluation index R of the simulated data in S302; if R < R' and R' ≥ R0, add the real-time data governance rules to the data governance rule library, where R0 represents the comprehensive evaluation index threshold; if R > R' or R' < R0, output a prompt message to the relevant personnel, and the relevant personnel perform further analysis and adjustment until R < R' and R' ≥ R0 are satisfied, and add the adjusted real-time data governance rules to the data governance rule library.

[0028] A rule-based automated data governance system, including: a historical data analysis and rule library construction module, a real-time requirement analysis and requirement determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module;

[0029] The historical data analysis and rule library construction module collects historical business requirement information and historical data, analyzes the historical business requirement information and historical data, and corresponds the historical business requirement information with the corresponding historical data; combines the processing process of the historical data, so as to obtain the historical data governance rules corresponding to the historical business requirement information, and constructs a data governance rule library;

[0030] The real-time requirement analysis and requirement determination module obtains real-time business requirement information and real-time data to be governed, divides the real-time data to be governed according to the real-time business requirement information, and corresponds the real-time business requirement information with the corresponding real-time data to be governed; compares and analyzes the real-time business requirement information with the historical business requirement information, and determines whether the real-time business requirement information is newly added business requirement information;

[0031] For the newly added business requirement information determined by the data governance simulation and effect evaluation module, match the newly added business requirement information with the historical business requirement information; simulate the data governance process of the real-time data to be governed based on the historical data governance rules of the matched historical business requirement information, and evaluate the data governance effect;

[0032] The data governance rule adjustment and application module generates adjustment suggestions for the real-time data governance rules corresponding to the newly added business requirement information according to the evaluation results of the data governance effect, outputs the adjustment suggestions to the relevant personnel, and the relevant personnel define the real-time data governance rules corresponding to the newly added business requirement information according to the adjustment suggestions.

[0033] Furthermore, the historical data analysis and rule library construction module includes a historical business requirement information analysis unit and a data governance rule library construction unit;

[0034] The historical business requirement information analysis unit collects historical business requirement information and historical data, analyzes the historical business requirement information, extracts a keyword set, calculates the correlation coefficient between historical data records and requirement keywords, and corresponds the historical business requirement information with the historical data;

[0035] The data governance rule library construction unit obtains corresponding historical data governance rules based on the historical business requirement information and its corresponding data records, summarizes all historical data governance rules, and establishes a data governance rule library.

[0036] Furthermore, the real-time requirement analysis and requirement determination module includes a real-time business requirement information analysis unit and a new requirement determination and matching unit;

[0037] The real-time business requirement information analysis unit collects real-time business requirement information and data to be governed, extracts keywords, and correlates the real-time business requirement information with the data to be governed according to the mapping relationship between the historical business requirement information and the data;

[0038] The new requirement determination and matching unit determines whether the real-time business requirement information is new requirement information by calculating the keyword set similarity; analyzes the keyword difference rate and semantic correlation degree to further determine whether the real-time business requirement information is new requirement information.

[0039] Furthermore, the data governance simulation and effect evaluation module includes a data governance simulation unit and a data governance effect evaluation unit;

[0040] The data governance simulation unit performs data governance simulation on the real-time data to be governed corresponding to the new business requirement information through the matched historical data governance rules, and records the simulated data as simulated data;

[0041] The data governance effect evaluation unit performs feedback analysis on the simulated data, calculates the accuracy improvement, missing value ratio, and consistency coefficient, and obtains a comprehensive evaluation index based on the accuracy improvement, missing value ratio, and consistency coefficient.

[0042] Furthermore, the data governance rule adjustment and application module includes a real-time data governance rule adjustment unit and a real-time data governance rule application unit;

[0043] The real-time data governance rule adjustment unit generates adjustment suggestions for the real-time data governance rules according to the data governance effect evaluation results and the new business requirement information, outputs the adjustment suggestions to relevant personnel, and they define and adjust the real-time data governance rules according to the adjustment suggestions;

[0044] The real-time data governance rule application unit performs actual data governance on the real-time data to be governed according to the defined real-time data governance rules, extracts the feedback results of the governed real-time data, and calculates comprehensive evaluation indicators; according to the evaluation results of the real-time data governance effect, relevant personnel perform corresponding processing and add the real-time data governance rules to the data governance rule library.

[0045] Compared with the prior art, the beneficial effects of the present invention are as follows: By comparing and analyzing the real-time business requirement information and historical business requirement information, the present invention can identify and judge new business requirements in real time, and simulate and adjust based on historical data governance rules, greatly improving the system's ability to handle dynamic business requirements. In the simulation and feedback stage, by comprehensively considering the accuracy improvement index, the proportion of missing values, and the consistency coefficient, the present invention can accurately evaluate the data governance effect and dynamically adjust the governance rules according to the evaluation results; this comprehensive evaluation and optimization mechanism significantly improves the accuracy and consistency of data governance, avoiding the problem of low governance efficiency caused by fixed rules being unable to adapt to new requirements. By analyzing the relationship between historical business requirement information and historical data, the present invention extracts historical data governance rules and constructs a rule library; with the help of this rule library, the system can quickly respond to new requirements based on historical experience and effectively govern in the real-time data stream; this method makes full use of the experience of historical data governance, reduces the repeated definition and adjustment of real-time data governance rules, and improves the automation level. When generating adjustment suggestions for real-time data governance rules, the present invention can output the adjustment suggestions to relevant personnel for real-time adjustment by combining with the suggestions; this combination of automation and manual intervention not only maintains a high degree of automation but also ensures that in the face of complex and uncertain business requirements, the advantages of manual judgment can be fully utilized. By dynamically partitioning, comparing, and analyzing the keyword differences of real-time business requirement information, the present invention can continuously learn and adjust to adapt to changes in different types and scales of business requirements, enhancing the system's adaptability and scalability in different application scenarios; as the enterprise's requirements continue to evolve, the system can achieve long-term stable and efficient operation by continuously updating the data governance rule library. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings are used to provide a further understanding of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention and do not constitute a limitation to the present invention. In the drawings:

[0047] Figure 1 It is a schematic diagram of the modules of an automated data governance system based on rules according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0049] Please refer to Figure 1 , the present invention provides a technical solution:

[0050] A rule-based automated data governance system, including: a historical data analysis and rule library construction module, a real-time requirement analysis and requirement determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module;

[0051] The historical data analysis and rule library construction module collects historical business requirement information and historical data, analyzes the historical business requirement information and historical data, and corresponds the historical business requirement information with the corresponding historical data; combines the processing process of the historical data to obtain the historical data governance rules corresponding to the historical business requirement information, and constructs a data governance rule library;

[0052] The real-time requirement analysis and requirement determination module obtains real-time business requirement information and real-time data to be governed, divides the real-time data to be governed according to the real-time business requirement information, and corresponds the real-time business requirement information with the corresponding real-time data to be governed; compares and analyzes the real-time business requirement information with the historical business requirement information, and determines whether the real-time business requirement information is new business requirement information;

[0053] For the new business requirement information determined by the data governance simulation and effect evaluation module, matches the new business requirement information with the historical business requirement information; based on the historical data governance rules of the matched historical business requirement information, simulates the data governance process of the real-time data to be governed, and evaluates the data governance effect;

[0054] The data governance rule adjustment and application module generates adjustment suggestions for the real-time data governance rules corresponding to the new business requirement information according to the evaluation results of the data governance effect, outputs the adjustment suggestions to relevant personnel, and the relevant personnel define the real-time data governance rules corresponding to the new business requirement information according to the adjustment suggestions.

[0055] The historical data analysis and rule library construction module includes a historical business requirement information analysis unit and a data governance rule library construction unit;

[0056] The historical business requirement information analysis unit collects historical business requirement information and historical data, analyzes the historical business requirement information, extracts a keyword set, calculates the correlation coefficient between historical data records and requirement keywords, and corresponds the historical business requirement information with the historical data;

[0057] Based on the historical business requirement information and its corresponding data records, the data governance rule library construction unit obtains corresponding historical data governance rules, summarizes all historical data governance rules, and establishes a data governance rule library.

[0058] The real-time requirement analysis and requirement determination module includes a real-time business requirement information analysis unit and a new requirement determination and matching unit;

[0059] The real-time business requirement information analysis unit collects real-time business requirement information and data to be governed, extracts keywords, and correlates the real-time business requirement information with the data to be governed according to the mapping relationship between the historical business requirement information and the data;

[0060] The new requirement determination and matching unit determines whether the real-time business requirement information is new requirement information by calculating the keyword set similarity; analyzes the keyword difference rate and semantic correlation degree to further determine whether the real-time business requirement information is new requirement information.

[0061] The data governance simulation and effect evaluation module includes a data governance simulation unit and a data governance effect evaluation unit;

[0062] The data governance simulation unit performs data governance simulation on the real-time data to be governed corresponding to the new business requirement information through the matched historical data governance rules, and records the simulated data as simulated data;

[0063] The data governance effect evaluation unit conducts feedback analysis on the simulated data, calculates the accuracy improvement, missing value ratio, and consistency coefficient, and obtains a comprehensive evaluation index based on the accuracy improvement, missing value ratio, and consistency coefficient.

[0064] The data governance rule adjustment and application module includes a real-time data governance rule adjustment unit and a real-time data governance rule application unit;

[0065] The real-time data governance rule adjustment unit generates adjustment suggestions for the real-time data governance rules according to the data governance effect evaluation results and the new business requirement information, outputs the adjustment suggestions to relevant personnel, and they define and adjust the real-time data governance rules according to the adjustment suggestions;

[0066] The real-time data governance rule application unit performs actual data governance on the real-time data to be governed according to the defined real-time data governance rules, extracts the feedback results of the governed real-time data, and calculates comprehensive evaluation indicators; according to the evaluation results of the real-time data governance effect, relevant personnel perform corresponding processing and add the real-time data governance rules to the data governance rule library.

[0067] A rule-based automated data governance method includes the following steps:

[0068] Step S100. Collect historical business requirement information and historical data through a cloud platform, analyze the historical business requirement information and historical data, and correspond the historical business requirement information with the corresponding historical data; combine the processing process of the historical data to obtain the historical data governance rules corresponding to the historical business requirement information, and construct a data governance rule library.

[0069] Step S200. Obtain real-time business requirement information and real-time data to be governed, divide the real-time data to be governed according to the real-time business requirement information, and correspond the real-time business requirement information with the corresponding real-time data to be governed; compare and analyze the real-time business requirement information with the historical business requirement information, and based on the comparison and analysis results, judge whether the real-time business requirement information is new business requirement information.

[0070] Step S300. For the judged new business requirement information, match the new business requirement information with the historical business requirement information; based on the historical data governance rules of the matched historical business requirement information, simulate the data governance process of the real-time data to be governed, and evaluate the data governance effect according to the feedback results of the simulation.

[0071] Step S400. According to the evaluation results of the data governance effect and combining the matching process between the real-time business requirement information and the matched historical business requirement information, generate adjustment suggestions for the real-time data governance rules corresponding to the new business requirement information, output the adjustment suggestions to relevant personnel, and relevant personnel define the real-time data governance rules corresponding to the new business requirement information according to the adjustment suggestions.

[0072] Step S100 includes:

[0073] S101. Collect historical business requirement information and historical data through the cloud platform. The historical business requirement information refers to the requirement descriptions and records formed in the past business activities of an enterprise or organization, which are related to business goals and operational requirements. Such information usually includes content such as business goals, decision-making bases, requirement descriptions, scenario settings, and processing procedures. The historical data refers to the actual data generated during the past business execution process, which is related to business operations and decision-making. This data is usually directly related to the historical business requirement information and is the result of business activity execution. For each piece of historical business requirement information, keyword extraction is performed to form a historical business requirement keyword set G, and G = {g1, g2,..., gn}, where g1 represents the first keyword of the historical business requirement information, g2 represents the second keyword of the historical business requirement information, and so on. gn represents the nth keyword of the historical business requirement information.

[0074] S102. Mark and match the historical data according to the historical business requirement keyword set G, analyze the mapping relationship between the historical business requirement keywords and the historical data, and calculate the correlation coefficient L(dk, gi) between each historical record information dk in the historical data and each keyword gi. The specific calculation formula is:

[0075] L(dk, gi) = Count((dk, gi) / Length(dk);

[0076] Where dk represents the kth historical record information in the historical data, and k represents the historical record information number in the historical data; gi represents the ith keyword in the historical business requirement keyword set, and i takes values from 1 to n; Count((dk, gi) represents the number of times the keyword gi appears in the kth record information of the historical data; Length(dk) represents the length of the kth record information of the historical data, such as the number of fields or characters. For each historical record information dk, summarize the correlation coefficients L(dk, gi) between the historical record information dk and all keywords in the historical business requirement keyword set G and calculate the average value to obtain the average correlation coefficient L0. Compare the average correlation coefficient L0 with the correlation threshold L. If L0 ≥ L, then correspond the historical record information dk with the historical business requirement information corresponding to the historical business requirement keyword set G, otherwise do not perform any processing. Traverse all historical record information in the historical data to correspond the historical business requirement information with the corresponding historical data.

[0077] S103. Obtain the processing process of historical data. According to the processing process of historical data, obtain the historical data governance rules corresponding to the historical business requirement information, and summarize the historical data governance rules corresponding to the historical business requirement information to form a data governance rule library Z, and Z = {z1, z2,..., zm}, where z1 represents the historical data governance rule corresponding to the 1st historical business requirement information, z2 represents the historical data governance rule corresponding to the 2nd historical business requirement information, and so on. zm represents the historical data governance rule corresponding to the mth historical business requirement information, and m represents the data number of the historical business requirement information.

[0078] In this embodiment, assume that the historical business requirement information is the approval process, loan interest rate requirements, approval standards, etc. when the bank processes loan approvals. The specific historical data includes data such as historical loan records, approval time, customer type, and loan amount.

[0079] Extract keywords from each piece of historical business requirement information to form a set G of historical business requirement keywords. For example, the keywords of a certain piece of historical business requirement information are: g1 = "loan approval", g2 = "interest rate", g3 = "customer credit", g4 = "amount"; then the corresponding set G of historical business requirement keywords = {g1, g2, g3, g4}.

[0080] Mark and match the historical data according to the set G of historical business requirement keywords, and analyze the mapping relationship between the historical business requirement keywords and the historical data. Assume that a certain historical information record d1 includes the following data:

[0081] d1 = {"loan number": "001", "customer name": "Zhang San", "loan amount": 10000, "interest rate": "4.5", "approval time": "2024-01-15"};

[0082] Calculate the correlation coefficient L(dk, gi):

[0083] Since g1 = "loan approval", and the results generated by the loan approval operation include loan number, customer name, loan amount, interest rate, and approval time, the judgment process can be analyzed by semantic analysis and context pattern matching. Therefore, L(d1, g1) = 1;

[0084] Since g2 = "interest rate", and the interest rate operation corresponds to the interest rate in d1, therefore L(d1, g2) = 1 / 5 = 0.2;

[0085] Since g3 = "customer credit", and customer credit is directly related to customer name and loan amount, therefore L(d1, g3) = 2 / 5 = 0.4;

[0086] Since g4 = "quota", and since the quota is directly related to the loan amount, L(d1, g4) = 1 / 5 = 0.2;

[0087] Then the average correlation coefficient L0 corresponding to this historical business requirement information is L0 = (1 + 0.2 + 0.4 + 0.2) / 4 = 0.45;

[0088] Assume that the correlation threshold L = 0.3, and L0 > L. Therefore, the historical record d1 corresponds to this historical business requirement information; obtain the processing process of the historical data corresponding to the historical record d1, and obtain the data governance rule z of this historical business requirement information for the historical data corresponding to the historical record d1.

[0089] Step S200 includes:

[0090] S201. Obtain real-time business requirement information and real-time data to be governed. The real-time data to be governed refers to the data that needs to be governed in combination with the real-time business requirement information; process the real-time requirement information in the same analysis manner as the historical requirement information. Thus, for each real-time business requirement information, a real-time business requirement keyword set S is formed, and S = {s1, s2,..., sf}, where s1 represents the first keyword of the real-time business requirement information, s2 represents the second keyword of the real-time business requirement information, and so on, and sf represents the f-th keyword of the real-time business requirement information; refer to the corresponding analysis process of the historical business requirement information and the corresponding historical data, and correspond the real-time business requirement information with the corresponding real-time data to be governed;

[0091] S202. Extract the historical business requirement keyword set G corresponding to each historical business requirement information, and calculate the similarity between the historical business requirement keyword set G corresponding to each historical business requirement information and the real-time business requirement keyword set S corresponding to each real-time business requirement information. The similarity calculation formula is: C = N(G ∩ S) / N(G ∪ S), where N(G ∩ S) represents the number of keywords in the intersection result of the historical business requirement keyword set G and the real-time business requirement keyword set S, and N(G ∪ S) represents the number of keywords in the union result of the historical business requirement keyword set G and the real-time business requirement keyword set S; when the similarity C is equal to 1 or the real-time business requirement keyword set S is a subset of the historical business requirement keyword set G, it means that the current real-time business requirement information is not a new business requirement information; when the similarity C is less than 1 or the historical business requirement keyword set G is a subset of the real-time business requirement keyword set S, go to S203 for further analysis;

[0092] When the similarity is low, it indicates that the real-time business requirements contain some new keywords, which may represent new requirements. However, this does not mean that all situations different from historical requirements are new requirements. Some cases with low similarity may also be due to minor adjustments or different expressions of the same requirement, rather than completely new business requirements. Therefore, in addition to similarity calculation, other supplementary judgment methods may be needed to further confirm whether it is a new requirement.

[0093] S203. Calculate the keyword difference rate B, and the specific calculation formula is: B = [N(S) - N(G ∩ S)] / N(S), where N(S) represents the number of keywords in the real-time demand information; calculate the semantic association degree Yg between the set X of new keywords and the union of the keyword set G of historical business requirements and the keyword set S of real-time business requirements, and X = S - G. The calculation formula for the semantic association degree Yg is: Yg = 1 / |X| Σ xj∈X [α · Sim(xj, G ∩ S)], where Sim(xj, G ∩ S) represents the semantic similarity between the new keyword xj and the union of the keyword set G of historical business requirements and the keyword set S of real-time business requirements, and α represents the weight parameter; according to the keyword difference rate B and the semantic association degree Yg, calculate the new requirement judgment index P, and P = β × B - γ × Yg, where β and γ respectively represent the weights of the keyword difference rate B and the semantic association degree Yg, and β + γ = 1; if the new requirement judgment index P is greater than or equal to the threshold P0, it means that the current real-time business requirement information is new business requirement information, otherwise, the current real-time business requirement information is not new business requirement information.

[0094] Step S300 includes:

[0095] S301. For the information judged as new business requirements, according to the similarity calculation between the historical business requirement information and the real-time business requirement information in S202, select the historical business requirement information with the maximum similarity C as the matching result, and C < 1; according to the matched historical business requirement information, search in the data governance rule library to find the corresponding historical data governance rule; according to the corresponding historical data governance rule, simulate the data governance process of the real-time data to be governed corresponding to the new business requirement information, and record the simulated data as the simulated data;

[0096] S302. Obtain the feedback results of the simulation data, and combine the simulation data to calculate the accuracy improvement index F, and F = Nv / N, where Nv represents the number of records with accuracy correction after data governance in the simulation data, and N represents the total number of records in the simulation data; calculate the missing value ratio Q, and Q = Mv / N, where Mv represents the number of missing records after data governance in the simulation data; calculate the consistency coefficient E, and E = Cs / N, where Cs represents the record data that conforms to data consistency after data governance in the simulation data; comprehensively consider the accuracy improvement index F, the missing value ratio Q, and the consistency coefficient E, and calculate the comprehensive evaluation index R, and R = w1×F - w2×Q + w3×E, where w1, w2, and w3 represent the weights of the accuracy improvement index F, the missing value ratio Q, and the consistency coefficient E respectively, and w1 + w2 + w3 = 1.

[0097] In this embodiment, assume that the following results are obtained after the simulation data feedback:

[0098] The accuracy improvement index F = 0.65, which means that 65% of the records have been accuracy-corrected;

[0099] The missing value ratio Q = 0.55, which means that 55% of the records have missing values;

[0100] The consistency coefficient E = 0.4, indicating that 40% of the records meet the data consistency requirements.

[0101] Assume that the given weights are: w1 = 0.4, w2 = 0.3, w3 = 0.3; then, the comprehensive evaluation index R is calculated as:

[0102] R = 0.4×0.65 - 0.3×0.55 + 0.3×0.4 = 0.215.

[0103] Step S400 includes:

[0104] S401. Based on the evaluation results of the data governance effect, and combining the matching process between the real-time business requirement information and the matched historical business requirement information, use the matched historical business requirement information as the basis for adjusting the data governance rules corresponding to the new business requirement information; extract the new keyword set X, and use the basis for adjusting the data governance rules and the new keyword set X as the adjustment suggestions for the real-time data governance rules corresponding to the new business requirement information;

[0105] S402. Output the adjustment suggestions for the real-time data governance rules corresponding to the newly added business requirement information to the relevant personnel. The relevant personnel shall define the real-time data governance rules corresponding to the newly added business requirement information in combination with the adjustment suggestions; perform data governance on the real-time data to be governed corresponding to the newly added business requirement information according to the real-time data governance rules, so as to obtain real-time data, extract the feedback results of the real-time data, and calculate the comprehensive evaluation index R' of the real-time data with reference to the calculation process of the comprehensive evaluation index R of the simulated data in S302; if R > R' or R' < R0, add the real-time data governance rules to the data governance rule library, where R0 represents the comprehensive evaluation index threshold; if R > R' or R' < R0, output a prompt message to the relevant personnel, and the relevant personnel shall conduct further analysis and adjustment until R < R' and R' ≥ R0 are satisfied, and add the adjusted real-time data governance rules to the data governance rule library.

[0106] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device.

[0107] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not used to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A rule-based automated data governance method, characterized by: The method comprises the following steps: Step S100. Collect historical business demand information and historical data through the cloud platform, analyze the historical business demand information and historical data, and correspond the historical business demand information with the corresponding historical data; combine the processing process of historical data to obtain historical data governance rules corresponding to the historical business demand information, and build a data governance rule library; Step S200. Acquire real-time business demand information and real-time data to be governed, divide the real-time data to be governed according to the real-time business demand information, and correspond the real-time business demand information with the corresponding real-time data to be governed; compare and analyze the real-time business demand information with the historical business demand information, and determine whether the real-time business demand information is new business demand information based on the comparison and analysis results; Step S300. For the new business demand information determined to be new, the new business demand information is matched with the historical business demand information; based on the historical data governance rules of the matched historical business demand information, the data governance process of the real-time data to be governed is simulated, and the data governance effect is evaluated according to the feedback results of the simulation; Step S400. Based on the evaluation results of the data governance effect and in combination with the matching process between the real-time business demand information and the matched historical business demand information, generate adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information, and output the adjustment suggestions to relevant personnel, who define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions.

2. A rule-based automated data governance method according to claim 1, characterized in that: The step S100 includes: S101. Collect historical business demand information and historical data through the cloud platform, wherein the historical business demand information refers to demand descriptions and records related to business objectives and operational needs formed by an enterprise or organization in past business activities; the historical data refers to actual data related to business operations and decisions generated in past business execution processes; for each piece of historical business demand information, perform keyword extraction to form a historical business demand keyword set G, where G={g1,g2,...,gn}, where g1 represents the first keyword of the historical business demand information, g2 represents the second keyword of the historical business demand information, and so on, gn represents the nth keyword of the historical business demand information; S102. Mark and match the historical data according to the historical business demand keyword set G, analyze the mapping relationship between the historical business demand keywords and the historical data, and calculate the correlation coefficient L(dk,gi) between each historical record information dk and each keyword gi in the historical data, and the specific calculation formula is: L(dk,gi)=Count((dk,gi) / Length(dk); Wherein, dk represents the kth historical record information in the historical data, k represents the historical record information number in the historical data; gi represents the i-th keyword in the historical business demand keyword set, and i ranges from 1 to n; Count((dk,gi) represents the number of times the keyword gi appears in the kth record information of the historical data; Length(dk) represents the length of the kth record information of the historical data; for each historical record information dk, the correlation coefficients L(dk,gi) between the historical record information dk and all the keywords in the historical business demand keyword set G are summarized and averaged to obtain the average correlation coefficient L0; the average correlation coefficient L0 is compared with the correlation threshold L, if L0≥L, the historical record information dk is matched with the historical business demand information corresponding to the historical business demand keyword set G, otherwise no processing is performed; all historical record information in the historical data are traversed to match the historical business demand information with the corresponding historical data; S103. Obtain the processing process of historical data, and according to the processing process of historical data, obtain the historical data governance rules corresponding to the historical business demand information, and summarize the historical data governance rules corresponding to the historical business demand information to form a data governance rule base Z, and Z={z1,z2,...,zm}, where z1 represents the historical data governance rule corresponding to the first historical business demand information, z2 represents the historical data governance rule corresponding to the second historical business demand information, and so on, zm represents the historical data governance rule corresponding to the mth historical business demand information, and m represents the data number of the historical business demand information.

3. The rule-based automated data governance method according to claim 2, characterized in that: The step S200 includes: S201. Acquire real-time business demand information and real-time data to be governed, wherein the real-time data to be governed refers to data that needs to be governed in combination with the real-time business demand information; analyze the real-time demand information in the same way as the historical demand information, thereby forming a real-time business demand keyword set S for each piece of real-time business demand information, and S={s1,s2,...,sf}, wherein s1 represents the first keyword of the real-time business demand information, s2 represents the second keyword of the real-time business demand information, and so on, sf represents the fth keyword of the real-time business demand information; refer to the historical business demand information and the corresponding historical data for the corresponding analysis process, and correspond the real-time business demand information with the corresponding real-time data to be governed; S202. Extract the historical business demand keyword set G corresponding to each historical business demand information, and calculate the similarity between the historical business demand keyword set G corresponding to each historical business demand information and the real-time business demand keyword set S corresponding to each real-time business demand information, and the similarity calculation formula is: C=N(G∩S) / N(G∪S), where N(G∩S) represents the number of keywords of the intersection result of the historical business demand keyword set G and the real-time business demand keyword set S, and N(G∪S) represents the number of keywords of the union result of the historical business demand keyword set G and the real-time business demand keyword set S; when the similarity C is equal to 1 or the real-time business demand keyword set S is a subset of the historical business demand keyword set G, it means that the current real-time business demand information is not new business demand information; when the similarity C is less than 1 or the historical business demand keyword set G is a subset of the real-time business demand keyword set S, go to S203 for further analysis; S203. Calculate the keyword difference rate B, and the specific calculation formula is: B=[N(S)-N(G∩S)] / N(S), where N(S) represents the number of keywords in the real-time demand information; calculate the semantic relevance Yg between the newly added keyword set X and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and X=SG, the calculation formula of the semantic relevance Yg is: Yg=1 / |X|Σ xj∈X [α·Sim(xj,G∩S)], where Sim(xj,G∩S) represents the semantic similarity between the newly added keyword xj and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and α represents the weight parameter; according to the keyword difference rate B and the semantic relevance Yg, the new demand judgment index P is calculated, and P=β×B-γ×Yg, where β and γ represent the weights of the keyword difference rate B and the semantic relevance Yg respectively, and β+γ=1; if the new demand judgment index P is greater than or equal to the threshold P0, it means that the current real-time business demand information is new business demand information, otherwise, the current real-time business demand information is not new business demand information.

4. The rule-based automated data governance method according to claim 3, characterized in that: The step S300 includes: S301. For the newly added business demand information, according to the similarity calculation between the historical business demand information and the real-time business demand information in S202, the historical business demand information with the largest similarity C is selected as the matching result, and C < 1; according to the matched historical business demand information, the corresponding historical data governance rules are searched in the data governance rule library; according to the corresponding historical data governance rules, the data governance process of the real-time data to be governed corresponding to the newly added business demand information is simulated, and the simulated data is recorded as the simulated data; S302. Obtain the feedback results of the simulation data, and calculate the accuracy improvement index F in combination with the simulation data, and F=Nv / N, where Nv represents the number of records in the simulation data whose accuracy has been corrected after data governance based on the feedback results, and N represents the total number of records in the simulation data; calculate the missing value ratio Q, and Q=Mv / N, where Mv represents the number of missing records in the simulation data after data governance based on the feedback results; calculate the consistency coefficient E, and E=Cs / N, where Cs represents the record data that meets the data consistency after data governance based on the feedback results in the simulation data; comprehensively consider the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E to calculate the comprehensive evaluation index R, and R=w1×F-w2×Q+w3×E, where w1, w2 and w3 represent the weights of the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E, respectively, and w1+w2+w3=1.

5. A rule-based automated data governance method according to claim 4, characterized in that: The step S400 includes: S401. Based on the evaluation results of the data governance effect, and in combination with the matching process between the real-time business demand information and the matched historical business demand information, the matched historical business demand information is used as the basis for adjusting the data governance rules corresponding to the newly added business demand information; the newly added keyword set X is extracted, and the data governance rule adjustment basis and the newly added keyword set X are used as adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information; S402. Output the adjustment suggestions of the real-time data governance rules corresponding to the newly added business demand information to the relevant personnel, who will define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions; perform data governance on the real-time data to be governed corresponding to the newly added business demand information according to the real-time data governance rules, so as to obtain real-time data, extract the feedback results of the real-time data, and calculate the comprehensive evaluation index R' of the real-time data with reference to the calculation process of the comprehensive evaluation index R of the simulated data in S302; if R<R' and R'≥R0, add the real-time data governance rules to the data governance rule library, where R0 represents the comprehensive evaluation index threshold; if R>R' or R'<R0, output prompt information to the relevant personnel, who will conduct further analysis and adjustment until R<R' and R'≥R0 are satisfied, and add the adjusted real-time data governance rules to the data governance rule library.

6. A rule-based automated data governance system, applied to a rule-based automated data governance method according to any one of claims 1 to 5, characterized in that: The system includes: a historical data analysis and rule base construction module, a real-time demand analysis and demand determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module; The historical data analysis and rule base construction module collects historical business demand information and historical data, analyzes the historical business demand information and historical data, and matches the historical business demand information with the corresponding historical data; combines the historical data processing process to obtain the historical data governance rules corresponding to the historical business demand information, and constructs a data governance rule base; The real-time demand analysis and demand determination module obtains real-time business demand information and real-time data to be governed, divides the real-time data to be governed according to the real-time business demand information, and matches the real-time business demand information with the corresponding real-time data to be governed; compares and analyzes the real-time business demand information with the historical business demand information, and determines whether the real-time business demand information is new business demand information; The data governance simulation and effect evaluation module matches the new business demand information with the historical business demand information for the new business demand information judged as new business demand information; simulates the data governance process of the real-time data to be governed based on the historical data governance rules of the matched historical business demand information, and evaluates the data governance effect; The data governance rule adjustment and application module generates adjustment suggestions for real-time data governance rules corresponding to the newly added business demand information based on the evaluation results of the data governance effect, and outputs the adjustment suggestions to relevant personnel, who then define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions.

7. The rule-based automated data governance system according to claim 6, characterized in that: The historical data analysis and rule base construction module includes a historical business demand information analysis unit and a data governance rule base construction unit; The historical business demand information analysis unit collects historical business demand information and historical data, analyzes the historical business demand information, extracts a keyword set, and calculates a correlation coefficient between the historical data record and the demand keyword, and matches the historical business demand information with the historical data; The data governance rule library construction unit obtains corresponding historical data governance rules based on historical business demand information and its corresponding data records, summarizes all historical data governance rules, and establishes a data governance rule library.

8. The rule-based automated data governance system according to claim 6, characterized in that: The real-time demand analysis and demand determination module includes a real-time business demand information analysis unit and a new demand determination and matching unit; The real-time business demand information analysis unit collects real-time business demand information and data to be governed, extracts keywords, and associates the real-time business demand information with the data to be governed according to the mapping relationship between historical business demand information and data; The new demand determination and matching unit determines whether the real-time business demand information is new demand information by calculating the similarity of the keyword set; and further determines whether the real-time business demand information is new demand information by analyzing the keyword difference rate and semantic relevance.

9. The rule-based automated data governance system according to claim 6, characterized in that: The data governance simulation and effect evaluation module includes a data governance simulation unit and a data governance effect evaluation unit; The data governance simulation unit performs data governance simulation on the real-time to-be-governed data corresponding to the newly added business demand information by matching the historical data governance rules, and records the simulated data as simulation data; The data governance effect evaluation unit performs feedback analysis on the simulation data, calculates the accuracy improvement, the missing value ratio and the consistency coefficient, and obtains a comprehensive evaluation index based on the accuracy improvement, the missing value ratio and the consistency coefficient.

10. The rule-based automated data governance system according to claim 6, characterized in that: The data governance rule adjustment and application module includes a real-time data governance rule adjustment unit and a real-time data governance rule application unit; The real-time data governance rule adjustment unit generates adjustment suggestions for the real-time data governance rules based on the data governance effect evaluation results and the newly added business demand information, and outputs the adjustment suggestions to relevant personnel, who define and adjust the real-time data governance rules based on the adjustment suggestions; The real-time data governance rule application unit performs actual data governance on the real-time data to be governed according to the defined real-time data governance rules, extracts the real-time data feedback results after governance, and calculates comprehensive evaluation indicators; according to the evaluation results of the real-time data governance effect, relevant personnel perform corresponding processing and add the real-time data governance rules to the data governance rule library.

Images