A rule-based automated data governance system and method

Abstract

The present invention discloses a rule-based automated data governance system and method, and relates to the technical field of data management. The system of the present invention includes: a historical data analysis and rule base construction module, a real-time demand analysis and demand determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module; the historical data analysis and rule base construction module collects and analyzes historical business demand information and historical data, and constructs a data governance rule base; the real-time demand analysis and demand determination module obtains real-time business demand information and data to be governed, and determines whether it is new business demand information; the data governance simulation and effect evaluation module performs data governance simulation on new demands based on historical data governance rules, and evaluates the data governance effect; the data governance rule adjustment and application module outputs adjustment suggestions according to the evaluation results, supports personnel to adjust and apply real-time data governance rules, and optimizes the data governance process.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to a rule-based automated data governance system and method. Background Art

[0002] With the rapid development of information technology and the advent of the big data era, enterprises and organizations are facing the challenge of managing massive and complex data. Data has not only become a core asset for decision support, but also plays a vital role in daily operations. Traditional data management methods usually rely on human intervention and manual processes, which are difficult to meet the needs of real-time and accurate data governance, especially when the amount of data is huge and the types of data are diverse. Manual management is often inefficient and prone to errors. Therefore, there is an urgent need for a system that can automatically, efficiently and flexibly manage data to cope with changing business needs and complex compliance requirements.

[0003] In order to solve the above problems, more and more enterprises and research institutions have begun to explore rule-based automated data governance methods, hoping to improve the efficiency and accuracy of data governance through automation and intelligent means. In the existing technology, although some data governance methods can be automated to a certain extent, they still face some key challenges in practical applications, especially in how to deal with real-time data streams and new business needs. Existing automated data governance systems often rely on preset rule bases and perform data governance according to fixed processes. However, with the continuous changes in enterprise business needs, especially the continuous emergence of real-time business needs, existing methods are difficult to flexibly adapt and adjust to new needs dynamically. This has led to a significant decrease in the applicability and governance efficiency of the system when dealing with complex and dynamic data environments. Summary of the invention

[0004] The purpose of the present invention is to provide a rule-based automated data governance system and method to solve the problems raised in the above background technology.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: A rule-based automated data governance method includes the following steps: Step S100. Collect historical business demand information and historical data through the cloud platform, analyze the historical business demand information and historical data, and correspond the historical business demand information with the corresponding historical data; combine the processing process of historical data to obtain historical data governance rules corresponding to the historical business demand information, and build a data governance rule library; Step S200. Acquire real-time business demand information and real-time data to be governed, divide the real-time data to be governed according to the real-time business demand information, and correspond the real-time business demand information with the corresponding real-time data to be governed; compare and analyze the real-time business demand information with the historical business demand information, and determine whether the real-time business demand information is new business demand information based on the comparison and analysis results; Step S300. For the new business demand information determined to be new, the new business demand information is matched with the historical business demand information; based on the historical data governance rules of the matched historical business demand information, the data governance process of the real-time data to be governed is simulated, and the data governance effect is evaluated according to the feedback results of the simulation; Step S400. Based on the evaluation results of the data governance effect and in combination with the matching process between the real-time business demand information and the matched historical business demand information, generate adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information, and output the adjustment suggestions to relevant personnel, who define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions.

[0006] Furthermore, step S100 includes: S101. Collect historical business demand information and historical data through the cloud platform. The historical business demand information refers to demand descriptions and records related to business objectives and operational requirements formed by an enterprise or organization in past business activities. Such information usually includes business objectives, decision-making basis, demand descriptions, scenario settings, and processing procedures. The historical data refers to actual data related to business operations and decisions generated in the past business execution process. Such data are usually directly related to business demand information and are the results of business activity execution. For each piece of historical business demand information, keyword extraction is performed to form a historical business demand keyword set G, and G={g1,g2,...,gn}, where g1 represents the first keyword of the historical business demand information, g2 represents the second keyword of the historical business demand information, and so on, gn represents the nth keyword of the historical business demand information. S102. Mark and match the historical data according to the historical business demand keyword set G, analyze the mapping relationship between the historical business demand keywords and the historical data, and calculate the correlation coefficient L(dk,gi) between each historical record information dk and each keyword gi in the historical data, and the specific calculation formula is: L(dk,gi)=Count((dk,gi) / Length(dk); Wherein, dk represents the kth historical record information in the historical data, k represents the historical record information number in the historical data; gi represents the i-th keyword in the historical business demand keyword set, and i ranges from 1 to n; Count((dk,gi) represents the number of times the keyword gi appears in the kth record information of the historical data; Length(dk) represents the length of the kth record information of the historical data, such as the number of fields or the number of characters; for each historical record information dk, the correlation coefficients L(dk,gi) between the historical record information dk and all the keywords in the historical business demand keyword set G are summarized and averaged to obtain the average correlation coefficient L0; the average correlation coefficient L0 is compared with the correlation threshold L, if L0≥L, the historical record information dk is matched with the historical business demand information corresponding to the historical business demand keyword set G, otherwise no processing is performed; all historical record information in the historical data are traversed to match the historical business demand information with the corresponding historical data; S103. Obtain the processing process of historical data, and according to the processing process of historical data, obtain the historical data governance rules corresponding to the historical business demand information, and summarize the historical data governance rules corresponding to the historical business demand information to form a data governance rule base Z, and Z={z1,z2,...,zm}, where z1 represents the historical data governance rule corresponding to the first historical business demand information, z2 represents the historical data governance rule corresponding to the second historical business demand information, and so on, zm represents the historical data governance rule corresponding to the mth historical business demand information, and m represents the data number of the historical business demand information.

[0007] Further, step S200 includes: S201. Acquire real-time business demand information and real-time data to be governed, wherein the real-time data to be governed refers to data that needs to be governed in combination with the real-time business demand information; analyze the real-time demand information in the same way as the historical demand information, thereby forming a real-time business demand keyword set S for each piece of real-time business demand information, and S={s1,s2,...,sf}, wherein s1 represents the first keyword of the real-time business demand information, s2 represents the second keyword of the real-time business demand information, and so on, sf represents the fth keyword of the real-time business demand information; refer to the historical business demand information and the corresponding historical data for the corresponding analysis process, and correspond the real-time business demand information with the corresponding real-time data to be governed; S202. Extract the historical business demand keyword set G corresponding to each historical business demand information, and calculate the similarity between the historical business demand keyword set G corresponding to each historical business demand information and the real-time business demand keyword set S corresponding to each real-time business demand information, and the similarity calculation formula is: C=N(G∩S) / N(G∪S), where N(G∩S) represents the number of keywords of the intersection result of the historical business demand keyword set G and the real-time business demand keyword set S, and N(G∪S) represents the number of keywords of the union result of the historical business demand keyword set G and the real-time business demand keyword set S; when the similarity C is equal to 1 or the real-time business demand keyword set S is a subset of the historical business demand keyword set G, it means that the current real-time business demand information is not new business demand information; when the similarity C is less than 1 or the historical business demand keyword set G is a subset of the real-time business demand keyword set S, go to S203 for further analysis; When the similarity is low, it indicates that the real-time business requirements contain some new keywords, which may represent new requirements. However, this does not mean that all situations that are different from historical requirements are new requirements. Some situations with low similarity may also be due to slight adjustments to the same requirement or different expressions, rather than completely new business requirements. Therefore, in addition to similarity calculation, other supplementary judgment methods may be needed to further confirm whether it is a new requirement.

[0008] S203. Calculate the keyword difference rate B, and the specific calculation formula is: B=[N(S)-N(G∩S)] / N(S), where N(S) represents the number of keywords in the real-time demand information; calculate the semantic relevance Yg between the newly added keyword set X and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and X=SG, the calculation formula of the semantic relevance Yg is: Yg=1 / |X|Σ xj∈X [α·Sim(xj,G∩S)], where Sim(xj,G∩S) represents the semantic similarity between the newly added keyword xj and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and α represents the weight parameter; according to the keyword difference rate B and the semantic relevance Yg, the new demand judgment index P is calculated, and P=β×B-γ×Yg, where β and γ represent the weights of the keyword difference rate B and the semantic relevance Yg respectively, and β+γ=1; if the new demand judgment index P is greater than or equal to the threshold P0, it means that the current real-time business demand information is new business demand information, otherwise, the current real-time business demand information is not new business demand information.

[0009] Furthermore, step S300 includes: S301. For the newly added business demand information, according to the similarity calculation between the historical business demand information and the real-time business demand information in S202, the historical business demand information with the largest similarity C is selected as the matching result, and C < 1; according to the matched historical business demand information, the corresponding historical data governance rules are searched in the data governance rule library; according to the corresponding historical data governance rules, the data governance process of the real-time data to be governed corresponding to the newly added business demand information is simulated, and the simulated data is recorded as the simulated data; S302. Obtain the feedback results of the simulation data, and calculate the accuracy improvement index F in combination with the simulation data, and F=Nv / N, where Nv represents the number of records in the simulation data whose accuracy has been corrected after data governance based on the feedback results, and N represents the total number of records in the simulation data; calculate the missing value ratio Q, and Q=Mv / N, where Mv represents the number of missing records in the simulation data after data governance based on the feedback results; calculate the consistency coefficient E, and E=Cs / N, where Cs represents the record data that meets the data consistency after data governance based on the feedback results in the simulation data; comprehensively consider the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E to calculate the comprehensive evaluation index R, and R=w1×F-w2×Q+w3×E, where w1, w2 and w3 represent the weights of the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E, respectively, and w1+w2+w3=1.

[0010] Furthermore, step S400 includes: S401. Based on the evaluation results of the data governance effect, and in combination with the matching process between the real-time business demand information and the matched historical business demand information, the matched historical business demand information is used as the basis for adjusting the data governance rules corresponding to the newly added business demand information; the newly added keyword set X is extracted, and the data governance rule adjustment basis and the newly added keyword set X are used as adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information; S402. Output the adjustment suggestions of the real-time data governance rules corresponding to the newly added business demand information to the relevant personnel, who will define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions; perform data governance on the real-time data to be governed corresponding to the newly added business demand information according to the real-time data governance rules, so as to obtain real-time data, extract the feedback results of the real-time data, and calculate the comprehensive evaluation index R' of the real-time data with reference to the calculation process of the comprehensive evaluation index R of the simulated data in S302; if R<R' and R'≥R0, add the real-time data governance rules to the data governance rule library, where R0 represents the comprehensive evaluation index threshold; if R>R' or R'<R0, output prompt information to the relevant personnel, who will conduct further analysis and adjustment until R<R' and R'≥R0 are satisfied, and add the adjusted real-time data governance rules to the data governance rule library.

[0011] A rule-based automated data governance system, including: a historical data analysis and rule base construction module, a real-time demand analysis and demand determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module; The historical data analysis and rule base construction module collects historical business demand information and historical data, analyzes the historical business demand information and historical data, and matches the historical business demand information with the corresponding historical data; combined with the historical data processing process, it obtains the historical data governance rules corresponding to the historical business demand information and builds a data governance rule base; The real-time demand analysis and demand determination module obtains real-time business demand information and real-time data to be governed, divides the real-time data to be governed according to the real-time business demand information, and matches the real-time business demand information with the corresponding real-time data to be governed; compares and analyzes the real-time business demand information with the historical business demand information, and determines whether the real-time business demand information is new business demand information; The data governance simulation and effect evaluation module matches the new business demand information with the historical business demand information for the new business demand information. Based on the historical data governance rules of the matched historical business demand information, it simulates the data governance process of the real-time data to be governed and evaluates the data governance effect. The data governance rule adjustment and application module generates adjustment suggestions for real-time data governance rules corresponding to the new business demand information based on the evaluation results of the data governance effect, and outputs the adjustment suggestions to relevant personnel, who then define the real-time data governance rules corresponding to the new business demand information based on the adjustment suggestions.

[0012] Further, the historical data analysis and rule base construction module includes a historical business demand information analysis unit and a data governance rule base construction unit; The historical business demand information analysis unit collects historical business demand information and historical data, analyzes the historical business demand information, extracts a keyword set, and calculates a correlation coefficient between the historical data record and the demand keyword, and matches the historical business demand information with the historical data; The data governance rule base construction unit obtains corresponding historical data governance rules based on historical business demand information and its corresponding data records, summarizes all historical data governance rules, and establishes a data governance rule base.

[0013] Furthermore, the real-time demand analysis and demand determination module includes a real-time business demand information analysis unit and a new demand determination and matching unit; The real-time business demand information analysis unit collects real-time business demand information and data to be governed, extracts keywords, and associates the real-time business demand information with the data to be governed based on the mapping relationship between historical business demand information and data; The new demand determination and matching unit determines whether the real-time business demand information is new demand information by calculating the similarity of the keyword set; and further determines whether the real-time business demand information is new demand information by analyzing the keyword difference rate and semantic relevance.

[0014] Further, the data governance simulation and effect evaluation module includes a data governance simulation unit and a data governance effect evaluation unit; The data governance simulation unit performs data governance simulation on the real-time data to be governed corresponding to the newly added business demand information by matching the historical data governance rules, and records the simulated data as simulation data; The data governance effectiveness evaluation unit conducts feedback analysis on the simulated data, calculates the accuracy improvement, missing value ratio and consistency coefficient, and obtains a comprehensive evaluation index based on the accuracy improvement, missing value ratio and consistency coefficient.

[0015] Further, the data governance rule adjustment and application module includes a real-time data governance rule adjustment unit and a real-time data governance rule application unit; The real-time data governance rule adjustment unit generates adjustment suggestions for real-time data governance rules based on the data governance effect evaluation results and new business demand information, and outputs the adjustment suggestions to relevant personnel, who then define and adjust the real-time data governance rules based on the adjustment suggestions; The real-time data governance rule application unit performs actual data governance on the real-time data to be governed according to the defined real-time data governance rules, extracts the real-time data feedback results after governance, and calculates comprehensive evaluation indicators; according to the evaluation results of the real-time data governance effect, relevant personnel shall perform corresponding processing and add the real-time data governance rules to the data governance rule library.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention can identify and judge new business needs in real time by comparing and analyzing real-time business demand information and historical business demand information, and simulate and adjust based on historical data governance rules, which greatly improves the system's ability to cope with dynamic business needs. In the simulation and feedback stage, the present invention can accurately evaluate the data governance effect by comprehensively improving indicators, missing value ratios, and consistency coefficients, and dynamically adjust governance rules according to the evaluation results; this comprehensive evaluation and optimization mechanism significantly improves the accuracy and consistency of data governance, and avoids the problem of low governance efficiency caused by the inability of fixed rules to adapt to new needs. The present invention analyzes the relationship between historical business demand information and historical data, extracts historical data governance rules, and constructs a rule base; with the help of this rule base, the system can quickly respond to new needs based on historical experience, and effectively govern in real-time data streams; this method makes full use of the experience of historical data governance, reduces the repeated definition and adjustment of real-time data governance rules, and improves the level of automation. When generating adjustment suggestions for real-time data governance rules, the present invention can output the adjustment suggestions to relevant personnel, who can make real-time adjustments manually based on the suggestions; this combination of automation and manual intervention not only maintains a high degree of automation, but also ensures that the advantages of manual judgment can be fully utilized when facing complex and uncertain business needs. The present invention can continuously learn and adjust to changes in business needs of different types and scales through dynamic division, comparison and keyword difference analysis of real-time business demand information, thereby enhancing the system's adaptability and scalability in different application scenarios; as enterprise needs continue to evolve, the system can achieve long-term stable and efficient operation by continuously updating the data governance rule base. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a module schematic diagram of a rule-based automated data governance system of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1 , the present invention provides a technical solution: A rule-based automated data governance system, including: a historical data analysis and rule base construction module, a real-time demand analysis and demand determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module; The historical data analysis and rule base construction module collects historical business demand information and historical data, analyzes the historical business demand information and historical data, and matches the historical business demand information with the corresponding historical data; combined with the historical data processing process, it obtains the historical data governance rules corresponding to the historical business demand information and builds a data governance rule base; The real-time demand analysis and demand determination module obtains real-time business demand information and real-time data to be governed, divides the real-time data to be governed according to the real-time business demand information, and matches the real-time business demand information with the corresponding real-time data to be governed; compares and analyzes the real-time business demand information with the historical business demand information, and determines whether the real-time business demand information is new business demand information; The data governance simulation and effect evaluation module matches the new business demand information with the historical business demand information for the new business demand information. Based on the historical data governance rules of the matched historical business demand information, it simulates the data governance process of the real-time data to be governed and evaluates the data governance effect. The data governance rule adjustment and application module generates adjustment suggestions for real-time data governance rules corresponding to the new business demand information based on the evaluation results of the data governance effect, and outputs the adjustment suggestions to relevant personnel, who then define the real-time data governance rules corresponding to the new business demand information based on the adjustment suggestions.

[0020] The historical data analysis and rule base construction module includes a historical business demand information analysis unit and a data governance rule base construction unit; The historical business demand information analysis unit collects historical business demand information and historical data, analyzes the historical business demand information, extracts a keyword set, and calculates a correlation coefficient between the historical data record and the demand keyword, and matches the historical business demand information with the historical data; The data governance rule base construction unit obtains corresponding historical data governance rules based on historical business demand information and its corresponding data records, summarizes all historical data governance rules, and establishes a data governance rule base.

[0021] The real-time demand analysis and demand determination module includes a real-time business demand information analysis unit and a new demand determination and matching unit; The real-time business demand information analysis unit collects real-time business demand information and data to be governed, extracts keywords, and associates the real-time business demand information with the data to be governed based on the mapping relationship between historical business demand information and data; The new demand determination and matching unit determines whether the real-time business demand information is new demand information by calculating the similarity of the keyword set; and further determines whether the real-time business demand information is new demand information by analyzing the keyword difference rate and semantic relevance.

[0022] The data governance simulation and effect evaluation module includes a data governance simulation unit and a data governance effect evaluation unit; The data governance simulation unit performs data governance simulation on the real-time data to be governed corresponding to the newly added business demand information by matching the historical data governance rules, and records the simulated data as simulation data; The data governance effectiveness evaluation unit conducts feedback analysis on the simulated data, calculates the accuracy improvement, missing value ratio and consistency coefficient, and obtains a comprehensive evaluation index based on the accuracy improvement, missing value ratio and consistency coefficient.

[0023] The data governance rule adjustment and application module includes a real-time data governance rule adjustment unit and a real-time data governance rule application unit; The real-time data governance rule adjustment unit generates adjustment suggestions for real-time data governance rules based on the data governance effect evaluation results and new business demand information, and outputs the adjustment suggestions to relevant personnel, who then define and adjust the real-time data governance rules based on the adjustment suggestions; The real-time data governance rule application unit performs actual data governance on the real-time data to be governed according to the defined real-time data governance rules, extracts the real-time data feedback results after governance, and calculates comprehensive evaluation indicators; according to the evaluation results of the real-time data governance effect, relevant personnel shall perform corresponding processing and add the real-time data governance rules to the data governance rule library.

[0024] A rule-based automated data governance method includes the following steps: Step S100. Collect historical business demand information and historical data through the cloud platform, analyze the historical business demand information and historical data, and correspond the historical business demand information with the corresponding historical data; combine the processing process of historical data to obtain historical data governance rules corresponding to the historical business demand information, and build a data governance rule library; Step S200. Acquire real-time business demand information and real-time data to be governed, divide the real-time data to be governed according to the real-time business demand information, and correspond the real-time business demand information with the corresponding real-time data to be governed; compare and analyze the real-time business demand information with the historical business demand information, and determine whether the real-time business demand information is new business demand information based on the comparison and analysis results; Step S300. For the new business demand information determined to be new, the new business demand information is matched with the historical business demand information; based on the historical data governance rules of the matched historical business demand information, the data governance process of the real-time data to be governed is simulated, and the data governance effect is evaluated according to the feedback results of the simulation; Step S400. Based on the evaluation results of the data governance effect and in combination with the matching process between the real-time business demand information and the matched historical business demand information, generate adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information, and output the adjustment suggestions to relevant personnel, who define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions.

[0025] Step S100 includes: S101. Collect historical business demand information and historical data through the cloud platform. The historical business demand information refers to demand descriptions and records related to business objectives and operational requirements formed by an enterprise or organization in past business activities. Such information usually includes business objectives, decision-making basis, demand descriptions, scenario settings, and processing procedures. The historical data refers to actual data related to business operations and decisions generated in the past business execution process. Such data are usually directly related to business demand information and are the results of business activity execution. For each piece of historical business demand information, keyword extraction is performed to form a historical business demand keyword set G, and G={g1,g2,...,gn}, where g1 represents the first keyword of the historical business demand information, g2 represents the second keyword of the historical business demand information, and so on, gn represents the nth keyword of the historical business demand information. S102. Mark and match the historical data according to the historical business demand keyword set G, analyze the mapping relationship between the historical business demand keywords and the historical data, and calculate the correlation coefficient L(dk,gi) between each historical record information dk and each keyword gi in the historical data, and the specific calculation formula is: L(dk,gi)=Count((dk,gi) / Length(dk); Wherein, dk represents the kth historical record information in the historical data, k represents the historical record information number in the historical data; gi represents the i-th keyword in the historical business demand keyword set, and i ranges from 1 to n; Count((dk,gi) represents the number of times the keyword gi appears in the kth record information of the historical data; Length(dk) represents the length of the kth record information of the historical data, such as the number of fields or the number of characters; for each historical record information dk, the correlation coefficients L(dk,gi) between the historical record information dk and all the keywords in the historical business demand keyword set G are summarized and averaged to obtain the average correlation coefficient L0; the average correlation coefficient L0 is compared with the correlation threshold L, if L0≥L, the historical record information dk is matched with the historical business demand information corresponding to the historical business demand keyword set G, otherwise no processing is performed; all historical record information in the historical data are traversed to match the historical business demand information with the corresponding historical data; S103. Obtain the processing process of historical data, and according to the processing process of historical data, obtain the historical data governance rules corresponding to the historical business demand information, and summarize the historical data governance rules corresponding to the historical business demand information to form a data governance rule base Z, and Z={z1,z2,...,zm}, where z1 represents the historical data governance rule corresponding to the first historical business demand information, z2 represents the historical data governance rule corresponding to the second historical business demand information, and so on, zm represents the historical data governance rule corresponding to the mth historical business demand information, and m represents the data number of the historical business demand information.

[0026] In this embodiment, it is assumed that the historical business demand information is the bank's approval process, loan interest rate requirements, approval standards, etc. when processing loan approval. The specific historical data includes historical loan records, approval time, customer type, loan amount and other data.

[0027] Keyword extraction is performed on each piece of historical business demand information to form a historical business demand keyword set G. For example, if the keywords of a piece of historical business demand information are: g1 = "loan approval", g2 = "interest rate", g3 = "customer credit", g4 = "limit", then the corresponding historical business demand keyword set G = {g1, g2, g3, g4}.

[0028] According to the historical business demand keyword set G, the historical data is marked and matched, and the mapping relationship between the historical business demand keywords and the historical data is analyzed. Assume that a historical information record d1 includes the following data: d1={"Loan Number":"001","Customer Name":"Zhang San","Loan Amount":10000,"Interest Rate":"4.5","Approval Time":"2024-01-15"}; Calculate the correlation coefficient L(dk,gi): Since g1 = "loan approval", and the results generated by the loan approval operation include loan number, customer name, loan amount, interest rate, and approval time, the judgment process can be analyzed by semantic analysis and context pattern matching, so L(d1,g1)=1; Since g2 = "interest rate", and since the interest rate operation corresponds to the interest rate in d1, L(d1,g2)=1 / 5=0.2; Since g3 = "customer credit", and since customer credit is directly related to customer name and loan amount, L(d1,g3)=2 / 5=0.4; Since g4 = "credit limit", and since the credit limit is directly related to the loan amount, L(d1,g4)=1 / 5=0.2; Then the average correlation coefficient corresponding to this historical business demand information is L0=(1+0.2+0.4+0.2) / 4=0.45; Assume that the association threshold L=0.3 and L0>L, so the historical record d1 corresponds to this historical business demand information; obtain the processing process of the historical data corresponding to the historical record d1, and obtain the data governance rule z of the historical data corresponding to the historical record d1 by this historical business demand information.

[0029] Step S200 includes: S201. Acquire real-time business demand information and real-time data to be governed, wherein the real-time data to be governed refers to data that needs to be governed in combination with the real-time business demand information; analyze the real-time demand information in the same way as the historical demand information, thereby forming a real-time business demand keyword set S for each piece of real-time business demand information, and S={s1,s2,...,sf}, wherein s1 represents the first keyword of the real-time business demand information, s2 represents the second keyword of the real-time business demand information, and so on, sf represents the fth keyword of the real-time business demand information; refer to the historical business demand information and the corresponding historical data for the corresponding analysis process, and correspond the real-time business demand information with the corresponding real-time data to be governed; S202. Extract the historical business demand keyword set G corresponding to each historical business demand information, and calculate the similarity between the historical business demand keyword set G corresponding to each historical business demand information and the real-time business demand keyword set S corresponding to each real-time business demand information, and the similarity calculation formula is: C=N(G∩S) / N(G∪S), where N(G∩S) represents the number of keywords of the intersection result of the historical business demand keyword set G and the real-time business demand keyword set S, and N(G∪S) represents the number of keywords of the union result of the historical business demand keyword set G and the real-time business demand keyword set S; when the similarity C is equal to 1 or the real-time business demand keyword set S is a subset of the historical business demand keyword set G, it means that the current real-time business demand information is not new business demand information; when the similarity C is less than 1 or the historical business demand keyword set G is a subset of the real-time business demand keyword set S, go to S203 for further analysis; When the similarity is low, it indicates that the real-time business requirements contain some new keywords, which may represent new requirements. However, this does not mean that all situations that are different from historical requirements are new requirements. Some situations with low similarity may also be due to slight adjustments to the same requirement or different expressions, rather than completely new business requirements. Therefore, in addition to similarity calculation, other supplementary judgment methods may be needed to further confirm whether it is a new requirement.

[0030] S203. Calculate the keyword difference rate B, and the specific calculation formula is: B=[N(S)-N(G∩S)] / N(S), where N(S) represents the number of keywords in the real-time demand information; calculate the semantic relevance Yg between the newly added keyword set X and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and X=SG, the calculation formula of the semantic relevance Yg is: Yg=1 / |X|Σ xj∈X [α·Sim(xj,G∩S)], where Sim(xj,G∩S) represents the semantic similarity between the newly added keyword xj and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and α represents the weight parameter; according to the keyword difference rate B and the semantic relevance Yg, the new demand judgment index P is calculated, and P=β×B-γ×Yg, where β and γ represent the weights of the keyword difference rate B and the semantic relevance Yg respectively, and β+γ=1; if the new demand judgment index P is greater than or equal to the threshold P0, it means that the current real-time business demand information is new business demand information, otherwise, the current real-time business demand information is not new business demand information.

[0031] Step S300 includes: S301. For the newly added business demand information, according to the similarity calculation between the historical business demand information and the real-time business demand information in S202, the historical business demand information with the largest similarity C is selected as the matching result, and C < 1; according to the matched historical business demand information, the corresponding historical data governance rules are searched in the data governance rule library; according to the corresponding historical data governance rules, the data governance process of the real-time data to be governed corresponding to the newly added business demand information is simulated, and the simulated data is recorded as the simulated data; S302. Obtain the feedback results of the simulation data, and calculate the accuracy improvement index F in combination with the simulation data, and F=Nv / N, where Nv represents the number of records in the simulation data whose accuracy has been corrected after data governance based on the feedback results, and N represents the total number of records in the simulation data; calculate the missing value ratio Q, and Q=Mv / N, where Mv represents the number of missing records in the simulation data after data governance based on the feedback results; calculate the consistency coefficient E, and E=Cs / N, where Cs represents the record data that meets the data consistency after data governance based on the feedback results in the simulation data; comprehensively consider the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E to calculate the comprehensive evaluation index R, and R=w1×F-w2×Q+w3×E, where w1, w2 and w3 represent the weights of the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E, respectively, and w1+w2+w3=1.

[0032] In this embodiment, it is assumed that the following results are obtained after simulating data feedback: The accuracy improvement index F=0.65, which means that 65% of the records have been corrected; The missing value ratio Q=0.55, which means that 55% of the records have missing values; The consistency coefficient E=0.4, indicating that 40% of the records meet the data consistency requirements.

[0033] Assuming the given weights are: w1=0.4, w2=0.3, w3=0.3; then, the comprehensive evaluation index R is calculated as: R=0.4×0.65-0.3×0.55+0.3×0.4=0.215.

[0034] Step S400 includes: S401. Based on the evaluation results of the data governance effect, and in combination with the matching process between the real-time business demand information and the matched historical business demand information, the matched historical business demand information is used as the basis for adjusting the data governance rules corresponding to the newly added business demand information; the newly added keyword set X is extracted, and the data governance rule adjustment basis and the newly added keyword set X are used as adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information; S402. Output the adjustment suggestions of the real-time data governance rules corresponding to the newly added business demand information to the relevant personnel, who will define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions; perform data governance on the real-time data to be governed corresponding to the newly added business demand information according to the real-time data governance rules, so as to obtain real-time data, extract the feedback results of the real-time data, and calculate the comprehensive evaluation index R' of the real-time data with reference to the calculation process of the comprehensive evaluation index R of the simulated data in S302; if R>R' or R'<R0, add the real-time data governance rules to the data governance rule library, where R0 represents the comprehensive evaluation index threshold; if R>R' or R'<R0, output prompt information to the relevant personnel, who will conduct further analysis and adjustment until R<R' and R'≥R0 are satisfied, and add the adjusted real-time data governance rules to the data governance rule library.

[0035] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0036] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A rule-based automated data governance method, characterized by: The method comprises the following steps: Step S100. Collect historical business demand information and historical data through the cloud platform, analyze the historical business demand information and historical data, and correspond the historical business demand information with the corresponding historical data; combine the processing process of historical data to obtain historical data governance rules corresponding to the historical business demand information, and build a data governance rule library; Step S200. Acquire real-time business demand information and real-time data to be governed, divide the real-time data to be governed according to the real-time business demand information, and correspond the real-time business demand information with the corresponding real-time data to be governed; compare and analyze the real-time business demand information with the historical business demand information, and determine whether the real-time business demand information is new business demand information based on the comparison and analysis results; Step S300. For the new business demand information determined to be new, the new business demand information is matched with the historical business demand information; based on the historical data governance rules of the matched historical business demand information, the data governance process of the real-time data to be governed is simulated, and the data governance effect is evaluated according to the feedback results of the simulation; Step S400. Based on the evaluation results of the data governance effect and in combination with the matching process between the real-time business demand information and the matched historical business demand information, generate adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information, and output the adjustment suggestions to relevant personnel, who define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions.

2. The rule-based automated data governance method according to claim 1, characterized in that: The step S100 includes: S101. Collect historical business demand information and historical data through the cloud platform, wherein the historical business demand information refers to demand descriptions and records related to business objectives and operational needs formed by an enterprise or organization in past business activities; the historical data refers to actual data related to business operations and decisions generated in past business execution processes; for each piece of historical business demand information, perform keyword extraction to form a historical business demand keyword set G, where G={g1,g2,...,gn}, where g1 represents the first keyword of the historical business demand information, g2 represents the second keyword of the historical business demand information, and so on, gn represents the nth keyword of the historical business demand information; S102. Mark and match the historical data according to the historical business demand keyword set G, analyze the mapping relationship between the historical business demand keywords and the historical data, and calculate the correlation coefficient L(dk,gi) between each historical record information dk and each keyword gi in the historical data, and the specific calculation formula is: L(dk,gi)=Count((dk,gi) / Length(dk); Wherein, dk represents the kth historical record information in the historical data, k represents the historical record information number in the historical data; gi represents the i-th keyword in the historical business demand keyword set, and i ranges from 1 to n; Count((dk,gi) represents the number of times the keyword gi appears in the kth record information of the historical data; Length(dk) represents the length of the kth record information of the historical data; for each historical record information dk, the correlation coefficients L(dk,gi) between the historical record information dk and all the keywords in the historical business demand keyword set G are summarized and averaged to obtain the average correlation coefficient L0; the average correlation coefficient L0 is compared with the correlation threshold L, if L0≥L, the historical record information dk is matched with the historical business demand information corresponding to the historical business demand keyword set G, otherwise no processing is performed; all historical record information in the historical data are traversed to match the historical business demand information with the corresponding historical data; S103. Obtain the processing process of historical data, and according to the processing process of historical data, obtain the historical data governance rules corresponding to the historical business demand information, and summarize the historical data governance rules corresponding to the historical business demand information to form a data governance rule base Z, and Z={z1,z2,...,zm}, where z1 represents the historical data governance rule corresponding to the first historical business demand information, z2 represents the historical data governance rule corresponding to the second historical business demand information, and so on, zm represents the historical data governance rule corresponding to the mth historical business demand information, and m represents the data number of the historical business demand information.

3. The rule-based automated data governance method according to claim 2, characterized in that: The step S200 includes: S201. Acquire real-time business demand information and real-time data to be governed, wherein the real-time data to be governed refers to data that needs to be governed in combination with the real-time business demand information; analyze the real-time demand information in the same way as the historical demand information, thereby forming a real-time business demand keyword set S for each piece of real-time business demand information, and S={s1,s2,...,sf}, wherein s1 represents the first keyword of the real-time business demand information, s2 represents the second keyword of the real-time business demand information, and so on, sf represents the fth keyword of the real-time business demand information; refer to the historical business demand information and the corresponding historical data for the corresponding analysis process, and correspond the real-time business demand information with the corresponding real-time data to be governed; S202. Extract the historical business demand keyword set G corresponding to each historical business demand information, and calculate the similarity between the historical business demand keyword set G corresponding to each historical business demand information and the real-time business demand keyword set S corresponding to each real-time business demand information, and the similarity calculation formula is: C=N(G∩S) / N(G∪S), where N(G∩S) represents the number of keywords of the intersection result of the historical business demand keyword set G and the real-time business demand keyword set S, and N(G∪S) represents the number of keywords of the union result of the historical business demand keyword set G and the real-time business demand keyword set S; when the similarity C is equal to 1 or the real-time business demand keyword set S is a subset of the historical business demand keyword set G, it means that the current real-time business demand information is not new business demand information; when the similarity C is less than 1 or the historical business demand keyword set G is a subset of the real-time business demand keyword set S, go to S203 for further analysis; S203. Calculate the keyword difference rate B, and the specific calculation formula is: B=[N(S)-N(G∩S)] / N(S), where N(S) represents the number of keywords in the real-time demand information; calculate the semantic relevance Yg between the newly added keyword set X and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and X=SG, the calculation formula of the semantic relevance Yg is: Yg=1 / |X|Σ xj∈X [α·Sim(xj,G∩S)], where Sim(xj,G∩S) represents the semantic similarity between the newly added keyword xj and the union of the historical business demand keyword set G and the real-time business demand keyword set S, and α represents the weight parameter; according to the keyword difference rate B and the semantic relevance Yg, the new demand judgment index P is calculated, and P=β×B-γ×Yg, where β and γ represent the weights of the keyword difference rate B and the semantic relevance Yg respectively, and β+γ=1; if the new demand judgment index P is greater than or equal to the threshold P0, it means that the current real-time business demand information is new business demand information, otherwise, the current real-time business demand information is not new business demand information.

4. The rule-based automated data governance method according to claim 3, characterized in that: The step S300 includes: S301. For the newly added business demand information, according to the similarity calculation between the historical business demand information and the real-time business demand information in S202, the historical business demand information with the largest similarity C is selected as the matching result, and C < 1; according to the matched historical business demand information, the corresponding historical data governance rules are searched in the data governance rule library; according to the corresponding historical data governance rules, the data governance process of the real-time data to be governed corresponding to the newly added business demand information is simulated, and the simulated data is recorded as the simulated data; S302. Obtain the feedback results of the simulation data, and calculate the accuracy improvement index F in combination with the simulation data, and F=Nv / N, where Nv represents the number of records in the simulation data whose accuracy has been corrected after data governance based on the feedback results, and N represents the total number of records in the simulation data; calculate the missing value ratio Q, and Q=Mv / N, where Mv represents the number of missing records in the simulation data after data governance based on the feedback results; calculate the consistency coefficient E, and E=Cs / N, where Cs represents the record data that meets the data consistency after data governance based on the feedback results in the simulation data; comprehensively consider the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E to calculate the comprehensive evaluation index R, and R=w1×F-w2×Q+w3×E, where w1, w2 and w3 represent the weights of the accuracy improvement index F, the missing value ratio Q and the consistency coefficient E, respectively, and w1+w2+w3=1.

5. A rule-based automated data governance method according to claim 4, characterized in that: The step S400 includes: S401. Based on the evaluation results of the data governance effect, and in combination with the matching process between the real-time business demand information and the matched historical business demand information, the matched historical business demand information is used as the basis for adjusting the data governance rules corresponding to the newly added business demand information; the newly added keyword set X is extracted, and the data governance rule adjustment basis and the newly added keyword set X are used as adjustment suggestions for the real-time data governance rules corresponding to the newly added business demand information; S402. Output the adjustment suggestions of the real-time data governance rules corresponding to the newly added business demand information to the relevant personnel, who will define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions; perform data governance on the real-time data to be governed corresponding to the newly added business demand information according to the real-time data governance rules, so as to obtain real-time data, extract the feedback results of the real-time data, and calculate the comprehensive evaluation index R' of the real-time data with reference to the calculation process of the comprehensive evaluation index R of the simulated data in S302; if R<R' and R'≥R0, add the real-time data governance rules to the data governance rule library, where R0 represents the comprehensive evaluation index threshold; if R>R' or R'<R0, output prompt information to the relevant personnel, who will conduct further analysis and adjustment until R<R' and R'≥R0 are satisfied, and add the adjusted real-time data governance rules to the data governance rule library.

6. A rule-based automated data governance system, applied to a rule-based automated data governance method according to any one of claims 1 to 5, characterized in that: The system includes: a historical data analysis and rule base construction module, a real-time demand analysis and demand determination module, a data governance simulation and effect evaluation module, and a data governance rule adjustment and application module; The historical data analysis and rule base construction module collects historical business demand information and historical data, analyzes the historical business demand information and historical data, and matches the historical business demand information with the corresponding historical data; combines the historical data processing process to obtain the historical data governance rules corresponding to the historical business demand information, and constructs a data governance rule base; The real-time demand analysis and demand determination module obtains real-time business demand information and real-time data to be governed, divides the real-time data to be governed according to the real-time business demand information, and matches the real-time business demand information with the corresponding real-time data to be governed; compares and analyzes the real-time business demand information with the historical business demand information, and determines whether the real-time business demand information is new business demand information; The data governance simulation and effect evaluation module matches the new business demand information with the historical business demand information for the new business demand information judged as new business demand information; simulates the data governance process of the real-time data to be governed based on the historical data governance rules of the matched historical business demand information, and evaluates the data governance effect; The data governance rule adjustment and application module generates adjustment suggestions for real-time data governance rules corresponding to the newly added business demand information based on the evaluation results of the data governance effect, and outputs the adjustment suggestions to relevant personnel, who then define the real-time data governance rules corresponding to the newly added business demand information based on the adjustment suggestions.

7. The rule-based automated data governance system according to claim 6, characterized in that: The historical data analysis and rule base construction module includes a historical business demand information analysis unit and a data governance rule base construction unit; The historical business demand information analysis unit collects historical business demand information and historical data, analyzes the historical business demand information, extracts a keyword set, and calculates a correlation coefficient between the historical data record and the demand keyword, and matches the historical business demand information with the historical data; The data governance rule library construction unit obtains corresponding historical data governance rules based on historical business demand information and its corresponding data records, summarizes all historical data governance rules, and establishes a data governance rule library.

8. The rule-based automated data governance system according to claim 6, characterized in that: The real-time demand analysis and demand determination module includes a real-time business demand information analysis unit and a new demand determination and matching unit; The real-time business demand information analysis unit collects real-time business demand information and data to be governed, extracts keywords, and associates the real-time business demand information with the data to be governed according to the mapping relationship between historical business demand information and data; The new demand determination and matching unit determines whether the real-time business demand information is new demand information by calculating the similarity of the keyword set; and further determines whether the real-time business demand information is new demand information by analyzing the keyword difference rate and semantic relevance.

9. The rule-based automated data governance system according to claim 6, characterized in that: The data governance simulation and effect evaluation module includes a data governance simulation unit and a data governance effect evaluation unit; The data governance simulation unit performs data governance simulation on the real-time to-be-governed data corresponding to the newly added business demand information by matching the historical data governance rules, and records the simulated data as simulation data; The data governance effect evaluation unit performs feedback analysis on the simulation data, calculates the accuracy improvement, the missing value ratio and the consistency coefficient, and obtains a comprehensive evaluation index based on the accuracy improvement, the missing value ratio and the consistency coefficient.

10. The rule-based automated data governance system according to claim 6, characterized in that: The data governance rule adjustment and application module includes a real-time data governance rule adjustment unit and a real-time data governance rule application unit; The real-time data governance rule adjustment unit generates adjustment suggestions for the real-time data governance rules based on the data governance effect evaluation results and the newly added business demand information, and outputs the adjustment suggestions to relevant personnel, who define and adjust the real-time data governance rules based on the adjustment suggestions; The real-time data governance rule application unit performs actual data governance on the real-time data to be governed according to the defined real-time data governance rules, extracts the real-time data feedback results after governance, and calculates comprehensive evaluation indicators; according to the evaluation results of the real-time data governance effect, relevant personnel perform corresponding processing and add the real-time data governance rules to the data governance rule library.

Images